Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods
Authors: Jonathan Kamp
Lisa Beinborn
Antske Fokkens
Venue: EMNLP 2023 Main (2023)
Keywords: interpretability
explainability
top-k
ranking
agreement
feature attribution

Abstract: Feature attribution scores are used for explaining the prediction of a text classifier to users by highlighting a k number of tokens. In this work, we propose a way to determine the number of optimal k tokens that should be displayed from sequential properties of the attribution scores. Our approach is dynamic across sentences, method-agnostic, and deals with sentence length bias. We compare agreement between multiple methods and humans on an NLI task, using fixed k and dynamic k. We find that perturbation-based methods and Vanilla Gradient exhibit highest agreement on most method--method and method--human agreement metrics with a static k. Their advantage over other methods disappears with dynamic ks which mainly improve Integrated Gradient and GradientXInput. To our knowledge, this is the first evidence that sequential properties of attribution scores are informative for consolidating attribution signals for human interpretation.

Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods
Jonathan Kamp'
Lisa Beinborn' Antske Fokkens 1.2
*Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam Dept. of Mathematics and Computer Science, Eindhoven University of Technology (j. b. kamp, 1. beinborn, antske. fokkens)evu.n!!
Abstract
Dynamiic
Feature attribution scores are used for explain- ing the prediction of a text classifier to users by highlighting & number of tokens. In this work, we propose a way to determine the num- ber of optimal & tokens that should be dis- played from sequential properties of the at- tribution scores. Our approach is dynamic across sentences, method- -agnostc, and deals with sentence length bias. We compare agree- ment between multiple methods and humans on an NLI task, using fixed &t and dynamic k. We find that ceturbattoonbbaed methods and Vanilla Gradient exhibit highest agreement most method-method and method-hunet human agreement metrics with static k. Their ad- vantage over other methods disappears with dynamic ks which mainly improve Integrated Gradient and GradientXInput. To our knowl- edge. this is the first evidence that sequential properties of attribution scores are informative for consolidating attribution signals for human interpretation.
Food k-?
shirts
Repper
Tier men
aggue will Baad
Figure Dynamic k versus fixed k (Vanilla Gradient).
factors such as the exact task, data, and model have led to contradictory conclusions. For exam- plc, Attanasio al. (2022) find that perturbation- based methods better agree with human preferences than gradient-based methods, while Atanasova al. (2020) report the reverse tendency.
1 Introduction
A factor that may also influence results but has been largely understudied the impact of the cho- sen number of k most salient tokens that are taken into consideration. Previous analyses either use a fixed value of k or determine the most salient to- kens based values. A clear drawback of using fixed num- ber of k is that can result in excluding tokens with scores close to the top-k and including tokens with low scores, disregarding significant score gaps with truly important tokens. Jesus et al. (2021) set k to fixed value of six and examine the effect of visualizing the most salient features on decision- making accuracy, time, and agreement for fraud analysis task. Bastings al. (2022) set l: the low fixed values of and 2 as their analysis focuses on the identification of specific shortcut cues in a closed experimental setup. Camburu et al. (2019) determine k using a fixed threshold and identify
Feature attribution scores are a glimpse behind the scenes of neural models, or at least that is the promise. Various interpretability methods have been developed that can generate attribution scores interpret the degree which a language model's s features (tokens) contributed to the predicted label. However, attribution values from different methods can vary considerably even on the same instance (Madsen et al., 2022, i.a.). Contradictory interpreta- tions cast doubts on their usefulness and reliability in practical setting.
fixed threshold for the attribution
When comparing attribution methods, the focus commonly lies assessing agreement with hu- man explanations (often referred to as plausibility (Jacovi and Goldberg, 2020)) and on agreement be- tween methods (Neely al., 2022; Krishna et al., 2022). However, the evaluation procedures have not been standardised yet and varying influential

---

tokens as salient if their attribution is 0.1. When visualizing attributions to users, they use constant values of (5 and 10). Absolute attribution scores tend to decrease for sentences with a larger number tokens which is not captured by their threshold- based approach. Krishna et al. (2022) determine k 1/1 fraction of the average sentence length in the data and set it to 11.
Data We use the e-SNLI dataset with the default split of 549,361 instances for training, 9,842 for development, and 9,824 for testing (Camburu al., 2018). Each instance consists of a premise, a hy- pothesis, and an output label that indicates the se- mantic relation between the premise and the hy- pothesis: contradiction, entailment, neutral. Each premise is paired with three hypotheses (one for each label) to obtain balanced classes. Instances span 21 tokens on average (5-113). 6,325 amno- tators highlighted tokens that they found most im- portant to explain the gold label (avg. number of highlighted tokens: 4+3). We used the dev and test instances which were annotated by at least three annotators (we used dev for exploration, and test for our experiments).
As token-level agreement generally increases the closer gets to the number of tokens in sentence (see Appendix C for a visualisation of the sentence length bias), the evaluation of attribution methods is clearly top-k dependent. Li ct al. (2020) indicate that intrinsic model evaluation is generally sensi- tive to different values of k but do not measure its effect on agreement. Both Neely et al. (2022) and Krishna et al. (2022) find consistent disagreement between attribution methods, but do not account for the influence of k.
Backbone Model We fine-tune DistilBERT (Sanh et al., 2019) using ten different random seeds and select the median model, i.e., the model with the least variation in attribution profiles compared to the other nine models, for further analysis. 2 The model yields a performance of 0.89 F1 on both the dev and test set.
In this paper, we systematically explore the role of & on the observed method- method agreement and method-human agreement for extracting explanations for natural language inference task. For this purpose, we develop the new metric agreement@k: k. We propose to deter- mine dynamically for each method and each in- stance based on the sequential properties of the attribution profile. Our approach inspired by methods for event detection and detects attribution peaks in a bethod-agnostie fashion that circum- vents the sentence-length bias by allowing different values of & for each instance. We find that:
Contributions
Feature Attribution We use the Ferret package v0.4.1 (Attanasio et al., 2023) to calculate attri- butions using the gradient-based methods Vanilla Gradient (Simonyan et al., 2014) and Integrated Gradient (Sundararajan et al., 2017), and the perturbation-basee methods Partition SHAP (Lund- berg and Lee, 2017) and LIME (Ribeiro al., 2016). For the gradient- methods, we use both the plain gradients and the gradientXInput version for which the gradient is multiplied by the input token embeddings (Shrikumar et al., 2017). Hence, we examine a total of six methods.
agreement at the token level is sensitive to dif- ferent values of & and the effect varies across attribution methods;
Evaluation Recent studies evaluate feature attri- butions as a ranking task. Atanasova et (2020) calculate the mean average precision (MAP) com- pared to human labels and Bastings et al. (2022) restrict the evaluation to the top k ranks (MAP@ k) but it remains an open question how k is selected.
determining k with respect to attribution pro- files consolidates the disagreement between at- tribution methods, in particular for Integrated Gradient and GradientXInput.
We take a novel perspective on human attribution evaluation by interpreting it as a ranking task.
An attribution method A assigns attribution vector {al, ; 02 an) to sentence consist- {wi, W2, ..., wr} so that each
ing of tokens a indicates the salience of token w; for the pre- dicted output label. We determine the topk {t1, f2, } by selecting the k tokens with the highest attribution values. We propose a new metric
2 Experimental Setup
We fine-tune a model on a natural language in- ference task and analyze the agreement between feature attribution methods.
All analyses are available at: https://ggiths com/ jbkamp/repo- hakamp/repo-DDyamii-- Dynamic-K.
"See Appendix A for implementation details on the model selection.

---

PetEUP 1999
0355
0.0 Mostogr
(a) Method-Method Agreement
(b) Miethod-Hummee Agreement
Figure 2: Mean agreement @ l for pairs of attribution methods (left) and between methods and humans (right).
for comparing m attribution methods A Am by calculating sentence-level agreement@A & based on token relevance. Relevance for a token wy is determined by the ratio of methods that include the token in the lopk. A high relevance score indicates that token is assigned high attribution by many of the compared methods.
Method-Method Agreement We identify three groups of mean agreement k across pairs at- tribution methods in Figure 2a. The agreement between Partition SHAP-LIME clearly stands out (yellow line), in particular for small k. All com- parisons involving Integrated Gradient or Gradient- XInput end up in the group with the least agree- ment (purple lines) and the remaining pairs obtain medium-level agreement (green lines). Agreement increases for bigger for pairs of methods with (1) low to medium agreement. Our results contrast pre- vious work which identified higher pairwise agree- ment between gradient- based methods compared to perturbation-based methods (Krishna et al., 2022).
m
Σ A1=1
w E topk
Relevance r(w;)
Σ r(ws)
(greement@k(si)
(2)
(r(w)) > 0]
Method-Human Agreement We calculate to- ken relevance for the three human annotators the ratio of annotators who selected the token, therefore in the range [33, .67, 1]. When we compare the attribution methods to human anno- tations, we find that the two perturbation-ban based methods lead to higher agreement than the gradient- based ones (Figure 2b) and that higher values of h: generally lead to better agreement. Our findings partly in line with Attanasio al. (2022) in that perturbation-base methods are more plausi- ble than most gradient- bastd methods, and partly with Atanasova al. (2020) for finding that Vanilla Gradient agrees more with human rationales than perturbation-based methods which in turn agree more with human than most of the other gradient- based methods. We contrast Ding and Koehn (2021) who find higher plausibility for Integrated Gradient over Vanilla Gradient. While consistency in performance across studies sheds light on the interrelatedness between methods, it is important to exercise caution when generalizing evaluation results across different models, datasets, and tasks.
This way, tokens not included in the top-k lection of any method are assigned relevance score of zero and are not considered in the cal- culation agreement@** This approach avoids artificially inflating the agreement@A & score by hav- ing high agreement on non-relevant tokens. The agreement@* k for a dataset of sentences D {s1, $2, . {a} is computed by averaging the sentence-level agreement@* @k scores.
(D) agrecment@k(s.)
d
We evaluate our experiments comparing mean agreement@K, between the six attribution methods on the test data of e-SL
3 Agreement at Fixed k
We compare attribution methods with each other and with human labels and analyze the role of k.

---

4 Dynamic Top-k Estimation
that when determining dynamically, the agree- ment of Integrated Gradient and GradientXInput with the other methods increases. We find that dy- namic indeed varies across instances and across methods. For example, attributions by Partition SHAP leads to fewer local maxima and therefore to lower values for dynamic & 14.5 on average) compared to IntegratedGradient (7.322.6). We note that these ranges close to human prefer- ences for on the NLI task (4+3)..
We have seen that the value of & has a strong influence on the agreement between methods. citrrbattionbbssed methods more in line with human annotations for smaller settings of k while the attribution profiles of gradient-basee based methods require relatively larger settings of to obtain a similar reflection of human preferences. We pro- pose to determine the number of k salient tokens dynamically based on the attribution profiles of the methods.
5 Further Insights
Inspired by event detection in time series (Taylor (2000), Palshikar et al. (2009), e.g.). we consider at- tribution profiles as sequences of token-level scores that indicate the local presence or absence of peak. Each peak is a point in the sequence (i.e., the sentence) to which the model attributed higher salience compared to neighboring points. We apply peak detection based on local maxima in the attri- bution profile to estimate a that dynamic across method-instance combinations. The attribution profile of each individual method thus serves as the indicator of its peaks. A local maximum is defined point zę in sequence such that it is greater than both its immediate left neighbor, T;-1, and its immediate right neighbor, 1 We additionally enforce the constraint that xy needs to be higher than the mean attribution of the sequence, corre- sponding to above- average model behavior. With our dynamic & approach, we favor relative differ- ences in attribution values over absolute thresholds for identifying the top-k tokens. Figure 3 shows
We briefly discuss complementary insights to the main results, as well as two examples of dynamic k on instances from the dataset.
Average Human Preference Human annotations of token-level explanations were available for this task and compared the effect of dynamic & to the average number of annotations. However, these types of annotations are expensive, which often leads to selecting a fixed k. Normally, a valid ap- proach choosing a fixed A that close to the av- erage number of annotated tokens per sentence, as the resulting ranges of dynamic k for our methods reflected. In absence of annotations, this average number is unknown.
Dynamic k highlights the implications of choos- ing a fixed & that deviates from the average. We ob- serve a clear distinetion between lower and higher than avcrage for GradientXInput and for Integrated Gradient. Table reports the improvement on mean agreement @ k by the dynamic approach over fixed values. Generally, the lower fixed k. the more im- provement we observe. For k > 5 there is no improvement.
K1
6.7
3.00
3.00
2,01
0.00
0.00
k
2
3
stricca
3.
3.01
3.00
4.00
0.01
0.00
Grad X
+.10
+.08
+.05
+.02
+.01
400
010
IntGrad +.01
+.04
+.02
+.01
+.00
US4
0:0
Table 1: Absolute difference on mean agreement @k by dynamic over fixed (GradientXInput and Integrated Gradient). Summed method-methoo agreement scores are reported. Dynamic k is compared to different values of fixed E 21, 4, 5).
PHTSHAP
1//
0.
0.03
Wedrad
D20
2:3
163
0.02
homan
055
2:5
9.62
ME2rdd
seDkadd F1
Lskg
Perbwa
Hestnad
Noman
This further suggests that, in the absence of human annotations, dynamic l may provide an estimate of the average human preference for k.
Figure 3: Mean agreement@* @k: dynamic l (bottom left) and difference (top right) compared to fixed human average A 4. The brighter the color, the higher the agreement.
Peaks as Signals better understand the dynamic & approach. Figures
We analyse two examples to

---

and 4 show the application of dynamic k versus fixed set to 3, and 7 for two different methods.
preference for k and is therefore viable alternative in the absence of task task-specific human data. Further- more, as dynamic k is estimated for each instance separately, it can account for sentence length bias. Our peak detection method is an intuitive ap- proach for determining salient tokens based solely on attribuution profiles. It focuses on isolated key tokens which might not adequately capture human tendencies of chunking words into phrases. In fu- ture work, we plan analyze how our findings generalize to other task and model conditions and want to explore alternative methods to dynamically determine by combining the attribution with lin- guistic information towards better span-level vi- sualisations (see Figure 5). This line of research needs to be closely coupled with cognitive analyses of human preferences.
Bynnmiic
Pradct
Fixed km?
Figure 4: Dynamic k versus fixed & (Integrated Gradi- ent).
In Figure 1, dynamic k highlights the same top three tokens that have been selected by fixed k. namely reading, paper and argue. Tokens that are only selected by
and = are not included as they do not form peaks in the attribution profile.
Span
Figure 4 illustrates differences in the attribution patterns captured by fixed l and dynamic k. While fixed & selects any tokens in descending order of attribution score, dynamic & captures signals. It highlights two peaks that were not captured by fixed l (due their lower absolute values) because they stand out relative the surrounding tokens. Signals can be described as tokens that represent a salient part or phrase in the sentence beyond the token level (e.g. a man in a suit; a lady). By focusing on signals, dynamic k skips tokens with relatively high score but that can be attributed to the same signal of neighboring salient token.
men
polo
atorts
resiing
the
papeo
argse selts
etter
Figure Towards future research on span-based rele- vance based dynamic k.
Limitations
In this study, we encountered certain limitations that should be taken into account when interpret- ing the results and when conducting subsequent research. First, due pragmatic constraints, we focused on single model and a selected set of attribution methods for comparison, which restricts the direct generalisation of our findings to meth- ods outside this set. However, the proposed metric agreement@ @ k and the concept of dynamic & can be readily applied to evaluate other methods in future research. The scarce availability of multiple human rationales the token level, necessary for creating human aggregation scores, limited our ability expand the scope of this research. Furthermore, is worth noting that the aggregation scores in
6 Discussion and Conclusion
Attribution methods disagree on the salience of to- kens. Our analyses show that the observed level of agreement is sensitive to the number & tokens taken into consideration. We propose a dynamic that can be directly applied to any attribution pro- file. In contrast to fixed k. it takes local relative differences of the attribution values into account. Our analyses with dynamic & indicate that different attribution methods capture varying degrees of at- tribution scope. Determining dynamic & purely on attribution profiles yields a level of plausibility that is comparable to determining the average human
our study fall within the range of [0, .33, .67, 1].

---

Consequently, the precision of the overlap between detected human peaks may be compromised when the number of annotators is low. The resolution of ties in these scores was resolved randomly, which introduces a potential source of improvement and variability in the results. While these limitations should be acknowledged, they do not invalidate the overall contributions of our research. They provide valuable insights into the effectiveness of the selected methods and highlight avenues for fu- ture investigations, such as incorporating additional datasets.
yny detection. In Proceedings NLP Power! The First Workshop on Efficient Benchmarking in NLP. pages 100-112, Dublin, Ireland. Association for Computational Linguistics.
Giuseppe Attanasio, Eliana Pastor, Chiara Di Bonaven- tura, and Debora Nozza, 2023. ferret: a framework for benchmarking explainers on transformers. In Proceedings of the 17th Conference the European Chapter of the Association for Computational Lin- guisties: System Demonstrations, pages 256-266, Dubrovnik, Croatia. Association for Computational Linguistics.
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia. Anders Sandholm, and Katja Filippova. 2022. "will you find these shortcuts?" protocol for evaluating the faithfulness of input salience methods for text classification. In Proceedings of the 2022 Confer- ence Empirical Methods in Natural Language Processing, pages 976-991, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ethics Statement
In the field of interpretability, results need to be communicated with particular caution avoid an- thropomorphizing neural models. With respect to this study, caution should be exercised when inter- preting findings from attribution methods. Attri- bution scores cannot be blindly relied upon to pre- cisely determine model functioning, they can be influenced by experimental factors such task and model performance. To avoid drawing generalised conelusions, it is advisable to employ multiple met- ries when studying feature attribution.
Mana-Maria Camburu, Eleonora Giunchiglia, Jakob Fo- erster, Thomas Lukasiewicz, and Phil Blunsom. 2019, Can i trust the explainer? verifying post-hoc explana- tory methods. arXiv preprint 19Xiv:190000000 10.00205..
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. Advances in Neural Information Processing Systems, 31.
Acknowledgements
Shuoyang Ding and Philipp Koehn. 2021. Evaluating saliency methods for neural language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies. pages 5034-5052.
Jonathan Kamp's 's research was funded by the Dutch National Science Organisation (NWO) through the project InDeep: Interpreting Deep Learning Models for Text and Sound (NWA. 1t..1.9.....999..
Antske Fokkens was supported by the EU Hori- zon 2020 project InTaVia: In In/aangible European Heritage- - Visual Analysis, Curation and Communi- cation (http:/intaviaaee)) under grant agreement No. 101004825. Lisa Beinborn's work was funded by the Dutch National Science Organisation (NWO) through the VENI program Veni.22 IC.039). We would like to thank the anonymous reviewers for their valuable contribution.
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable nlp systems: How should we de- fine and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4998-4205.
Sérgio Jesus, Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, and João Gama. 2021. How can choose explainer? an application- grounded evaluation of post-hoc explanations. Proceedings of the 2021 ACM Conference on Fair- ness, Accountability, and Transparency, pages 805- 815.
References
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pom- bra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. 2022. The disagreement problem in ex- plainable machine learning: A practitioner's perspec- tive. arXiv preprint 2.2.::2222.0166.
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classifi- cation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256-3274
Jierui Li. Lemao Liu, Huayang Li, Guanlin Li. Guoping Huang, and Shuming Shi. 2020. Evaluating expla- nation methods for neural machine translation. In Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 365-375.
Giuseppe Attanasio, Debora Nozza, Eliana Pastor, and Dirk Hovy. 2022. Benchmarking post-hoc inter- pretability approaches for transformer-bsee misog-

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv e-prints, pages arXiv-1907.
First, it is possible compute the Average Dif- ference (AD) between two matrices T'1 and T2 of the same size. AD is a measure of dissimilarity and provides an indication of the overall average magnitude of differences hetween the matrices. In our case, Tl and T2 are two attribution matrices and are constructed by concatenating the vectors of token-wise attribution scores {a1. 42..... On (computed for all attribution methods A) for each instance in our dataset of size d, after O-padding as to maximum sentence length. More precisely, let Tl and T2 be two matrices of size (d + 1) x m. where n is the number of sentences in the dataset, is the number of attribution methods and m is max- imum sentence length. The AD between matrices T1 and T2 is calculated by taking the average of the element- wise absolute differences between the corresponding elements of T1 and T2.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach interpreting model predictions. Advances in neural information processing systems, 30.
Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoe interpretability for neural nip: A survey. ACM Computing Surveys, 55(8):
Michael Neely, Stefan F Schouten, Maurits Bleeker, and Ana Lucie. 2022. A song of (dis) agreement: Evaluat- ing the evaluation of explainable artificial intelligence in natural language processing. In HHAI2022: Aug- menting Human Intellect, pages 60-78. IOS Press.
Girish Palshikar et al. 2009. Simple algorithms for peak detection in time-series. In Proc. 1st Int. Conf. Advanced Data Analysis, Business Analytics and In- telligence, volume 122.
We then construct two attribution matrices and calculate AD for every pair of runs from the pool of 10 models each trained with a different random seed, assigning an APD score to each model by av- eraging the AD scores for that model's attribution matrix in pairwise relation to the other models' at- tribution matrices. The model with lowest APD (in bold in Table 2) was selected for our experiments.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. why should trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135- 1144.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1910001910.01100
run_ #
DistilBERT
run_
0.00613
run_2
0.00620
Avanti Shrikumar, Peyton Greenside, and Anshul Kun- daje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, ICML' 17, page 3145-3153. JMLR.org.
run_3
0.00611
run_4
0.00599
run_5
0.00606
run 6
0.00628
run 7
0.00614
K Simonyan. A Vedaldi, and A Zisserman. 2014, Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceed- ings of the International Conference Learning Representations (ICLR). ICLR.
run 8
0.00595
run_9
0.00604
run_10 10
0.00621
Table 2: Average Pairwise Difference between attribu- tion scores produced by different runs trained on 10 different random seeds.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Interna- tional conference on machine learning, pages 3319-
3328. PMLR.
B Fine-tuning and Analysis
Wayne A Taylor. 2000. Change-point analysis: pow- erful new tool for detecting changes.
The input instances for fine-tuning are premises and hypotheses concatenated by single [SEP] token. We removed 6 instances from the training set where the hypothesis was missing. The main hyperparam- eters for our models are the following: 15 training epochs with carly stopping, training batch size 32, learning rate set to 5e-6, weight decay set to
A Average Pairwise Difference (APD)
The Average Pairwise Difference (APD) indicates how the sets of attribution scores yielded by dif- ferent combinations of classification models with attribution methods differ from one another.

---

0.01 and warmup steps set to 6% of the total. We found that computing the attributions for a larger model such as RoBERTa (Liu et al., 2019) takes significantly longer and aligning the attributions with human annotated text is less straightforward for tokenisation reasons. When pre- procesrcessng the human annotations, we assign 0 score to punctua- tion characters as they did not receive a dedicated annotation label.
Sentence Length Bias
0.8
sont
sont jen
06-
sent_ien
sent
sent_ien
sont Jcn
0.3
5
20
20
30
20
Top.
Figure 6: Sentence length bias overall agreement be- tween methods at different values of top-k, for different groups instances based sentence length.


Reviews:
[{"review_id": "ifycmFbBoH", "reviewer": "Reviewer_Mndj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Message to all reviewers:\n\nWe would like to inform the reviewers that we identified an indexing error in our code. This error affects the column and row in Figure 3 that reports human-method agreement. In particular, the values for GradXI and IntGrad now slightly decrease instead of the increase that we reported in the original submission. We will update the figure in the next version and the sentence at lines 247-248 accordingly, i.e. by removing “and with humans”.\n\nThis does not alter the contributions listed in the introduction nor the conclusion. If these values were important for your assessment, we understand that you might revisit it.\n\n—\n\nYes, we agree with you that extra examples would be a very nice addition. With the additional page of the next version, we will dedicate more space to them. Examples that we would like to discuss:\n\n- Agreement@k with low k (risk of not covering truly relevant tokens) versus high k (risk of sentence length bias) versus dynamic k.\n- Dynamic k not assigning relevance to tokens with negative attribution scores (whereas fixed k does).\n- The selection of tokens by dynamic k that would not make it into a relatively high (e.g. 8) fixed k selection. This relates to the span-based intuition described in Figure 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting analysis of the effect of the number of tokens (k) selected and their selection criteria in improving the agreement for feature attribution using multiple interpretability methods that map model predictions to input features.  The paper is very clearly written and easy to understand, overall. ", "reasons_to_reject": "For experiments with dynamic k, a few more examples from the dataset with the resulting k values from the dynamic selection process as well as corresponding agreement scores could provide further support to the discussion arguments. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work presents a simple method for optimizing the number of tokens selected to be shown to humans when presenting results from explainability methods that attribute model performance and predictions to specific tokens of the input. They test this approach by using it with Vanilla Gradients, Integrated Gradients, and GradientXInput methods for feature attribution for the SNLI dataset. They find that a dynamically adapted value of k leads to better agreement metrics for both method-method and method-human settings. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RaaUsLmfFV", "reviewer": "Reviewer_HqLk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Message to all reviewers:\n\nWe would like to inform the reviewers that we identified an indexing error in our code. This error affects the column and row in Figure 3 that reports human-method agreement. In particular, the values for GradXI and IntGrad now slightly decrease instead of the increase that we reported in the original submission. We will update the figure in the next version and the sentence at lines 247-248 accordingly, i.e. by removing “and with humans”.\n\nThis does not alter the contributions listed in the introduction nor the conclusion. If these values were important for your assessment, we understand that you might revisit it.\n\n—\n\nThank you for your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written, effectively delineating existing issues and establishing the need for the proposed method.\n2. The dynamic k selection approach provides valuable insights into the disparities among attribution methods in terms of token salience.", "reasons_to_reject": "1. I don't have compelling reasons to oppose this paper at the conference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel approach, which is motivated by event detection in time series and span-based peak detection, to find salient tokens from attribution profiles across sentences by dynamically selecting the number of optimal k (tokens) from sequential properties. In comparison to previous methods that used a static k, the approach presented in this paper offers several advantages: (1) It avoids sentence-length bias inherent in static k methods; (2) It consolidates the estimation of disagreement among feature attribution methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4nS8mt9l54", "reviewer": "Reviewer_Cz9e", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Message to all reviewers:\n\nWe would like to inform the reviewers that we identified an indexing error in our code. This error affects the column and row in Figure 3 that reports human-method agreement. In particular, the values for GradXI and IntGrad now slightly decrease instead of the increase that we reported in the original submission. We will update the figure in the next version and the sentence at lines 247-248 accordingly, i.e. by removing “and with humans”.\n\nThis does not alter the contributions listed in the introduction nor the conclusion. If these values were important for your assessment, we understand that you might revisit it.\n\n—\n\nThank you for your review. We agree with you on the fact that including more types of models would expand the scope of the results, and we will leave this exploration to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study seeks to answer a relevant question regarding feature attribution methods applied to language models. Methodology and empirical evaluation are carefully explained, and the conclusions are supported be the empirical results, to the limited scope of the experimental setup. The paper is overall very well written.", "reasons_to_reject": "There is no specific reason to reject this paper in my opinion.\nWhile not a reason to reject and rightly disclaimed by the authors, experimenting on a single model significantly reduces the scope of the findings, thus limiting the score to be given to this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach for dynamically determining (sentence-wise) the top k tokens to be displayed on feature attribution methods. It also conducts empirical analysis on the role of the number k on the observed method–method agreement and method–human agreement for extracting explanations for a natural language inference task. The results confirm the hypothesis that the level of agreement between different attribution methods is sensitive to k, and that perturbation-based methods lead to higher agreement than gradient-based ones.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rSjQice26u", "reviewer": "Reviewer_EnSP", "summary": "", "questions": "A. The denominator of eq. 2 is always positive. What have I missed?\n\nB. The mean agreement@k is said to be obtained by averaging over sentence-level agreement@k, but relevance depends on attribution methods by definition (nominator of Eq. 1), so where is this averaging computed exactly?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Message to all reviewers:\n\nWe would like to inform the reviewers that we identified an indexing error in our code. This error affects the column and row in Figure 3 that reports human-method agreement. In particular, the values for GradXI and IntGrad now slightly decrease instead of the increase that we reported in the original submission. We will update the figure in the next version and the sentence at lines 247-248 accordingly, i.e. by removing “and with humans”.\n\nThis does not alter the contributions listed in the introduction nor the conclusion. If these values were important for your assessment, we understand that you might revisit it.\n\n—\n\nA: We agree with you that we should have been more explicit. If a token has not been included in the top-k selection of any method, it is assigned a relevance score of 0 and it should not be considered in the calculation of agreement@k (hence, $r(w_i)>0$ at the denominator of Eq. 2). The reason for this is that we wanted to avoid artificially inflating the agreement@k score by having high agreement on non-relevant tokens. We will modify Eq. 1 in the next version so that it is explicit that a 0 score is assigned to non-relevant tokens.\n\nB: You are right that the calculation of mean agreement@k, which averages over sentence-level agreement@k scores, is missing in the equations and only appears in-text. With the extra page in the next version, we will add this third equation.\n\nThank you for the suggestion of including more references to event detection in series. We agree that a more elaborate discussion on event detection would strengthen the paper and we will include studies such as Benkő et al. (2022) and Pang et al. (2021) in the next version. However, the methods outlined in these papers are less suited for event detection in attribution profiles compared to the approach we based our method on, as established by Palshikar et al. (2009). We will explain more clearly in the related work section why that is. In short:\n\n- Benkő et al. (2022) propose a method that finds a unicorn (a single anomaly) that is rare and unique, whereas the type of events described in our study does not meet that description. We are looking for potentially multiple tokens that receive a high signal from attribution methods.\n- The survey by Pang et al. (2021) would help us further define attribution profiles as series; however, DL methods are less suited given the relatively small size of sentences compared to the number of data points in e.g. a stock market series, and given the meaningfulness of the attributions that is only local to the specific sentence.\n\nFinally, thank you for the remaining improvements that you propose. We will take them into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* In depth study of attribution, contradicting few and verifying other studies in the field. \n* Evaluation of attribution in relation with k, a parameter that is overlooked, according to the authors.\n", "reasons_to_reject": "* The equation of agreement@k, which is used to compute most of the results in this study, is not clear. Based on the response of the authors, I will need to revisit my assessment regarding this weakness.\n* The authors overlook existing work in event/anomaly detection (e.g.: https://arxiv.org/abs/2007.02500 or https://www.nature.com/articles/s41598-021-03526-y) and implement a new algorithm to define the optimum k dynamically. \n", "questions_to_authors": "A. The denominator of eq. 2 is always positive. What have I missed?\n\nB. The mean agreement@k is said to be obtained by averaging over sentence-level agreement@k, but relevance depends on attribution methods by definition (nominator of Eq. 1), so where is this averaging computed exactly?\n", "paper_topic_and_main_contributions": "This paper proposes an evaluation measure for feature attribution methods along with a way to define the optimum number of attributing features dynamically. The authors used DistilBERT for NLI, on a single dataset, comparing gradient (four: vanilla or integrated, plain or by multiplying with the input embeddings) and perturbation (two: partition SHAP or LIME) methods. The authors proposed using the ratio of attribution methods comprising the token in question, averaged across the sentence, as a measure of relevance/agreement. By studying different attribution methods and human annotations, the authors conclude that (i) vanilla gradient attribution agrees more with the human annotations compared to the rest; (ii) perturbation methods perform better than other gradient methods; (iii) higher agreement is achieved for higher k. The study also implements a basic event detection algorithm to define the number of k automatically. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Related work regarding event/anomaly detection should be included. To provide and example, two such studies are:\n* https://arxiv.org/abs/2007.02500\n* https://www.nature.com/articles/s41598-021-03526-y"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* L074: what kind of agreement?\n* L090: some intuition about the metric (do you perhaps mean measure?) could be shared\n* L098: please define the agreement; human-vs-method or something else? \n* L127: please consider renaming model to backbone model, in order to distinguish from attribution models\n* L127: the motivation for why DistilBERT was selected (e.g. , instead of BERT) is missing\n* Eq.1 & Eq.1: it is relevance that depends on k, not agreement while the latter is rather an average - hence, my understanding is that it is relevance@k and average-relevance (or sentence-relevance). \n * L180: please provide a figure (e.g., in the appendix) where the legend is more informative and each curve has a name.\n* L236: please define surrounding; one, two, three tokens?\n* Figure 3: this is an interesting figure, but please consider re-producing by using a consistent color scale"}}]

Author rebuttal:

Meta-review: The authors propose a method which determines the optimal number of tokens (k) which should be displayed to humans based on feature importance attribution scores. The authors perform their experiments on six explainability methods (SHAP, LIME, vanilla-grad, int-grad, the latter two in the plain & gradXinput form) on one model (DistilBERT) and one dataset (eSNLI). The authors show that using their dynamic method for determining k leads to better agreement and avoids the sentence-length bias.

The reviewers agree that the paper is well written, the gap is well motivated and the problem is relevant (Mndj, HqLk, Cz9e). Furthermore, the reviewers commend an interesting analysis which produces valuable insights and answers a relevant question (Mndj, HqLk, EnSP, Cz9e).

There are relatively few criticisms of the paper in the reviews, with two reviewers even noting they cannot find any reason to reject the paper (Cz9e, HqLk) barring perhaps the narrow experimental scope (Cz9e). The main criticisms were omission of some related work (EnSP9), unclarity within an equation (EnSP) and a request for a few more illustrative examples (Mndj). All of these criticisms were addressed in the discussion period, and the authors have either resolved them through comments or committed to include additional material given an extra page.
Decision: Accept-Main