An Attribution Method for Siamese Encoders
Authors: Lucas Moeller
Dmitry Nikolaev
Sebastian Padó
Venue: EMNLP 2023 Main (2023)
Keywords: feature attribution
interpretability
explainability
siamese encoder
sentence transformer
integrated gradients
integrated Jacobians
TL;DR: This paper derives a local attribution method for siamese encoders by generalizing the principle of integrated gradients to models receiving multiple inputs.

Abstract: Despite the success of Siamese encoder models such as sentence
transformers (ST), little is known about the aspects of inputs they
pay attention to.  A barrier is that their predictions cannot be
attributed to individual features, as they compare two inputs rather
than processing a single one.
This paper derives a local attribution method for Siamese encoders by generalizing
the principle of integrated gradients to models with multiple inputs.
The output takes the form of feature-pair attributions and in case of STs it can be reduced to a token--token matrix. 
Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model's full computation graph and is guaranteed to converge to the actual prediction.
A pilot study shows that in case of STs few token pairs can dominate
predictions and that STs preferentially focus on nouns and verbs.
For accurate predictions, however, they need to attend to the majority of tokens and parts of speech.

An Attribution Method for Siamese Encoders
Lucas Möller and Dmitry Nikolaev and Sebastian Padó
Institute for Natural Language Processing, University of Stuttgart, Germany Slucas.moelles dmitry.nikolaev, pado] | @ @ims.uni-stuttaar..de
Abstract
et al. (2022) analyze IR models with samples con-
sisting of queries and contrastive documents that differ in certain aspects. Opitz and Frank (2022)
Despite the success of Siamese encoder mod- els such as sentence transformers (ST), little is known about the aspects of inputs they pay at- tention A barrier is that their predictions can- not be attributed to individual features, as they compare two inputs rather than processing a sin- gle one. This paper derives local attribution method for Siamese encoders by generalizing the principle of integrated gradients models with multiple inputs. The output takes the form feature-pat attributions and in case STs it can be reduced token-token matrix. Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model's full computation graph and is guaranteed to converge to the actual prediction. A pilot study shows that in case of STs few token pairs can dominate predictions and that STs preferentially focus nouns and verbs. For accurate predictions, however, they need attend to majority tokens and parts of speech.
train an ST to explicitly encode AMR- prop- erties in its sub-embeddings.
More is known about the behavior of standard transformer models; see Rogers et al. (2020) for overview. Hidden representations have been probed for syntactic and semantic information (Ten- ney et al., 2019; Conia and Navigli, 2022; Jawahar et al., 2019). Attention weights have been ana- lyzed with regard to linguistic patterns they capture (Clark et al., 2019; Voita et al., 2019) and have been linked to individual predictions (Abnar and Zuidema, 2020; Vig, 2019). However, attention weights alone cannot dictions (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). To obtain local explanations for indi- vidual predictions (Li et al., 2016), Bastings and Fil- ippova (2020) suggest the use feature attribution methods (Danilevsky et al., 2020). Among them, integrated gradients are arguably the best choice due to their strong theoretic foundation (Sundarara- jan et al., 2017; Atanasova et al., 2020) (see Ap- pendix A). However, such methods not directly applicable to Siamese models, which compare inputs instead of processing a single one.
as explanations for pre-
1 Introduction
Siamese encoder models (SE) process two inputs concurrently and map them onto single scalar output. One realization are sentence transformers (ST), which learn predict similarity judgment between two texts. They have lead to remarkable improvements in many areas including sentence classification and semantic similarity (Reimers and Gurevych, 2019), information retrieval (IR) (Thakur et al., 2021) and automated grading (Bexte et al., 2022). However, little is known about aspects of inputs that these models hase their decisions on, which limits our understanding of their capabilities and limits.
In this work, we derive attributions for an SE's predictions to its inputs. The result takes the form of pair-wise attributions to features from the two inputs. For the case of STs it can be reduced to token-token matrix (Fig. 1). Our method takes into account the model's full computational graph and only requires it be differentiable. The com- bined prediction of all attributions is theoretically guaranteed to converge against the actual predic- tion. To the best of our knowledge, we propose the first method that can accurately attribute pre- dictions of Siamese models to input features. Our code is publicly available.'
Nikolaev and Padó (2023) analyze STs with sen- tences of pre-defined lexical and syntactic structure and use regression analysis to determine the relative importance of different text properties. MacAvaney
'tttps: / thub. com/lucasmmlr//xbbrr

---

2 Method
CLS
2.1 Feature-Pair Attributions
the
coffee
Let f be Siamese model with an encoder e which maps two inputs and b to a scalar score
0.0
bad
S(a, b) e (a) =
-0.1
EOS
-0.2
Additionally, let be reference inputs that always result in a score of zero for any other input c: f(r, c) = 0. We extend the principle that Sun- dararajan et al. (2017) introduced for singlc-input models (Appendix A) to the following ansatz for two-input models, and reformulate it as integral:
CLS
this
65
not
a
good
coffee
EOS
Figure I: An example token-token attribution matrix to layer nine. The model correctly relates not... good to bad and matches coffee. Similarity score: 0.82, attribu- tion error: 10-3 for N = 500.
f(a, b) fa) f(b,rt) + rb).
Intuitively, Eq. embeds all inputs between ra and a along the path x(a) and computes their sensi- (2) tivities w.r.t. input dimensions (Samek et al., 2017). It then collects all results on the path and com- bines them into the matrix Jai analogously for J.. Eq. combines the sensitivities of both inputs and computes pairwise attributions between all feature combinations a and
b
02
Jr dr,dyy I (x, dx; dyj
Σ (a-rro) ro); (JrJ (b-rb), rb),
This Ansatz is entirely general any model with two inputs. In the last line, we then make explicit of the Siamese architecture to derive the final attributions (details in Appendix B). Indices and j are for dimensions of the two inputs a and b, respectively. Individual summands on the right- hand-side can be expressed in an attribution matrix, which we will refer to as Aij-
In transformer model, text representations are typically of shape S x D, where S is the sequence length and D is the embedding dimensionality. Therefore, A quickly becomes intractably large. Fortunately, the sum Eq. allows us combine individual attributions. Summing over the embed- ding dimension D yields a matrix of shape Sa x Sb. the lengths of the two input sequences. Figure shows an example.
By construction, all terms involving reference input on the sft-hand-siee vanish, and the sum over this attribution matrix is exactly equal to the model prediction:
Since Eq. is an equality, the attributions pro- vided by A are provably correct and we can say that they faithfully explain which aspects of the inputs the model regards as important for a given prediction. For efficient numerical calculation, we approximate the integral by a sum of N steps cor- responding to equally spaced points an along the integration path (Eq. 4). The resulting approxima- tion error is guaranteed to converge zero the sum converges against the integral. is further per- fectly quantifiable by taking the difference between the left- and right-hand side Eq. 3 (cf.) 3.2).
f(a.b) = A
(3)
In the above result, we define the matrices as:
f'd δen(x(())
(Ja)ss
da
Jazi
Ox;
(4)
N
de(((((nn)
N
0x:
==1
The expression inside the integral, dez/Ox,. is the Jacobian of the encoder, i.e. the matrix of partial derivatives of all embedding components & w.r.t. all input components i. We therefore, call J integrated Jacobian. The integral proceeds along positions a on an integration path formed by the linear interpolation between the reference ra and input a: x(a) - -r, + txxx-xx
2.2 Adapting Existing Models
For attributions to take the form of Eq. 3, we need to adapt standard models in aspects:
Reference input. It is crucial that consistently yields a score of zero for inputs involving refer- ence r. A solution would be to set r to an input

---

that the encoder maps onto the zero vector, so that (c, e² (c) e(r) e² (c) O. However, is not trivial to find such an input. We avoid this issue by choosing an arbitrary reference and shift- ing all embeddings by r in the embedding space, e(c) = e'(c) e' (r)), where e' is the original en- coder, so e(r) O. For simplicity, we use a se- quence of padding tokens with the same length as the respective input as reference
Base model
adjusted
cosine
dot
85.9
82.6
S-MPNet
X
87.6
83.9
J
85.7
80.7
S-distilRoBERTa
X
86.3
77.4
85.1
80.4
MPNet
86.3
84.2
80.4
73.4
distillRoBRRTa
84.6
76.2
Similarity measure. Sentence transformers typi- cally use cosine distance to compare embeddings, normalizing them to unit length. Unfortunately, normalization of the zero vector, which we map the reference to, is undefined. Therefore, we re- place cosine distance with the (unnormalized) dot product when computing scores as shown in Eq. 1.
77.7
68.8
RoBERTa
X
86.1
68.8
Table 1: Spearman correlations between labels and scores computed by cosine distance and dot product of embeddings. We evaluate pre-trained sentence trans- formers (top) and vanilla transformers (bottom). Ad- justed indicates modification according Sec. 2.2. Best results for (non-)at Jadjusted models are (underlined) bold.
2.3 Intermediate Representations
Different from other deep models, in transformers, due to the sequence-to-sequeee architecture and the language Lunguage-modeling pre-training, intermediate representations still correspond to (the contexts of) input tokens. Therefore, attributing predictions to inputs is one option, but it is also interesting con- sider attributions to intermediate and even output representations. In these cases, f maps the given intermediate representation to the output. Attribu- tions then explain, which dimensions within this representation the model consults for its prediction.
our adjustments from $ 2.2 applied for the model to obtain exact-attribution ability (ADJUSTED). Train- ing details are provided in Appendix H. For all models, report Spearman correlations between predictions and labels for both cosine distance and dot product of embeddings.
Our main focus is on already pre-trained sen- tence transformers. Results for them are shown in the top half of Table 1. Generally, adjusted models cannot reach the predictive performance of standard STs. However, the best adjusted model (S-MPNet) only performs 1.7 points worse (cosine) than its standard counterpart. This shows that the necessary adjustments the model incur only a modest price terms of downstream performance.
3 Experiments and Results
our experiments, we evaluate the predictive per- formance of different model configurations and then test their attribution accuracy, Generally, the two are independent, so that model with excellent attribution ability may not yield excellent predic- tions or vice versa. In the following, we analyze statistical characteristics of attributions. To demon- strate our method, perform pilot on which parts of speech (POS) models attend to.
The bottom half of the table shows performances for vanilla transformers that have only been pre- trained language modeling tasks. Results for these models are more diverse. However, do not expect their predictions be comparable to STs, and we mostly include them evaluate attribution accuracies on a wider range of models below.
3.1 Predictive Performance
3.2 Attribution Accuracy
We begin by evaluating how much the shift of em- beddings and the change of objective affect the predictive performance of STs. To this end, we fine-tune STs off different pre-trained base models the widely used semantic text similarity (STS) benchmark (Cer et al., 2017) We tune all base mod- els in two different configurations: the standard setting for Siamese sentence transformers (NON- ADJUSTED, Reimers and Gurevych 2019), and with
As shown in $ 2.1, all attributions in A must sum up to the predicted score if the two integrated Ja- cobians approximated well by the sum in Ey. 4. We test how many approximation steps N are quired in practice and compute the absolute error between the sum of attributions and the prediction score as a function N for different intermediate representations. Fig. 2 shows the results for the

---

10c
out
10"
105
11
Absolute approximation
10
Frequency
10
10a
10-2
10a
10-3
10
10a
11
10a
10"
10
10a
10a
1.0
Number of approximation steps (N)
Individuul Token Attributions
Absalute approximation error
Figure 3: Distribution of individual token-token attri- butions to different intermediate representations of the S-MPNet model.
10-1
10-2
-
3
+
10
.7.
butions different (intermediate) representations across 1000 STS test examples. large fraction of all attributions to intermediate representations is negative (38% for layer 11). Thus, the model can balance matches and mismatches. This be- comes apparent in the example in Fig. 4. The word poorly negates the meaning of the sentence and contributes negatively to the prediction. Interest- ingly, attributions the output representation do not capture this characteristic, as they are almost exclusively positive (95%). Other models behave similarly (Appendix E).
10
11
10a
10a
Number approximation steps (N)
Figure 2: Layer-wise attribution errors for the S-MPNet (top) and the RoBERTa based model (bottom). Standard deviations are shown exemplary.
S-MPNet model. Generally, attributions to deeper representations, which are closer to the output, can be approximated with fewer steps. Attributions to c.g. layer 9 are only off by (5=5)x10-3 with as few N 50 approximation steps. Layer 7 requires N 1000 steps reach an error of (2 + 3) 10-3 and errors for shallower layers have not yet started converging for as many as N 2500 steps, this model. In contrast, in the equally deep RoBERTa model, errors for attributions to all layers including input representations have started to converge at this point. The error for attributions to input repre- sentations remains at only (1+1)x10 --viddenlyy, attribution errors are highly model specific.
It further interests us how many feature-pairs the model typically takes into consideration for indi- vidual predictions. We sort attributions by their absolute value and add them up cumulatively. Av- eraging over 1000 test-instances results in Fig. 5. The top 5% of attributions already sum up 177=133)2 of the model prediction. However, the large standard deviation (blue shading in Fig. 5) shows that these top attributions alone do not yet reliably explain predictions for all sentence pairs. For trustworthy prediction with standard devi- ation below 5% (2%), the model requires at least 78% (92%) of all feature-pairs.
Our current implementation and resources limit us to N < 2500. However, we emphasize that this is not a fundamental limit. The sum in Equa- tion 4 converges against the integral for large N, thus it is only matter of computational power to achieve accurate attributions to shallow layers in any model.
3.4 POS Relations
We evaluate which combinations of POS the model relies on compute similarities between sentences. For this purpose, we combine token- word- attributions by averaging. We then tag words with a 3
3.3 Distribution of Attributions
Fig. shows shares of the ten most frequent POS- relations among the highest 1 10%, 25% and 50%
For an overview of the range of attributions that our best test-performing model S-MPNet assigns to pairs of tokens, Fig. 3 shows a histogram of attri-
"cumulative sums top attributions can negative. http:///uuginggaaooo.ooottr////e

---

output layer
layer 11
layer
CLS
this
0.1
concept
general
-..0
1285
poorly
-0.1
EOS
CLS
the shown principle general
12e5
to other areas
Eos
CLS the shown principle generak- Izes
to other areas
EO5
CLS the shown principle generat
Izes
to other areas
EOs
Figure Attributions same example to different representations the S-MPNet model.
NN: Noun
top 10%
Fraction of Pradiction
1.2
Fraction Actributions
0.20
VB: Verb
PT: Punctuation
top 25%
top 50%
1.0
DT: Determiner
0.15
Il: Adjective
0.8
Preposition
0.6
0.10
PR: Pronoun
0.05
0.2
20
40
NN-NN
VB-V8
NN-V8
PT-PT
DTNN
JINN
IN-VB
IN-NN
DT-DT
JH]
PR-PR
Number of Attributions
Figure 5: Mean cumulative prediction and standard- deviation of token-token attributions sorted by their absolute value.
POS Relation
Figure 6: Distribution of the highest 10%, 25% and 50%
attributions among the most attributed parts speech.
4 Conclusion
of attributions the STS test set. Within the top 10%, noun-noun attributions clearly dominate with a share of almost 25%, followed by verb-verb and noun-verb attributions. Among the top 25% this trend is mitigated, the top half splits more evenly. When we compute predictions exclusively from attributions to specific POS-relations, nouns and verbs together explain (53 + 90)% %. and the top ten POS-relations (ef. Fig. 6) account for (66 + 98)% of the model prediction. The 90% most important relations achieve (95 + 29)%. Thus, the model largely relies on nouns (and verbs) for its predic- tions. This extends the analysis of Nikolaev and Padó (2023), who find in study on synthetic data that SBERT similarity is determined primarily by the lexical identities of arguments (subjects / ob- jects) and predicates of matrix clauses. Our find- ings show that this picture extends largely to ralistic data, but that it is ultimately too simplistic: on the STS corpus, the model does look beyond nouns and verbs, taking other parts of speech into account to make predictions.
Our method can provably and accurately attributc Siamese model predictions input and intermedi- feature-pairs, While in sentence transformers output attributions are not very expressive and tributing to inputs can be computationally expen- sive, attributions to deeper intermediate represen- tations are efficient to compute and provide rich insights.
Referring to the terminology introduced by Doshi-Veles and Kim (2017) our feature-pair attri- butions are single cognitive chunks that combine additively the model prediction. Importantly, they can explain which feature-pait are relevant to individual predictions, but not why (Lipton, 2018).
Improvements may be achieved by incorporating the discretization method of Sanyal and Ren (2021), and care must be applied regarding the possibility of adversarially misleading gradients (Wang et al., 2020). In the future, we believe our method can serve as a diagnostic tool to better analyze the pre- dictions of Siamese models.

---

Limitations
Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NIP, pages 149-155, Online. Association for Com- putational Linguistics.
The most important limitation of our method is the fact that the original model needs to be adjusted and fine-tuned in order to adopt to the shift of embed- dings and change of objective that we introduced in Section 2.2. This step is required because the dot-product (and cosine-similarity of shifted em- beddings does not equal that of the original ones." Therefore, we cannot directly analyze off shellf models.
Marie Bexte, Andrea Horbach, and Torsten Zesch. 2022. Similarity-basse content scoring how to make S- BERT keep up with BERT. In Proceedings of the 17th Workshop Innovative Use of NLP for Build- ing Educational Applications (BEA 2022), pages 118- 123, Seattle, Washington. Association for Computa- tional Linguistics.
Second, when a dot-product is used to compare two embeddings instead of a cosine-distance, self- similarity is not preserved: without normalization, the dot-product of an embedding vector with itself is not necessarily one.
Daniel Cer, Mona Diab, Encko Agirre, lnigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2011 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11hh International Workshop on Semantic Evaluation (SemEval-2011) pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Third, our evaluation of predictive performance is limited the task of semantic similarity and the STS benchmark (which includes multiple datasets). This has two reasons: we focus on the derivation of an attribution method for Siamese models and the evaluation of the resulting attributions. The preservation of embedding quality for downstream tasks in non- Siamese settings is out of the scope of this short paper.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neurul Networks for NIP, pages 276-286, Florence, Italy, Association for Com- putational Linguistics.
Simone Conia and Roberto Navigli. 2022. Probing for predicate argument structures in pretrained language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4622-4632, Dublin, Ireland. Association for Computational Linguistics.
Ethics Statement
Our work does not involve sensitive data nor appli- cations. Both, the used pre-trained models and datasets are publicly available. Computational relatively
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yan- nis Katsis, Ban Kawas, and Prithviraj Sen. 2020. A survey of the state of explainable Al for natural lan- guage processing. In Proceedings of the Ist Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 447-459, Suzhou, China, Association for Computational Linguistics.
costs for the required fine-tuning cheap. We believe our method can make Siamese models more transparent and help identify potential errors and biases their predictions.
References
Samira Abnar and Willem Zuidema. 2020. Quantify- ing attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190 On-
Finale Doshi-Velez and Been Kim. 2017. Towards rigorous science of interpretable machine lcarning. 1rXiv:1702.08600
line. Association for Computational Linguistics.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers), pages 3543-3554 3556. Minneapolis, Minnesota. Association for Computational Linguistics.
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classifi- cation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256-3274, Online. Association for Computational Linguistics.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651 3651-3657, Florence, Italy. Association for Computational Linguistics.
Jasmijn Bastings and Katja Filippova. 2020. The ele- phant in the interpretability room: Why use attention explanation when we have saliency methods? In
' (x-cf cf (y - c)= = x² y- x' c- d'y ce x"y (v-c)

---

Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 68 181-691, San Diego, California. Association for Computational Linguisties.
Machine Learning, volume 70 of Proceedings of Ma- chine Learning Research, pages 2319-3328. PMLR.
lan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Zachary C. Lipton. 2018, The mythos of model in- terpretability: In machine learming, the concept of interpretability is both important and slippery. Queue, 16(3):11-57.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Tryna Gurevych. 2021, BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan. 2022. ABNIRML: Analyzing the behavior of neural IR models. Trans- actions of the Association for Computational Linguis- tics, 10:224-233
Jesse Vig. 2019. A multiscale visualization attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37-42, Florence, Italy. Association for Computational Lin- guistics.
Dmitry Nikolaev and Sebastian Padó. 2023, Represen- tation biases in sentence transformers. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 3701-3716, Dubrovnik, Croatia. Association for Computational I Linguistics.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguisties.
Juri Opitz and Anette Frank. 2022. SBERT studies
meaning representations: Decomposing sentence em- beddings into explainable semantic features. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 625-638, Online only. Association for Computational Linguistics.
Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh. 2020. Gradient-based analysis of NI.P mod- els is manipulable. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 247-258, Online, Association for Computational Lin- guistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3992, Hong Kong, China. Association for Com- putational Linguistics.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pages 11 11-20, Hong Kong, China. Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 1842-866.
A Integrated Gradients
Our method builds on the principle that was in- troduced by Sundararajan et al. (2017) for models with single input. Here we derive the core concept of their integrated gradients.
Wojciech Samek, Thomas Wiegand, and -obert Müller. 2017. Explainable artificial intelligence: Un- derstanding, visualizing and interpreting deep learn- ing models. arXiv: 1008, 08296.
Let f be a differentiable model taking a single vector valued input x and producing a scalar output E [o, 1]: f(x) = s. In addition let r be a refer- ence imput yielding a neutral output: f(r) = 0. We can then start from the difference in the two inputs and reformulate it integral (regarding f an anti- inti-derivative):
Soumya Sanyal and Xiang Ren. 2021. Discretized in- tegrated gradients for explaining language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 10285-10299, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Pro- ceedings of the 34th International Conference on
f(a) - ff())) (0(((x)
dx;
(5)
Ox:

---

This path integral from the point in the input space. We use component-wis notation, and double indices are summed over. To solve the integral, we parameterize the path from r to a by the straight line x(o) = r-t a(a r) and substitute it:
(xx(x)) Oxi
dxc foe dex(y) θy, dy;
(10)
Different from above, the encoder is vector- valued function. Therefore, de,(x) is a Jaco- bian, not gradient. We integrate along straight lines from r to a. and from ra to b, parameterized by a and 3, respectively, and receive:
0f(x(a)) (x)(a) da (6) da
(x,
The first term inside the above integral is the gradient of f at the position x(a:). The second term is the derivative of the straight line and reduces to dx(a)/ da (a - r). which is independent of and can be pulled out of the integral:
Des(x((a) do 0x,
=(a (ar)a)
ra)i
dek (y(3)) d3
[ V.f(x(a)) do Ja==
(b
ro)s
= ((a-r))
(7)
B
dy,
(11)
This last expression is the contribution of the 7th input feature to the difference in Equation If f(r) = 0, then the sum over all contributions equals the model prediction f(a) = s. Note, that the equality between Equation 5 and Equation 7 holds strictly. Therefore, Equation 7 is an exact reformulation of the model prediction.
With the definition of integrated Jacobians from Equation 4, we can use vector notation and write the sum over the output dimension & in square brackets as matrix product: Jh. If r consis- tently yields prediction of zero, the last three terms on left-hand-sice of Equation 8 vanish, and we arrive our result in Equation 2, where we denote the sum over input dimensions and j explicitly.
Detailed Derivation
For the case of a model receiving two inputs, extend the ansatz from Equation to:
C Intermediate Attributions
f(a, b) f(a, ra) rb) ) f(ra; rb)
Fig. 4 shows attributions for one example to differ- ent representations in the S-MPNet model. Attribu- tions to layer eleven and seven capture the negative contribution of poorly, which is completely absent in the output layer attributions. As Fig. shows output attributions are less pronounced and almost exclusively positive.
= [[((,, b) f(rai b)] [f(a, rb) f(ra: rb)]
= Imd 0 [s(a,y) - [(ra. dy;
dy;
-p I
b
l 8²
0x,0,; f yx, dx, dy;
(8)
D Attribution Accuracy
We plug in the definition of the Siamese model (Equation 1), using element-wise notation the output embedding dimensions h; and again, omit sums over double indices:
In Fig. 7 we include the attribution accuracy plot for the shallower S-distillRoBERT: model. Attri- butions to all layers converge readily for small N.
E Attribution Distribution
82 Ox,0y, (x(x)) (y) dx; dy,
Fig. 8 shows distribution plots for attributions different intermediate representations of the RoBERTa and the S-distillRoBERTa models. In both cases we also observe positivity of attributions to the output representation. For RoBERTa this characteristic proceeds to the last encoder layers.
(9)
Neither encoding depends on the other integra- tion variable, and we can separate derivatives and integrals:

---

10a
10a
out
Absolute approximation
10a
10-1
10a
-
10a
10a
10-1
10a
109
109
10
10a
0.0
0.1
0.2
0.4
Number of approximation steps (N)
Individual Token Attributions
10a
Figure 7: Layer-wise attribution errors for the distilled
out
Roberta based model
10a
5
104
Frequency
F Different Models
10a
10a
Attributions of different models can characterize differently even agreement on the overall score is good. Fig. shows two examples.
10a
10a
0.1
0.3 Individual Attributions
0.1
G Prediction Failures
Fig. shows examples in which the S-MPNet prediction is far off from the label. the future, a systematic analysis of such cases could provide insights into where the model fails.
Figure Attribution Distributions the RoBERTa- based model (top), and the S-distilloBERTa model (bottom).
Model
Link
H Training Details
S-MPNet
all-mpnet-base-v2
S-distilloBERRTa
all-distirroberta-v
We fine-tune all models in a Siamese setting on the TTS-benchmark train split. Models either use shifted embeddings combined with a dot-product objective or normal embeddings together with cosine objective. All trainings run for five epochs. with batch size of 16, learning rate of 2 10 and weight decay of 0.1 using the AdamW: optimizer. 10% of the training data is used for linear warm-up
MPNet
mpnet-base
distillRoBERTa
distilroberta base
RoBERTa
roberta-bare
Table 2: Links to huggingface weights the used mod- els.
computation. Regarding computational complex- ity, this process hence requires N/ forward and backward passes through the encoder, where is the used batch size. Attributions to intermediate representations do not require the full backward pass and are thus computationally cheaper. Once the two integrated Jacobians derived, the com- putation of the final attribution matrix in the last line of Eq. 8 is matter of matrix multiplication.
I Implementation
This sections intends bridge the gap between the shown theory and its implementation. In Eq. e(x(an)) is single forward pass for the input x(a,, through the encoder e. (e,(x(a,,)/Oxx, is the corresponding backward pass of the 1thh em- bedding dimension w.r.t. the 7th input (or inter- mediate) dimension. In order to calculate either integrated Jacobian, N such passes through the model need be computed for all interpolation steps n N) along the integration paths between references and inputs.
Model Weights
Table 2 includes links to the huggingface model weights that we use in this paper.
N)
Fortunately, they are independent for different in- terpolation steps and we can batch them for parallel

---

S-MPNet (s=0.54)
S-distillRaberta (s=0.45)
Roberta (5=0.88)
CLS
CL5
CLS
yard
yard
yard
has
has
has
dog
dog
dog
EOS
EOS
EOS
CL5
the
dog
running
86
after
another
dog
EDS
CLS
The
dog
running
16
after
another
dog
EDS
CLS
The
dog
running after
is
another
dog
EDS
S-MPNet 0.60)
S-distillRoberta (s=0.59)
Roberta (s= 0.58)
CLS
CLS
the
The
The
ball
ball
ball
er
##ernna
is
dancing
dancing
dancing
EOS
EOS
EOS
CLS
man
dancing
EOs
CLS
man
dancing
EO5
CLS
man
dancing
EOS
Figure Attributions for identical sentences by different models. Model and scores are given in the titles.
error: 0.66
error: 0.70
error: 0.69
CLS
CLS
funeral
some one
you
of
paisley
not
take
any
place
in
EOS
EOS
EOS
CLS
Juneral
m
eeh
#*17
victim
iian
swerney
takes
place
newcadtia EOs
in
CLS
the person
making
blanket
Eos
CLS
you
not
nead
sauce
ab
error: -0.55
error: -0.47
error: -0.53
CLS
CLS
1-
reallzed
think
rather
already
depends
there
accepted
an
on
1s
which
not
but
country
figure
general
answer
my
cents
EOS
EOS
CLS
do not think there
single loltnition
8OS
CLS
kngg
questti
Age
feed
ahon moote
cents
EOs
CLS
that
whtch country
you
EOS
Figure 10: Failure cases of the M-PNet. Examples in the top show estimations, the bottom row shows under estimations of semantic similarity.


Reviews:
[{"review_id": "SzQS1ZeJIF", "reviewer": "Reviewer_PsQ5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback and your deep engagement with our study. In the following, we reply to your most substantial points of criticism.\n\n### Derivation in the Appendix\nPresenting a main result in the main body of a paper and relegating the formal derivation/proof to an appendix has become a relatively standard presentation method (compare Hu et al. NAACL 2022, Mou et al. ACL 2023, etc. etc.). In this way, we can provide the application-oriented reader with a usable model, and the technically oriented readers may consult the appendix for a deeper understanding. If the paper is accepted, we will integrate the most important parts of the derivation in the main body. We would still argue that our size of study is appropriate for a short paper.\nRegarding the typo in Eq. 7, you are absolutely right. The first input to the second term must, of course, be the reference $r$ and not the input $a$. We will fix it.\n\n### Clarity, Detail, Reproducibility\nRegarding your criticism about lack of clarity, we will take them into account when revising the paper. In particular, we acknowledge the suboptimal presentation in Tab 1.\nWe disagree about the lack of detail, though: The STS benchmark is one of the most widely used and known datasets in semantic benchmarking and, in our opinion, does not require an explicit introduction. Similarly, the models we use are standard Huggingface downloads (we will add the specific links). \nWe also do not agree with your comments regarding lack of reproducibility: We have submitted the full code of our implementation and provide all relevant hyper-parameter settings in Appendix H. We would be grateful to know what specific information you feel is missing, and add it. \nAdmittedly, the implementation of our method is not trivial. We could add a paragraph about central implementation concepts in order to close the gap between theory and code. \n\n### Vanilla transformers\nIt is a misunderstanding that our method does not achieve good results on vanilla transformers that have only been pre-trained on language modeling. They do not show competitive predictive performance for semantic similarity, but this is entirely expected as they were never trained on semantic similarity before. They also do not adapt to the change of objective as well (Roberta being an extreme case). But this instability is also very likely due to the lack of large scale pre-training for semantic similarity.\nThese models, however, **do have excellent attribution accuracy**. This is described in Section 3.1 and Appendix D.\nIn this work they are really of secondary interest, because they are not trained to predict similarities. We mostly include them to evaluate attribution accuracies on a wider range of models. We will make this point clearer in a revised version.\n\n### Gradient Stability\nIt is a reasonable hypothesis that gradients may become unstable due to the fact that prediction scores are not naturally bound to [0, 1] by the dot-product. However, in practice we do not experience any issues in this regard. Training runs converge smoothly, and the model adjusts its predictions to the required range well. We believe the drop in predictive performance compared to a cosine objective is largely due to the fact that a dot-product does not preserve self-similarity (as discussed in the limitations section).\n\n### Applicability to shallow layers\nYou conclude that the method can only be used with the highest layers or in shallow networks. Generally it is true that earlier layers require more approximation steps – however, this is not a *conceptual* problem but a *practical* problem of approximation. Theoretically, the attribution error is guaranteed to converge to zero for sufficiently large $N$, because the sum in Eq. 3 will converge against the integral (by definition of the Riemann integral). In Fig. 2 we present the attribution error of the S-MPNet model, which has the best predictive performance (cf. Table 1), but does not yet have good attribution accuracy for shallow layers with $N<2500$. This may have been misleading; Fig 7 shows that other models converge much more readily.\nThe only reason why we cannot run our method with a higher number of approximation steps $N$ is lack of computational resources. However, our code does support arbitrary large $N$ and could be run on a more powerful setup to achieve better attribution accuracy.\nFinally, we would like to emphasize that attributions to intermediate layers do already provide meaningful insights and will be a good compromise to run large scale analyses.\n\n### Conclusion\nWe apologize for the apparent misunderstandings arising from the first paragraph of the conclusions. First, we never claim that our method is computationally less intensive than other methods. We only claim that its attributions are provably correct (line 127-129, cf. Faithfulness - reviewer 4954). Second, to the very best of our knowledge, there exist no other interpretability methods for bi-encoder models. Standard integrated gradients, relevance backpropagation, or attention visualization may provide feature importance attributions to individual inputs. However, no other method can explain which combination of aspects of the two inputs contributed to their similarity. For this reason, comparison against other methods is difficult. We hope to make these points clearer in a revised version of the manuscript.\n\n### Future Work - Sparsity of Attributions\nYou wonder whether the objective could be modified to encourage sparsity. This is definitely an interesting point. In fact, we were somewhat surprised by how sparse attributions already are (cf. Fig. 4 - few attributions often explain a large fraction of the prediction).\nA priori, the main advantage of our method is that it is provably *faithful* to the behavior of the model, which is at odds with the idea of using it to impose an additional constraint on the model during training – such as sparsity. \nThat being said, it could be possible to use output layer attributions for this regard, which are however the least informative. Utilizing attributions to intermediate layers would be tricky as they require backpropagation themselves.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is promising, and shows how the integrated gradient method is adaptable to different settings.", "reasons_to_reject": "See above for a list of my concerns. To sum up, this paper should have been a long paper. As it is, it lacks clarity, and uses appendices to compensate for its succinctness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an attribution method, in other words a method to study the relation between the global score of a neural network and the input (or intermediate) features, specially tailored for Siamese networks based on the method of integrated gradients.\nTo accommodate with the design of Siamese networks, and to attribute scores to feature interactions across the two input sentences, this method is extended to deal with 'integrated jacobians', and these interactions can then be summarized in a matrix form.\nThese feature interactions can be factorized further by summing over the features corresponding to the same token in order to give token-to-token attributions.\nThis method relies on the computation of integrals, which are approximated as sums over (the differential of) the objective applied to points close to each other,\nthe quality of the approximation depending on the number of intermediate points.\n\n\nThe rest of the paper consists of experiments on the STS-benchmark. Different systems are fine-tuned (MPNet, Roberta, distillRoberta) with the proposed attribution method.\n\nFirst the new objective is tested, showing it can be used in practice with models that have been already trained as sentence encoder. In other words, vanilla pretrained transformer cannot be directly trained and achieve good results.\nAlthough no hypothesis is fathomed by the authors, I suspect that the change in the objective where the score is now unbounded (above and below) could lead to unstable gradient during training.\nThis would justify the use of this method as a fine-tuning procedure, where the search space the gradient wrt the loss function are small.\n\nThen the quality of the approximation is tested, and this is where I was disappointed by the experimental results. It seems that the approximation gap of the integrated jacobian must be really tight, and so this method is unable in practice to use the input features as the abstract and the introduction would let the reader thinks. In practice the attribution method can only be used with the highest layers, or must be reserved to shallow networks.\n\nThe distribution of the attributions is then studied on token-token pairs directly and also as POS-POS pairs. In each case, a few attributions are responsible for the majority of the final scores but in order to get reliable results, the vast majority of attributions must be taken into account.\nThis might be an avenue for future research: can the objective be modified to encourage sparsity?\n\nOverall, the contribution looks promising but:\n- The experiments are not properly described. For instance neither the models nor the corpora used in the experiments are presented. Moreover, what the Figures or Tables represent are not really explained, and the conclusion are directly given. This all adds up to make readers struggle to fully understand what they are presented with.\n  - For instance, Table 1 is really difficult to understand, because we do not know precisely what the \"labels\" are, nor what the \"unmodified models with identical training\" stand for. The accompanying test (l.163-l.178) does not help here: which ones are the \"vanilla transformers\", what is a \"Siamese setting\"? These expressions could be applied to numerous objects, and in this context it is too vague.\n  - In Section 3.1, l. 185-186, the \"error between the sum of attributions and the score as a function of N\". This is ambiguous since \"as a function of N\" applies to \"the error\" and not to \"the score\". In Figure 2, the legend covers the left part of the graph\n  - In Section 3.2, Figure 3 is too vaguely described (l.208-212): authors need to describe the axes, the plot etc. the cognitive burden adds up again. I fail to understand the meaning of the example in Appendix C (which should be in the main text) to address matches/mismatches. Again, it may be because the lack of explanation\n- I had to read the appendix (A and B) carefully, and the paper is not intelligible otherwise, so to me this material should be in the main paper.\n- The paper refers several times to figures in Appendix, for instance Fig.6 in appendix C in Section 3.2, and they should be incorporated in the main paper.\n- the authors should provide more details about the training setup. Appendix H does not give enough details to reproduce the results (I am not talking about running the code in the supplementary zip file but being able to redo the experiments from the description given in the paper) **EDIT: see discussion, this is irrelevant**\n- I found the first paragraph of the conclusion a bit puzzling. It seems to imply that computing attribution in this setting is less compute intensive than with other methods, especially for deeper layers, but no comparison is performed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I believe there is an error in equation 7 (appendix b): $f(a,y) - f(a,y)$ should be $f(a,y) - f(r,y)$."}, "extra_text": {}}, {"review_id": "6hPfVteSui", "reviewer": "Reviewer_4954", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We respond to the two most substantial aspects.\n\n### Applicability to other architectures\nRegarding your question whether our method could be applied to architectures other than Siamese encoders which deal with multiple inputs, we admit that our current study focuses on Siamese encoders since they are widely used and arguably one of the least interpretable type of model.\nNow, our Ansatz from Eq. 7 is entirely general to any differentiable model with two inputs. In the following, our derivation, however, is specific to the Siamese architecture – the separation of the two integrals in Eq. 9 is ultimately what allows our attributions to take the comparatively simple form of a matrix. Other architectures call for a more general treatment, or will result in another architecture-specific solution.\n\n### Faithfulness\nYou are right that we actually do not explicitly comment on faithfulness, which we should.\nWe refer to attributions being „provably correct“ (line 127-129) because Eq. 2 is an equality. Thus, our attributions faithfully explain which features (or tokens) of the inputs the model attends to, in order to make predictions. This is different from e.g. attention visualization as it does not account for the full computational graph of the model. In lines 271-273 we conclude that our attributions „can explain which feature-pairs are relevant to individual predictions, but not why“. This is because our method *cannot* explain the reasoning within the model — not faithfully and not at all.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is an interesting extension to Sundararajan et al. 2017 for siamese encoders (pair-wise attributions). The paper not only shows why the idea works in theory, but also demonstrate how to apply that in practice using approximation. \n* In terms of theory, it is nice to see how reference input r is chosen. \n* In terms of practice, the paper fairly shows the approximation ability, and that rightly points out the weaknesses of the choices related to input r (table 1). \n\n", "reasons_to_reject": "Although being able to contribute to explainable AI, the paper seems to have narrow applications due to siamese encoders. Could the approach be applied to a wider range of network architectures, e.g. for machine translation?\n\nThe paper doesn't provide evidence whether the computed pair-wise feature attributions faithful or not. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method to measure feature attributions for siamese encoders. The idea is to break out the compatible output score into several components related to input feature pairs, using path integrals proposed by Sundararajan et al 2017. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fCPIqlfK0m", "reviewer": "Reviewer_Ug1E", "summary": "", "questions": "- The method requires evaluating on multiple points along with the line between a, b and r. How will be the computational overhead of applying this method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. In the following we reply to your main concerns.\n\n### Restriction to Seq-to-Seq models\nOur method is applicable to any kind of Siamese model and is not restricted to sequence-to-sequence encoders. In the detailed derivation (Appendix B) this is mirrored in the fact that we only require an abstract formulation of an encoder $e$ and do not need to make any further assumptions about this model component. If $e$ was not a token-based seq-to-seq model the result would be a feature-feature matrix and could not be reduced to a token-token version, but the method would still be applicable as it is. Presumably, a different kind of aggregation by summation over the entries of the attribution matrix could be carried out (cf. lines 118-126). \nAn interesting aspect of transformer models is the fact that all intermediate representations are of the same shape and can be associated with individual input tokens (Section 2.3). We make direct use of this property. However, our method does not depend on it as it can compute attributions to any representation — input or intermediate ones.\n\n### Clarity and detail\nWe agree that Section 3.1 together with Table 1 are currently not reader friendly and acknowledge that all reviewers criticize this part. We will update it in the revised version.\n\n### Computational complexity\nGenerally, our method requires N forward and backward passed through the model to compute attributions ($N$ being the number of approximation steps from Eq. 3). Fortunately, approximation steps are independent of each other and we can compute them in parallel by batching them. Thus, the computational complexity scales linearly by a factor of $N/B$ respective to the complexity of the forward and backward pass (As $B$ is a constant the complexity is $O(N)$). Attributions to intermediate layers do not require the computation of the full backward pass, and are thus cheaper. We will include this information into the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The extension of Sundararajan et al. (2017)'s method into Siamese setting is done naturally and thoroughly\n- Various analyses comparing multiple layers and properties are given.", "reasons_to_reject": "- From my understanding, the method can only be applied to sequence-to-sequence sentence encoder setting, i.e. we have multiple output vectors in token-level\n- More elaboration on experiment settings will be thankful\n  - For me, it's not clear what 1.7 points of sacrifice (line 175) means.", "questions_to_authors": "- The method requires evaluating on multiple points along with the line between a, b and r. How will be the computational overhead of applying this method?", "paper_topic_and_main_contributions": "The work is about understanding which parts of the sentence representations attribute to their classification results, in the Siamese encoder setting (especially in the Siamese transformer (ST) setting).\nThe motivation of this work comes from the observation that unlike cross-sentence attention models which directly compare two sentences as part of model components, in the Siamese setting it is hard to figure out pair-wise attributions of two inputs.\nThe works starts from extending the attribution method designed for single-input models (Sundararajan et al. 2017), where a vector which outputs zero-score is used as a reference for measure attributions.\nSpecifically, the authors rewrite the score function (dot production) with the two displacement vectors and integrated Jacobians.\nBy switching the scoring function from cosine similarity to dot product, to make a reference vector exist, the derivation of the scoring function can be derive expressed with the above terms can be done exactly, up to approximation error introduced by summation instead of exact integration.\nFrom experiments on multiple models and their layer outputs, it is empirically proven that the proposed method exhibits the desired effect, by responding to identical or similar words, or negation words.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "As the authors noted, I don't believe the proposed method introduce additional ethical concerns than the ones originating from benchmark datasets."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper adapts Integrated Gradients to siamese encoders (which can be useful e.g. in tasks that learn similarity judgements for two texts). For this, the authors introduce “integrated jacobians” that summarize feature interactions between the two inputs in a matrix form.

The reviewers agreed that the method and its theoretical justification are solid. They also appreciated that the authors showed how it can be used in practice and discussed the potential weaknesses (e.g., the choice of the baseline input r, etc.) There were concerns about applicability of the method to early layers and other architectures, but those were resolved during discussion. Overall, the reviewers are positive about the paper.
Decision: Accept-Main