Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models
Authors: Sean Xie
Soroush Vosoughi
Saeed Hassanpour
Venue: EMNLP 2023 Findings (2023)
Keywords: Prototypes
NLP
Interpretability
Faithfulness
TL;DR: A white-box framework that offers inherent interpretability for LLMs via interpretable prototypes that can explain model decisions and function as feature extractors on downstream tasks.

Abstract: Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), but their lack of interpretability has been a major concern. Current methods for interpreting LLMs are post hoc, applied after inference time, and have limitations such as their focus on low-level features and lack of explainability at higher-level text units. In this work, we introduce proto-lm, a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings during the fine-tuning stage while maintaining competitive performance. Our method's applicability and interpretability are demonstrated through experiments on a wide range of NLP tasks, and our results indicate a new possibility of creating interpretable models without sacrificing performance. This novel approach to interpretability in LLMs can pave the way for more interpretable models without the need to sacrifice performance. We release our code at https://github.com/yx131/proto-lm}{https://github.com/yx131/proto-lm.

Proto-Im: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models
Sean Xie
Soroush Vosoughi"
Saeed Hassanpour"
Department Computer Science
Department Computer Science
Department of Biomedical Data Science
Dartmouth College
Dartmouth College
Dartmouth College
sean.xie xie. grwearrrddrrrre trouth edu
soroush. wow.hhoooouggghdrrruoo th, edu
saeed. hawsanpouredar tmouth. Seruutheeeee edu
Abstract
from the original decision-making process of the model (Shrikumar et al., 2016, 2017; Springenherg et al., 2014). These limitations add extra compu- tational complexity to the explanation process and can result in approximate explanations that are un- faithful' to the original model's decision-making process (Sun et al., 2020; DeYoung et al., 2019) Finally, current interpretability methods focus attributing importance to different words in the input and do not explain the model's decision at the sentence or sample level. (Sundararajan et al., 2017; Vaswani et al., 2017; Murdoch et al., 2018).
Large Language Models (LLMs) have sig- nificantly advanced the field of Natural Lan- guage Processing (NLP), but their lack of in- lerpretability has been major concern. Cur- rent methods for interpreting LLMs are post hoc, applied after inference time, and have limitations such as their focus on low-level features and lack of explainability at higher- level text units. In this work, we introduce proto-lm, prototypical network-based white- box framework that allows LLMSs to learn im- mediately interpretable embeddings during the fine-tune fine-tuning stage while maintaining competi- tive performance. Our method's 's applicability and interpretability are demonstrated through experiments wide range of NLP tasks, and our results indicate a new possibility of creating interpretable models without sacrific- ing performance. This novel approach to in- terpretability in LLMS can pave the way for more interpretable models without the need to sacrifice performance. We release our code at http://github..om/xll.m 1protto-mm.
To address these challenges, propose novel framework that utilizes prototypical network learn an interpretable prototypical embedding layer on top of fine-tuned LLM, which can be trained end-to-end for a downstream task. Our framework utilizes trainable parameters, called prototypes, to both perform the downstream task by capturing important features from the training dataset and provide explanations for the model's decisions via projection onto the most influential training exam- ples. Additionally, we utilize a token-level atten- tion layer before the prototypical layer to select relevant parts within each input text, allowing our model to attribute importance to individual words like existing interpretability methods. Our frame- work, named proto-Im, 1m, achieves competitive re- sults on wide range of NLP tasks and offers in- terpretability while addressing the previously men- tioned limitations of current interpretability meth- ods.
1 Introduction
recent years, Large Language Models (LLMs) have significantly improved results on a wide range of Natural Language Processing (NLP) tasks. However, despite their state-of-the-art perfor- mance, LLMs, such BERT (Devlin al., 2018), RoBERTa (Liu et al., 2019) and BART (Lewis et al., 2019), are not easily interpretable. Interpretability is a crucial aspect of language models, especially LLMs, as it enables trust and adoption in various domains. To address this issue, there is a grow- ing interest in improving model interpretability for LLMs and neural models in general.
Fig. 1 shows our proposed proto- 1m framework applied to custiiclassiiiiaaiion example from the SST5 dataset (Socher et al., 2013). As the figure illustrates, the model's decision decision-making process is simple, transparent, and accurate as it classifies the input as "Very Positive" based on its highest
Current interpretation methods have several lim- itations, such as requiring a surrogate model to be built (Ribeiro al., 2016; Lundberg and Lee, 2017) or being applied post-hoe to each instance, separate
'Fithfulness is defined as how accurately the explanation reflects the decision-making process of the model (Jacovi and Goldberg, 2020).
"Co-corresponding Authors.

---

Irput Shorr, looel, fanery and metely vottring lim.
Prototypical Layer
Incoder
hariibl crek
Similanty
Very Negative Protetypos
Sharph liwely funny and ultimtely sobeeing Rin
the film stricly routine
Token lovel Attention
Similarity
Nogative Prototypes
0,010
perfectly perfectly bland
Vary Positive
Dectaon:
Simillrity
Prototypes
0:237
al it pretty good exccution of story
Sinillartt
Positive Prototypos
0.899
funny, sox, devastating and Incurably romantec
Similanity
Very Postive Prototypes
Figure I: Musstaaiio of the inherently interpretable decision-making process of the proto-Im 1m model for example from the SST5 (5-class ane-grained sentiment analysis) dataset. The figure highlights the words identified by token-level attention, with a stronger shade indicating higher attention score. proto-lm correctly predicts the input relating prototypes of the "Very Positive" class. The decision process of proto-Im is transparent, and no separate interpretability method is required.
similarity to prototypes from the "Very Positive" class and the input's distance (lack of similarity) "Negative" and "Very Negative" prototypes. key advantage our framework its inherent in- terpretability. Our prototypical network does not require an additional surrogate model for expla- nation, and we can observe the exact contribution of each prototype to the final output, resulting in framework faithful explanations. Additionally, offers interpretability at both the token and sample level, allowing for the identification of important words the input text as well as the attribution importance to impactful samples in the training data. In this work, we make the following contribu- tions:
2 proto-lm
The architecture of our proposed framework, proto-lm, shown Figure 2. We utilize a pre- trained LLM as the underlying language model to encode the input and build prototypical layer on top of it to provide interpretable prototypes. The goal of the learned prototypes is to capture features representative enough of each class that they can be fine-tuned for the downstream classification task. In the classification process, similarities be- tween the encoded input and the learned prototypes are fed through a fully-comnected layer to produce logits. We formalize this below.
2.1 Token-level Attention
Let represent single input text and ys its as- sociated label. We denote f(æ;) as the encoding of x; by an underlying LLM, f. We pass f(x) through token-level attention layer that learns to emphasize different parts of the text, allowing identify important words within each sample. We first calculate th,, the output of a fully-connected layer with bias (*) ), applied to ht, which the em- bedding of each token t in f(x;), followed by a tanh activation function (a). We then compute Vt, the dot product of iz and token-level weight vector W,.
We introduce a novel framework, proto-lm based on prototypical networks that provides in- herent interpretability to LLMSs.
We demonstrate proto-lm's applicability on three LLMs and show its competitive perfor- mance on a wide range of NLP tasks.
We conduct ablation studies to demonstrate im- portant characteristics of proto-1m under differ- ent hyperparameters.
We evaluate the interpretability of proto-Im 1m un- der several desiderata and show that the explana- tions provided by proto-lm are of high quality.
24 = (bb(h))
14. = U4 W.
(1)

---

Pr(neg) 0.01
(ppps000.99
2i in the prototypical space. Each dimension of M; can be interpreted as the similarity between M, and a prototype. Subsequently, M, is fed through a fully- layer W₁ of dimension N x |C] to obtain logits for each class. Let W. denote the set of weights in Wh. that connect similarities in M, to the logit of class Our model's output probabili- ties are computed as follows:
Cussificction Layer
Sontma
Nermalization
Similerity Come onncttentionn
Aalloynn Aeveeee apper
Tmiratble Trinstio Parametrrs
Prolotypoo
exp(M.we)
Similarity Calculation
Pr(ý - e |2x))
(3)
exp(M
Sommation
Tovernilene Altention
eEC
2.3 Training Objective
Saltmas
To create more representative prototypes and shape the clusters of prototypes in the prototypical em- bedding space, we introduce cohesion loss term Leoh and separation loss term Ssep into loss function in addition the eross entropy loss Leo. Let K be an integer hyperparameter, Pj denote a singular prototype, and Py, represent all prototypes in P that belong to class yis our loss terms are defined as follows:
17
Phanintte rozen
Base LIM
Fine-tured Base Large Language Model (BERT/RoSERTa/BART)
Nothing
less
than
cinematic masterpiece
Fi
Figure 2: The protoypical network-based architecture of proto-lm.
Looh
1
Σ Visp,Ess
max||S: py12 K
(4)
K
We calculate the attention score a₁ for each token using the softmax function. The attended encod- ing of fI;), S,, is then obtained by taking the weighted sum of the token embeddings, where the weight for each token is determined by its attention score.
L Lee
1 K
Σ VirjEE,
min S: p31
(5)
K
For every Sü in the input batch, Looh penalizes the average distance between S; and the K most dis- tant prototypes of its class (y:) while & sep penalizes the average distance between S; and the K most similar prototypes that do not belong yi- Intu- itively, for every S,, Co "pulls" K prototypes of the correct class close while & sep "pushes" pro- totypes of the incorrect class away. We then add cross-entn cross-entropy loss Loe and weight each loss term with hyperparameter À such that Àg + +1 + X2 to obtain the following loss function:
exp(1t)
S = Σ ogh
(2)
Σ exp(vi) tCf(x)
(E)(a)
2.2 Prototypical Layer
the prototypical layer P, we create N proto- type vectors and assign n prototypes to each class e E C, where C represents the set of all classes in the dataset. To ensure an equal number of proto- type vectors are allocated to capture representative features for each class, we set
Lotttl Ao Lee + A1 Leoh + d2 Sep
(6)
N where |C|
2.4 Prototype Projection
[
denotes the total number of classes in the dataset. The input St then fed into P. where we calcu- late the similarity between S, and every prototype vector p E P by inverting the Tæz-distance. We then concatenate all N similarities to obtain the vector M,, which serves as the embedding of input
To understand cach prototype vector pj in natural language, we project each prototype onto the near- token-level attended encoding (S,) of a sample text (xy)) in the training data that belongs to the same class as pj and assign the token-attended text of that prototype. Let D the training

---

dataset. We formally denote the projection as in eq.7: ((z),))) E D such that By Text of Pj + argmin ||, Pyll 8)
us visualize examples in prototypical space, where each dimension represents the normalized similarity E [0, 1] between an example and one prototype.
In Fig. 3, select positive and one neg- ative prototype from the prototypical layer of the model create a 2-dimensional space in which we place examples #1-#4. The vertical axis repre- sents similarity to the negative prototype, and the horizontal axis represents similarity to the positive prototype. From this, we can see that example #1 is much more similar to the negative prototype than to the positive prototype, and example #3 is much more similar to the positive prototype than the negative prototype, and both are correctly classified as a result.
In other words, for each prototype Pj. we find the nearest Sj in the training dataset that belongs to the same class as Py and assign the corresponding token-attended text to Pj (Details in App. D & E),
3 Performance Experiments
As we do not want interpretabilityy to come at the cost of performance, periments to assess the modeling capability of proto-lm. We implement proto-Im 1m using BERT (base-uncased and large-uncased), RoBERTa (base and large) and BaRT-large as the base LLM en- coders and train the token-level attention module, prototypical layer and classification head as de- scribed in $2. We evaluate the predictive accu- racy of proto-lm on 5 tasks (SST-2, QNLI, MNLI, WNLL, RTE) from the GLUE dataset (Wang et al., 2018), IMDb sentiment analysis (Maas al., 2011) and SST-5 (Socher et al., 2013). For baselines, we use the same fine-tuned versions of the LLMM with a classification head. We tune all hyperparame- ters using the respective validation data in each dataset. We present the mean performance as well the standard deviation over 5 runs under their re- spective optimal configurations of hyperparameters (cf. App.A) and compare them their baselines in Table 1. Across our experiments 2, we observe that proto-Im 1m either closely matches or exceeds the performance of its baseline LLM, proving that there is not a trade-off between proto-lm's inter- pretability and performance.
first conduct ex-
From the prototypical space, we can see clearly that the model's decision to misclassify example #2 positive is due to elements in the example that make it similar to the positive prototype. Sim- ilarly, the prototypical space of proto reveals that example #4 was misclassified because, while it contains elements that are similar to both the positive and negative prototypes, it is closer to the negative prototype. The interpretable prototypi- cal space of proto-lm provides explanation for why difficult cases such as examples #2 and #4 are misclassified. should be noted that while similar analyses and explanations can be obtained through post-hoc techniques such as generating sentence embeddings (Reimers and Gurevych, 2019; Gao et al., 2021), proto-lm has this capability built-in.
4.2 Prototype quality and performance
We investigate the effect of different weightings of the loss terms in our loss function. We train proto-lm, holding all other hyperparameters con- stant, under 11 different values of Ao evenly dis- tributed on [0, 1]. and report our results in Figure 4. For these experiments,
4 Prototype Interpretability
4.1 Interpretable prototypical space and decision-making
place equal weight Leoh and Lep such that A = d2 = 1-A0 (ad- ditional details in App. C).
Compared to black-box models such as BERT/ROBERTT/BRRT, which have uninter- pretable embedding dimensions, proto-1m offers the added benefit of interpretability in conjunction with competitive performance. We show example of 2-dimensional prototypical space in Fig. 3. where proto-lm 1m provides insight into the model's 's decision-making process by allowing
Because prototypes not only help interpret the model but also capture the most representative fea- tures from the data, placing an excessive emphasis on Lee actually achieves the adverse effect. As shown in Figure 4, increasing À comes at the cost of learning representative prototypes (which Looh and & eep do) and this is reflected in decreasing clas- sification accuracy on the downstream task. We
Wo conduct additional performance experiments with proto-Im in App.B

---

SST2
QNLI
MNLI
WNLI
RIE
IMDb
SSTS
Prota-InBBER
40.2/92.2
90.5
/902
47,9/46.4
68.1
000913
55.3
0.03/54.9
proto-lwBERRT-inge
98.2 1003944.5
92.4 1003 92.7
36.3
10.00 186.7
47,944..*
76.2
00077011
93.6
L'009220
56.4
0.00/6622
prota- Rell--inRBBERTaHto base
94.0 #0.0/93.6
92.2 2002 /925
86.9
10:0 167.2
36.3/6.34
84.2 000078.777
95.0
200294.7
56.8
0003564
prota-1nRootiRii lirge
96.5 50.0//6.44
94.6
F0.2
90.1
50.00
196.2
56.4/56.4
68.6 000/8666
9538 000556
58.0
proto-InBBART-aagg
97.0 1001/96.6
94.2 1005 M4.9
89.3
10.00 /90.2
89.2 0.08/7..0
Table 1: Predictive accuracy of proto-lm compared against fine-tuned versions of their base LLM's. We observe competitive performance when compared to baselines all tasks, with bolded and underlined values representing better and equivalent performances, respectively. indicates that the performance is result of our experiments on that task and not reported in the original work cited, All results proto-Im are results under the optimal arrangement of hyperparameters for that task, which we obtain through experimentation.
Example (Negative, cllsectly-classifeed::
Example #3 (Positive, perrectly-classffer
Quality entertainment all around...
hurt to watch this movie, it really did... ... the acting and editing was terrible...
there's just not a bad thing to be said. ..whatever you want, it's here...
Example #2 (Negative, mis-classified as positive):
Example #4 (Positive, mis-classified as negative)
If you haven 't enjoyed Van Damme
...this is probably the most pathetic
movie... you probably will not like this...
movie have seen in a while, and yet that what makes it so hilarious... everything that is supposed to sound serious comes out really funny... This movie is bad actually good...
...may not have the best plots... but enjoy these kinds of movies for what they are... the movie only worth watching to Van Damme fans... good fun stuff!
Negative prototype:
2 2-Dimensional Prototypical Space
This is the worst thing the TMNT franchise has ever spawned... It's atrocious...So much so that I didn't 't even remember... it's
Normal ladd similanity to negative prototype
miserably forgettable.
Positive prototype:
0.5
This movie has everything fantasy movie should have... great acting and a fair dose of magic... thoroughly enjoyed this movie and was drawn to its original plot...and colorful characters...
0.5
Normalized similarity positive prototype
Figure 3: Two-dimenne -iveeimeniional prototypical space representation of four test examples, where each dimension represents the normalized similarity between the sample and prototype. A positive prototype is chosen for horizontal axis, and negative prototype for the vertical axis. Examples #1 and #3 are correctly classified, they much more similar to the prototype of their respective ground-truth class. Examples #2 and #4 are misclassified, due to their high similarities the prototype of the incorrect class.
observe that the optimal accuracy is achieved when Ao = 0.3. As we increase A from 0.3, we see not only decline in accuracy but also a decline in the quality of the prototypes associated with the input. On the other, placing sole emphasis ooh and sep without any emphasis on Lie (as in the case À = 0) leads to prototypes that do not help with the downstream task.
4.3 Size of prototypical space
As noted in $4.2, prototypes serve to capture fea- tures from data that are useful for making predic- tions in addition to interpretability. As result, the number of prototypes (N) in the prototypical layer directly affects the expressiveness of our model. We conduct experiments varying the number of pro- totypes N using RoBERTa-Large as the base and show our results in Fig. 5. We observe a general in- crease in accuracy up until N = 1000, after which

---

(pashiich
heartbrrenching performance that will make oven the stamest Cit
proto-Im accuracy IMDs
Performance of proto at different number of prototypes 100
Prototype (Poxdies, Samtartyyyc
.OOT DNly amadiooal pull..
70
PPosiiie
Accuracy
Accuracy
fonchar 2rt svited posteeecii (o cnpharzztee tieeet
60
50
SST2
ONLI
Protatype 1 Dlegatiin Smilarty 0.76
Photobype Posthn, Similrtt di
MNLI
TTe (eedndem wasld decisanng onte bustedaeee 2td
The
WNLI
manemss www.mrrroniiiinnoooooooo sell.. Der Entirracc Jorja.t
IMDb
Protooype (Negative Similarty
INegative
SSTS
Ve of aatudd cat afo Aarric bad-
Ncs inirgt Eaoitdon..
200
400
500
800
1000
1200
1400
Number prototypes (N)
0.1
0.4
0.5
0.6
0.7
0.9
Figure 5: Performance of proto-1m tasks under vary- ing numbers of prototypes in the prototypical layer (N).
Ag)
Figure 4: Predictive accuracy of the proto-lm model with RoBERTa-large as the base LM on the IMDb dataset as a function of An. with N 1400, n = 700, and K 350. We illustrate the closest (most similar) prototypes to the positive input generated by proto-Im when trained under different values λg We observe non-linear relationship between accuracy and λo-
and disassociates with during training, As chang- ing K also influences the decision-making process of the model by altering the number of samples the models compare for each input, examine the impact K on both prototype segregation and model performance. ploy proto-Im with RoBERTa-large as our base on SST2 and IMDb. We set N, the total number of prototypes, to be 1000, and n = 1000/2, the number of prototypes per class, to be 500. We train models under seven different K values, keeping all other hyperparameters constant. We report the curacy of the models and the percentage of unique prototypes in P under each different K in Fig.6.
our experiment, we em-
accuracy plateaus for some tasks while increasing only slightly for others. We reason that increasing N can only improve the model's expressiveness until the point when N reaches the embedding di- mension of the underlying LLM, after which more prototypes in the prototypical space longer aid capturing more useful features from the training data. In Fig. 5's case, as RoBERTa-large's output dimension is 1024, we see the increasing trend of accuracy stop at around 1000 prototypes.
A prototype is considered unique if no other pro- totype in P projects onto the same sample in the dataset. We observe that the best prototype segrega- tion (highest number of unique prototypes) occurs when K 1, and the number of unique prototypes significantly drops as we increase K. Intuitively, if more prototypes are drawn close each sample during training (eq.4), it becomes challenging to create unique prototypes. It is important to note that eq. is less related to the uniqueness of pro- totypes as prototypes are projected onto samples from the same class. We also witness a slight rise in model accuracy
4.4 Prototype uniqueness and performance
The interpretability of proto-lm stems from the retrieval of the most informative training exam- ples. If all prototypes are predominantly associ- ated with limited number of training examples, this reduces their utility from an interpretability perspective. Hence, is beneficial to encourage prototype segregation, that is, a broader projec- tion onto the dataset and more diverse represen- tation of different samples. Besides normalizing prototype distances, which has been shown to in- fluence prototype segregation (Das al., 2022), proto-lm introduces an additional hyperparame- ter, K. This parameter controls the number of prototypes that each training example associates
increase K. We conjecture that while unique prototypes contribute inter- pretability, the model doesn't necessarily require prototypes to be unique to make accurate decisions. Thus, we observe minor trade-off between inter- pretability and model performance.

---

percentage difference in model confidence when p are removed (by setting pf's weights in Wi, are set to O). Specifically, let ÿ be the prediction of a model .M on Iú, let pry. be the output logit of M for it. and let pry. Ppp)) be the output logit when p prototypes are removed from the prototypical layer, we calculate Comp as follows:
Performance protolype uniqueness different
of unique prototypee /Aocurayy
legend
IMDb Acc.
SST2 Unio
Uniq.
Comp Pr pr (P p
(8)
prù;
We analogously calculate Suff as the percentage difference in model confidence when the k% prototypes are retained:
K-1
K=5
K=10
K-250
K-500
Suff = pi pr (pf)
Figure Performance and prototype uniqueness in proto-lm when trained with different K's.
(9)
prg.
We compute Comp and Suff & E 1, 10, 20, 50 and our present mean results across SST2, QNLI, MNLI and SST5 in Fig Our formulation of pro- totype Comp and Suff are inspired by Do et al. (2019). We note here that under this formu- lation, lower sufficiency is better i.c., smaller amount of reduction in model confidence when only salient features are retained is better. As we remove more top k% prototypes, we observe a gen- eral increase Comp. We also note a general decrease in Suff (a lower Suff is preferable) as we retain more top k% prototypes. Moreover, when the bottom k% of prototypes are removed, there are relatively small changes in Comp and large changes in Suff when only the bottom 1% of pro- totypes retained. These trends underscore the influence of the learned prototypes on the model's decision-making process and their faithfulness.
5 Evaluating the Interpretability of proto-1m
Through the usage of its prototypical space, proto-Im is white-box, inherently interpretable model. But just how well do the explanations pro- vided by proto-1m satisfy common desiderata in interpretability? We conduct experiments to try to answer that question in this section. Specifically, we evaluate the inherent interpretability provided by proto-lm 1m via measures of faithfulness (De oung et al., 2019; Jacovi and Goldberg, 2020) and simulatability (Pruthi et al., 2022; Fernandes et al., 2022).
5.1 Faithfulness experiments
Recently, the concept of faithfulness, which mea- sures the extent to which an explanation accu- rately reflects the true decision-making process of a model, has garnered attention as a criterion for evaluating explainability methods (Jacovi and Goldberg, 2020, 2021; Chan et al., 2022; Dasgupta et al., 2022). For textual inputs, faithfulness is con- cretely evaluated using the metries of comprehen- siveness (Comp) and sufficiency (Suff), defined by Younu et al. (2019). Specifically, Comp and Suff quantify the reduction in the model's confi- dence for its output when salient features are re- moved and retained, respectively.
5.2 Simulatability experiments
Simulatability refers to the capacity of a human mimic or replicate the decision-making process of a machine learning model (Doshi- Velez and Kim, 2017; Pruthi et al., 2022). It is a desirable attribute as it inherently aligns with the goal of transparently communicating the model's 's underlying behavior to human users (Fernandes et al., 2022). We assess the simulatability of proto-Im by providing human annotators with various explanations of model out- puts on SST2/QNLI and recording the percentage of model outputs that the annotators can replicate. We provide explanations under the following four settings:
We extend the application of Comp and Suff to prototypical networks assess the faithfulness of proto-1m. For an input 2;, we initially identify the top 4% of most similar prototypes and the bottom 5% of least similar prototypes. Let p\ denote the 4% prototypes identified, we compute Comp as the
No Explanations (NE): Annotators are pre- sented with only the sample data, without any explanations, and asked to make a decision.

---

Reemoiing (for Comp) Retaiinnn top kis of protolypes
tfor Compl ifor Suff; bottom k% of prototypes
Remowing Compl Sufft
random Kioo of prototypes
Prototype comprehensiveness
Prototype sufficiency
100
Comprehensivenees
80
80
Sufficiency
60
60
40
20
20
k
196
5%%
k- 10%
20% k- 50%
k
1%
k- 5%
k- 10% k 20% 50%
Figure 7: The comprehensiveness and sufficiency of proto-lm with k% prototypes removed.
Simultability with different explanations
Accuracy with different explanations
1.0
1.0
Simulatabiitty
0.8
0.8
Accurscy
0.4
0.4
0.0
0.0
Output Figure 8: Simulatability and accuracy of human annotators under different settings provided explanations from proto-Im.
Output
Random Explanations (Random): Each sample in the dataset is presented along with prototypes from proto-Im chosen randomly as explanations. This setting serves as our baseline.
attribute this the fact that the models are not en- tirely accurate themselves, and instances where the model is inaccurate, presenting the output leads annotators to replicate inaccurate decisions.
Prototypical Explanations (PE): Each sample in the dataset is presented along with the top prototypes most similar the sample when proto-lm made its decision.
Furthermore, we compare proto-lm's simulata- bility against 3 other well-know interpretability methods: LIME, Integrated Gradients, and Guided Backprop. We employed three workers for each example and reported the mean percentage of ex- amples where the workers correctly replicated the model's decision. For an example of the a simu- latability questionnaire with PE, see Fig. For an example of an accuracy questionnaire with Ran- dom explanations, the questionnaire for LIME/IG/GB explanations, see Fig. 14. For results of proto-lm against LIME, IG and GB, please see Table Our addditional results further indicate that proto-1m allow the workers to replicate the model's decisions better than all the other interpretability methods on both tasks.
PE Output: In addition the prototypical explanations, the model decision for each sample is also presented.
We employ workers from Amazon Mechanical Turk to crowdsource our evaluations and presented 3 workers with 50 examples each from the SST2 and QNLI datasets, along with explanations for the model's decisions in the settings mentioned $5.2. We use the best performing models for SST2 and QNLI (those presented Table 1), since previous studies found that the utility of PE's are reliant on model accuracy (Das et al., 2022). Additionally, we assess the accuracy of the human annotators in relation to the ground truth labels. We present results for both in Fig. 8. The results show that the case-based reasoning explanations offered by PEs more effective assisting annotators in simulating model predictions than the random base- line. We also notice a minor drop in accuracy when we provide PE output compared to just PE. We
Fig. 13. For example of
Related Works
Prototypical networks (Snell et al., 2017) have shown strong performance in few-shot image and text classification tasks (Sun al., 2019; Gao al., 2019). However, these approaches do not actively learn prototypes and instead rely on summarizing

---

SST-2
iting prototype diversity and representation power. Friedrich al. (2021) counteracts this potential limitation by introducing an additional diversity loss term. proto-lm, in contrast, ensures proto- 84.3%
QNLI
Random Explanations
42.3%
48.7%
LIME
87.3%
90.3%
Integrated Gradient
84.7%
Guided Backprop
78.0%
88.0%
type diversity by leveraging the K hyperparameter, which we delve into in section $4.4.
Prototype explanations
from proto-lm
90.0%
92.0%
7 Conclusion
Table 2:
Simulatability of proto-1m on SST-2 and QNLI reported as the mean pereentage of model de- cisions replicated by three workers.
We introduce proto-lm, a white-box framework designed to offer inherent interpretability for Lan- guage Model Learnnng (LLMs) through inter- pretable prototypes. These prototypes not only explain model decisions but also serve as feature extractors for downstream tasks. Our experimental results indicate that proto-lm delivers competitive performance across range of Natural Language Processing (NLP) tasks and exhibits robustness under various hyperparameter settings. The in- terpretability of proto-Im is evaluated, and our findings show that proto-lm delivers faithful ex- planations that can effectively assist humans in understanding and predicting model decisions.
salient parts of the training data. Their focus is on learning one representative prototype per class while their performance is dependent on the size of the support set in few-shot learning scenarios. Due to these limitations, there have been relatively few works that utilize prototypical networks provide interpretability for LLM's in NLP (Garcia-Olano et al., 2022; Das et al., 2022; Van Aken et al., 2022). Chen et al. (2019) and Hase et al. (2019) use pro- totypical parts networks with multiple learned pro-
totypes per class but only apply their methods to image classification tasks.
8 Limitations
While proto-lm offers inherent interpretability by creating a connection between input text and perti- nent parts of training data through the use of proto- types, it remains dependent on an underlying lan- guage model to convert text into a semantic space. Consequently, the interpretability of proto-1m is constrained by the interpretability of the founda- tional language model.
Our work is most closely related to Das et al. (2022) and Van Aken et al. (2022) in terms of the approach taken. However, our work differs in that the architecture in (Das et al., 2022) only utilizes single negative prototype for binary classifica- tion, while proto-1m enables multi-class classifi- cation by using multiple prototypes for each class, Additionally, we have extended the work of (Das et al., 2022) by implementing token-level atten- tion to identify not only influential samples but also influential sections of text within each sample. Moreover, different from the single-proootype-as-aa summary approach in Van Aken et al. (2022), by learning multiple prototypes per class, proto-lm creates a prototypical space 1$4.1), where, unlike the embedding space of LLM'ss s. each dimension is meaningful, specifically the distance to a learned prototype, and can be used to explain a decision. Our work is also similar to Friedrich et al. (2021) in terms of our loss function design. However, Friedrich et al. (2021)* 's loss function aims to max- imize the similarity of the closest prototypes of the same class. Conversely, our approach strives to minimize the distance of the furthest prototypes of the same class. This results in Friedrich al. (2021)'s approach tending to draw a single proto- type closer to a specific example, potentially lim-
Moreover, interpreting the significance of a learned prototype in the context of Natural Lan- guage Processing (NLP) tasks is still an open re- search area. Computer Vision (CV) methods used for visualizing prototypes in image-based tasks, such as upsampling a prototypical tensor, are not transferrable language embeddings. Instead, searchers depend on projecting prototypes onto nearby training examples decode prototype ten- sors into comprehensible natural language.
Ethics Statement
We present proto-lm, a framework that enhances the interpretability of Language Model Learning (LLMs) by providing explanations for their deci- sions using examples from the training dataset. We anticipate that the broad applicability proto-lm across various NLP tasks will promote trans- parency and trust in the use of LLMs, thereby en-

---

couraging their wider adoption. As observed by authors like Rudin (2019) and Jiménez-Luna et al. (2020), models with higher levels of interpretability inspire more confidence and enjoy greater utiliza- tion. We aim to contribute to advancing the field of interpretability research in NLP through our work. For assessing the simulatabilityy of our method, we employed Amazon Mechanical Turk (MTurk) for our human evaluation. To ensure English pro- ficiency among workers, we restricted their loca- tion to the United States. Furthermore, only work- ers with a HIT approval rate of at least 99% were permitted to undertake our tasks. We provided a compensation of $0.20 per task, which roughly translates to about $24 per hour, significantly ex- ceeding the US federal minimum wage. To uphold anonymity, we refrained from collecting any per- sonal information from the annotators.
Explaining model decisions with prototype tensors. arXiv preprint arXiv:2204.05422
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. 2022. Framework for evaluating faithfulness of local explanations. International Conference on Ma- chine Learning, pages 4794-4815. PMLR.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Jay De DoYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong. Richard Socher, and Byron C. Wallace. 2019. Eraser: benchmark to evaluate rationalized nip models. arXiv preprint arXiv:1911.0342,
Finale Doshi-Veles and Been Kim. 2017. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1e..117022..0..
Patrick Fernandes, Marcos Treviso, Danish Pruthi, An- dré Martins, and Graham Neubig. 2022. Learning to scaffold: Optimizing model explanations for teach- ing. Advances in Neural Information Processing Systems, 35:36108-36122
10 Acknowledgements
This research was supported in part by grants from the US National Library of Medicine (R0ILM012837 & ROILM013833) and the US National Cancer Institute (R01CA249758). In addition, would like to extend our gratitude Joseph DiPalma, Yiren Jian, Naofumi Tomita, Weicheng Ma, Alex DeJournett, Wayner Barrios Quiroga. Peiying Hua, Weiyi Wu, Ting Shao, Guang Yang, and Chris Cortese for their valuable feedback on the manuscript and support during the research process.
Felix Friedrich, Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. 2021. Interac- tively providing explanations for transformer lan- guage models. arXiv preprint arXiv:2110.0200
Tianyu Gao. Xu Han, Zhiyuan Liu, and Maosong Sun. 2019. Hybrid attention-basee prototypical networks for noisy few-shot relation classification. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6407-6414.
Tianyu Gao. Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104.0882)
References
Diego Murcia-Olano Yasumasa Onoe, Joydeep Ghosh, and Byron C Wallace. 2022. Intermediate entity- based sparse interpretable representation learning. arXiv preprint arXiv:2212.01641
Simon Baker, Iloaa Silins, Yufan Guo, Imran Ali, Johan Högberg. Ulla Stenius, and Anna Korhonen. 2016. Automatic semantic classification of scientific litera- ture according to the hallmarks of cancer. Bioinfor- matics, 422-4432
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xianchang Peng, Xiang Ren, and Hamed Firooz. 2022. Unirex: A unified learning framework for language model rationale extraction. In International Conference on Machine Learning. pages 2867-2889. PMLR.
Peter Hase, Chaofan Chen. Oscar Li, and Cynthia Rudin. 2019. Interpretable image recognition with hierar- chical prototypes. In Proceedings of the AAAI Con- ference on Human Computation and Crowdsourcing. volume 7, pages 40.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. 2019. This looks like that: deep learning for interpretable image recog- nition. Advances in neural information procexsing systems, 32.
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable nip systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685.
Anubrata Das, Chitrank Gupta, Venelin Kovatchev, Matthew Lease, and Junyi Jessy Li. 2022. Prototex:

---

Alon Jacovi and Yoav Goldberg. 2021. Aligning faithful interpretations with their social attribution. Transac- rions of the Association for Computational Linguis- tics, 9:294-310.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. CoRR, abs/1704.0280
José Jiménez-Iuuna, Francesca Grisoni, and Gisbert Schneider. 2020. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573-84.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. 2016. Not just black box: Learning important features through propagating acti- vation differences. arXiv preprint arXiv:/605. 01710.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, 30.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zeitlemoyer. 2019. Bart: De- noising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:7910. 13461.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta:
Gizem Sogancroğlu, Hakime Öztürk, and Arzucan Özgür. 2017. Biosses: semantic sentence simi- larity estimation system for the hiomedical domain. Bioinformatics, 33(14)i44--58.
robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: arXiv:1907.1 1907.
1692.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for sim- plicity: The all convolutional net. arXiv preprint 1rXiv:1412.660
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019. Hierarchical attention prototypical net- works for few-shot text classification. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- tional joint conference on natural language process- ing (EMNLP-IJCNLP), pages 476-485.
James Murdoch, Peter J Liu, and Bin Yu. 2018. Be- yond word importance: Contextual decomposition extract interactions from Istms. arXiv preprint arXiv:1801.05453.
Zijun Sun, Chun Fan, Qinghong Han, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020. Self- explaining structures improve NLP models. CORR. abs/2012.01786,
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. 2022. Evaluating explanations: How much do ex- planations from the teacher aid students? Transac- tions of the Association for Computational Linguis- tics, 10:359-375.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. CoRR. abs/1703.01365.
Betty Van Aken, Jens-Michalle Papaioannou, Marcel G Naik, Georgios Eleftheriadis, Wolfgang Nejdl, Fe- lix A Gers, and Alexander Löser. 2022. This patient looks like that patient: Prototypical networks for in- terpretable diagnosis prediction from clinical text. arXiv preprint arXiv22210.08000
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese nett-nesworks. arXiv preprint arXiv:1908.I0 10084.
Marco Tulio Ribeiro, Sameer Singh. and Carlos Guestrin. 2016. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135- 1144.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:1804.074611
Cynthia Rudin. 2019. Stop explaining black box ma- chine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelliience, 1(55:206-215.

---

A Training Details
HoC (F))
BIOSSES (MSE)
proto Broo-la/PubMMBBBkk 83.15 1 0.55882.32
1.32 0.14 1.14
In our experiments, we utilized two NVIDIA Titan RTX and two GeForce RTX 3090 GPUs to run our experiments. We conducted extensive search for the best hyperparameters through experimenta- tion. Our models were trained for maximum of 40 epochs. We initialize weights (in the classifica- tion layer) connecting each prototype's similarity to the logit of their respective weights to and other weights to -.5, This technique (Chen et al.. 2019) allows for better association of prototypes and faster convergence. In terms of initializing the initialized
protoIn/BioGPT
82.78 1 1.11/88.12
1.16 0.107 1.08
Table 4: proto- 1m's performance on biomedical
datasets HoC and BIOSSES
RoBERTa-large:
http://hugginngaecccooohetttttt
BART-large: htts://facebook/brtlaggg
B
Additional experiments and interpretability examples
parameters within the prototypes, them randomly from [0.1). We used learning rate and batch size be 3e-6 and 128, respectively. We employed Adam (Kingma and Ba, 2014) as our optimizer with Ies of (0.9, 0.999). Additionally, we limited the input size (number of tokens) to 512 during tokenization. We also investigated the optimal number of N and tested proto-nlp un- der N {100, 200, 400, 800, 1000, 1200, 1400). We discuss thsi in $4.3. Similarly, we tested K 21,2,5, 5, 10, 100, 250, 500} and reported the results in $4.4. Furthermore, we evaluated the ef- feet λo Å₁, and X2 on our framework. The re- sults of one of these tasks (IMDb) are presented in Figure in the paper. The optimal A for the remaining tasks are reported in Table.3.
We perform additional experiments with proto- 1m on two biomedical datasets: The Hallmarks of Can- Corpus (HoC) (Baker al., 2016) and Sen- tence Similarity Estimation System for the Biomed- ical Domain (BIOSSES) (Sogancroğlu al., 2017). HoC is a text classification dataset comprising 1852 abstracts from PubMed publications that have been annotated by medical experts based on a tax- onomy. BIOSSES consists of 100 pairs of PubMed sentences, with each pair having been evaluated by five expert annotators. The sentences assigned similarity score ranging from 0 (indicating no relation) to 4 (indicating equivalent meanings).
We build proto-lm with two pretrained L.M'' for the biomedical domain: PubMedBERT (Gu et al., 2021) and BioGPT (Gu et al., 2021). We report our results on the biomedical datasets in Ta- ble 4. For the HoC results, we use N = 1000, Ao 0.3, a 0.35, X2 0.35, and K 1. For the BIOSSES results, use N 25, Åo 0.6, Å1 0.25, X2 0.25, and K 1. Also, BIOSSES regression task, we set Co 1, forgo the softmax layer in the classification head, and replace L.e with Cyse. Similar to our results in Ta- ble. 1, we observe competitive performances from proto-lm, 1m, once again showing that proto-In 1m of- fers interpretability, but not at the cost of perfor- mance. To demonstrate proto-lm's interpretability. additionally show 3 examples from HoC and the most/least similar prototypes found for those examples in Fig. 11.
Ag 2
A2
SST2
0.2
0.4
0.4
QNLI
0.1
0.45 0.45
MNLI
0.45 0.45
WNLI
0.-
RTE 0.3 0.35 0.35
IMDE 0.3 0.35 0.35
SST5
0.2 0.4 0.4
Table Optimal X's for each task found through exper-
iments.
We utilized pretrained transformer models from
Hugging Face, including:
BERT-basc-uncassett
C Effect of unequal t and X2
BERTT-arge-uncased:
We conduct experiments on proto-lm, varying only ê₁ and A (the weights of coh and sep losses. respectively) while keep other hyperparameters the
http:://naggiiiggaacccoooorrrees
RoBERTa-base: hhtp://huggingfaae..ooooooo
same. We show the results for these instances proto=Im on SST5 Figure. 9. We observe that

---

while differing A and X2 values lead conver- gence, placing equal emphasis on Å and 22 leads to convergence at a lower loss. Intuitively, rely- ing on either only pulling the correct prototypes together (cohesion) or only relying on pushing the incorrect prototypes apart (separation) is sufficient to create meaningful prototypical space that al- lows for adequate performance the task. Our experimental results show that placing equal em- phasis leads to better performance.
Avaraga nama ined' distance between prototyons and their projected examples.
proto-m
0.-
proto (Dast et.
Prototypical et.
0.-
oo Dalance
0.5-
0.4-
82
Vilicatioo lass training stops under diftrennt combiraations offt and and while holöing K-0.3 constant
HDC
BOOSSES
Legend:
11-0112::66
11-06J2-2.1
Valditin
0.34
Figure 10: Average normalized distance between pro- tolypes and the output embedding of prototypes' projected samples.
11:02J2:05
0.3
11-0LI2-04
1--4-1-33
13-035
training samples and j. Let II; " be the probability distribution for prototype Py over the dataset D, where xद and 1) is constant for d?
500
600
Stops
Figure Loss proto-lm on SST5 with different combinations of X and Az. We observe faster conver- gence at lower loss when X = A2.
normalization. Let and b represent two training samples, their relative probabilities (for being pro- df
jected onto by Pý) would be xá
n//d
%/ di
D Projecting prototypes
In addition, EEED Tे = = ERED /j + MJ Σxen 1/d;" which means that each prototype is soft-clustering since da/d?, x//=3"
In order to help human users understand prototypes in natural language, we identify the closest train- ing data sample for each prototype and project the prototype onto that sample. The projection of proto- types onto the nearest sample a well-studied and established technique (Das et al., 2022; Van Aken et al., 2022; Chen et al., 2019; Hase et al., 2019). We compare the quality of our projections against the projections obtained via the training procedure of Proto-tex (Das al., 2022) and the loss function used in Protoypical Network (Chen al., 2019) by measuring the average normalized distance be- tween each prototype and their projected sample's embedding. We show the results for two datasets in Fig. 10. We observe that proto-1m is able to train prototypes that are much closer in distance to their projected samples' embeddings than the prototypes obtained via Protoypical Network loss and within margin of error to that of Proto-tex.
examples in D. Moreover, if is n times further away from pj than b, then is n times more probable in the probability distribution IIy. Similarly, dur- ing inference time, for example k. a prototype i is n times away from k than prototype j. then di / d2 more probable to be the prediction than i for k.
/ = n, i.e. will be times
F Interpretability examples and sample Mturk questionnaires
Prototype faithfulness
In addition experiments in $5.1 and $5.2, provide a theoretical justification for the inherent interpretability in proto-1m in this section. Denote df as the distance between two prototypes and/or

---

Input Example
Predicted class
Most similar prototypes
Least similar prototypes
<oclass
Alloogh Lawe REMAA
I8 Awahigrnewe meeeoe
Ovenl, the rould prouids celieee:
Genwee Genwerreaell risiliy fimparion Sat Caneer DhAngai'
to potental nd gaseti wwwdeeeeeettieiieeeeooooooo
cardageeiity Inmansnagrooke
system' Canser Inmursee wwt.roogpptttttoccooo
fos bdlwina oussure ty Qureitar sovaition losb sapant the eockingg dewpoin mary sona
of pomary jumesa reporsaves
13 Foode
ceceunnnpperrr Canser semuroaspreaaatto
firslog andior unbal
cinal eptopeinn
16 Anddinglnneee deer elovitty teeunccypler Canes
Eiteopn brondessss nitnet ieportent patheloge featiigt
Geoewiecsallt Amataiio Meuathinn
Litani monorylogen
rewurcalirersirs activty ofeike
vemnnnnng laney denkiticc (OOKng hariing suppeszn iitytty exprestio waltus of inn perunoogpomntie
show regatweyy (he Martung offect cance col
Aplanoodi Cancer Jursphagg Cancr
gorsstion modender
Argpogenn othittyrrat conditanee mede prwiiamdd mecraphgge
Tanororcnora (77
Incoopeede di petcets compured So Suctarnn probfict sagalingg pespecian cassewaetion acliiiy (ngiesstene
er celpppppcccc dirainne antbodlu to othrilk 1 sumsr recuds
"Del diversssss Carus Informatiin on Tomnr raporde
Growth meder Cander Canoe
ad Bane Sood lalas lamoroals out (2829)
Tunansonnaatniiieee internaattneeeeneeeee (TP) Toaletiee Cancer Infammntion raperaa
1011/00 sanali the rosas of vilegee
Taroeperaeee
Iaaeeeee Coneer
senavarne azcuctivated atterccrpport
Deadiig growch supeessoo "ontae Bitoo
mino peguulatonee
'Oedethe Carcer Intonviation femmrr Cancrt
aoppadl-nddubteeee. (MIT-) iddctiio
peoslate Uil
webk me tamargrnelfirr rncrumber the srouet of noselisted woolt
Infammetaa Iefremoaiio Vnmur resione Carcn
(T
bnceratt with incuedd
Furthemone combirattion
trald splkutiv iematallyyyt InnertallaaiinnCrre
dggitbart Incuctorr suteshepy uili callitnggeee wilid
fumerr cellin wneennanne auclstash
Residing relldnatattO Necrods Caroer
tytaciennnal arrctuses, isperssed
Inducing on plogeendc
plpolyysiion the sepply during spoctons exertmants diyytuud glanss
wnrched Ivcetsssennal
Exccioni
wilewer ling alemetacoraeen
Tumoor recoall vas snsuee diai cboocer onch
indaciig anploemess
Meglogerchhss heose
A431 iigreann cctory fonmution veart wi wely pelen rensegiphhy
arglogeneet pigmees
the Slcod of cance pathents and can
Mitaalirn repair Cencer
Inducing plogenelllllopoooooooooooo
and DCC shoeelype Iitt vill which onhence chanoviiianne
HUR020 muutac more spld
Olecctoo Canuu
Angiogoiiflcte
Figure Example inputs from HoC, their predictions prototypes identified by proto-1m

---

Sentiment Analysis:
A model has made a decision on the sentiment (positive/negative) for the following movie review (#30).
This model made its decision this review's sentiment by matching this review with 5 other reviews that considers very similar this review.
What was the model's decision? (positive/nngative)
Review #30:
[BEGIN] could easily be called the best korean film of 2002 [END]
5 matched reviews:
Matched Review #1: [BEGIN] impresses as skilfully ssembled, highly polished and professional adaptation [END)
The sentiment of review was: positive
Matched Review #2: [BEGIN] the film 's intimate camera work [END]
The sentiment of this review was: positive
Matched Review #3: [BEGIN] of masterpiece [END]
The sentiment of this review was: positive
Matched Review #4: [BEGIN] this intricately structured and well-realized drama [END]
The sentiment of this review positive
Matched Review #5: [BEGIN] utterly absorbing (END)
The sentiment of this review was: positive
Figure 12: Example Amazon Mechanic Turk questionnaire used evaluate simulatability with prototypical explanations (PE)

---

Sentiment Analysis:
is is the sentiment of the following review (#48) (positive/negatlvy You may use matched reviews below to help you make your decision
Review #48:
[BEGIN] proves once again he has n't lost his touch, bringing off superb performance in admittedly middling film [END]
A model has made decision on the sentiment of review #48 by matching it with reviews it considers to be similar to #48. Here are 5 of the matched reviews:
Matched Review #1: [BEGIN] it does n't follow the stale, standard, connect the-dots storyline which has become commonplace movies that explore the seamy under- belly the criminal world [END]
The sentiment of this review was: positive
Matched Review #2: [BEGIN] that's such superficial treatment (END)
tragic merit
+ sentiment review was: negative
Matched Review #3: [BEGIN] such premise is ripe for all manner of lunacy, but kaufman and gondry rarely seem sure of where it should [END]
The sentiment of this review negative
The sentiment of this review was: positive sentiment this review was: negative Figure 13: Example Amazon Mechanic Turk questionnaire used evaluate accuracy with random explanations.
Matched Review #4; [BEGIN] enables shafer to navigate spaces both large and small with considerable aplomb [END]
Matched Review #5: [BEGIN] noticing some truly egregious lip-non-synching(END)]
Example MTurk questionnaire with explanations provided by LIME//ntegrated Gradients/ Guided Backprop
Choose primary sentiment expressed by the following movie review.
Please pay close attention to the highlighted words help you make your decision.
Words highlighted in green carry positive sentiments, Words highlighted in red carry negative sentiments.
The derker the shade the stronger the sentiment.
You may not agree with the colors and sentiments that certain words You will have chance reflect your thoughts this
in the answers.
Lagend:
Negative Nautral Positive
Review #25:
[BEGIN]
deadly
watching
proverbial paint
improvement
[END]
Figure 14: Example Amazon Mechanic Turk questionnaires we used to evaluate simulatability of LIME/Interaated Gradients/Guided Backprop.


Reviews:
[{"review_id": "mB9XuxV9nL", "reviewer": "Reviewer_9W3P", "summary": "", "questions": "A. While choosing the size of the prototypical space, class imbalance, which is a major artifact of many NLP datasets, is never discussed. Could you add a discussion of how that interferes with choosing the size of the prototypical space ?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedbacks and their recognition of our work's contributions.\n\n**Comparison with Friedrich et al.:**\nRegarding the similarity to the work of Friedrich et al. 2021 (hereafter referred to as 'F'): While at first glance, the loss functions may appear similar, there are pivotal differences between F and our approach that warrant emphasis:\n\n**Loss function:** In terms of cluster/cohesion loss, F aims to maximize the similarity of the closest prototype of the same class. Conversely, our approach strives to minimize the distance of the furthest prototypes of the same class. This results in F's approach tending to draw a single prototype closer to a specific example, potentially limiting prototype diversity and representation power. They counteract this potential limitation by introducing an additional diversity loss term.  The distribution loss term in F similarly reinforces the *same* example to be associated with each prototype and exacerbates F's reliance on the diversity term to learn representative prototypes.\n\nOur method, in contrast, ensures prototype diversity by leveraging the K hyperparameter, which we delve into in section 4.4. This results in a method that is not only more straightforward but also, as demonstrated by our experiments, performs superiorly on the Yelp and and MovieReviews benchmarks:\n\n\n\n|                   | Yelp | MovieReviews |\n| :---------------- |:------:|:------:|\n| BERT + Proto-Trex  (Friedrich et al.)| 92.10$\\pm$0.08| 75.51$\\pm$0.42 |\n| BERT + proto-lm                      |**92.88**$\\pm$0.04|**80.01**$\\pm$0.22|\n\n**Performance-Interpretability Trade-offs:** Perhaps as a result of their loss function, F reports notable trade-offs between interpretability and performance across experiments. This is not a characteristic shared by our work.\n\n**Questions For The Authors:**\nRegarding the size of the prototypical space and the influence of class imbalance, we acknowledge the importance of these considerations. In brief, the optimal performance requires a balance in the prototypical space such that all classes have sufficient representation. Our experiments with the SST5 dataset, known for its imbalanced sample distribution, highlight this.\n\nWe conduct experiments varying the size of prototypical space on the SST5 dataset, which has an imbalanced amount of samples, specifically [1092/2218/1624/2322/1288] samples for classes 1-5. We observe in the table below when increasing the number of prototypes assigned to classes 2/4 from 100 to 200, the marginal gain in accuracy is about (55.23-48.33)=6.9. On the other hand, when increasing the number of prototypes assigned to classes 1/3/5, the marginal gain in accuracy is only about (58.00-55.23)=2.77. This is because there were less samples in classes 1/3/5 in the dataset from which prototypes can learn information.  We agree that a discussion on the effect of class imbalance in the dataset is important and will add a detailed discussion upon revision.\n\n| # prototypes assigned for each class| Performance (Acc.) | Gain $⇑$ |\n| :---------------- |:------:|:------:|\n| 20/20/20/20/20| 30.79$\\pm$0.02|-|\n| 100/100/100/100/100| 48.33$\\pm$0.04|+17.54|\n| 100/**200**/100/**200**/100| 55.23$\\pm$0.02|+6.90|\n| **200**/200/**200**/200/**200**|58.00$\\pm$0.04|+2.77|\n| 300/300/300/300/300|58.03$\\pm$0.04|+0.03|\n\n**Missing references**\n\n- Friedrich et al 2021: Please see Reasons to Reject\n- Luo et. al. 2023, ACL: We will include a section on similarities with this work in our related works section.\n\n\nWe are committed to refining our work based on this feedback and will include a discussion of the references above in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors extend current work  jointly learning prototypes representative of a class and making the final prediction of the model go through a calculation of similarities with representative protypical spaces. \n2. Experiments varying the size and uniqueness of prototypical space and its effect on model accuracy. \n3. Experiments to judge faithfulness with Comprehensiveness and Sufficiency scores, and a human study for simulatability.", "reasons_to_reject": "1. The framework proposed in the method is very similar in spirit to the proposed in Friedrich et. al. 2021 [Interactively Providing Explanations for Transformer Language Models], who use a modified loss function, although using two different networks for word and sentence level interpretability. The similarities in the modification of the loss function [Equation (1-2), https://arxiv.org/pdf/2110.02058.pdf] might be a bit too similar for the work to meet the criteria of novelty.", "questions_to_authors": "A. While choosing the size of the prototypical space, class imbalance, which is a major artifact of many NLP datasets, is never discussed. Could you add a discussion of how that interferes with choosing the size of the prototypical space ?", "paper_topic_and_main_contributions": "The paper proposes a prototype-based method that adds a layer of interpretability to language models, and show its viability in a multi-class classification setting. The authors add an addition layer (dubbed the prototypical layer) on top of a pretriained LLM that learns representative examples of a particular class. They alter the standard cross-entropy loss function to include terms that make tighten a particular prototype group while placing it away from a group it doesn't belong to but is close in the embedded space. The modified loss function is comparable in performance with the original LLM, but can point to examples in the representative space that explain the model's choices.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The paper seems to miss two major related work - \n\n1. Friedrich et al 2021: Interactively Providing Explanations for Transformer Language Models. The version I am referring to is https://arxiv.org/pdf/2110.02058.pdf. \n\n2. Luo et. al. 2023, ACL: Prototype-Based Interpretability for Legal Citation Prediction. Although I can understand if the authors missed this one."}}, {"review_id": "iRxOPpnJB6", "reviewer": "Reviewer_pjWZ", "summary": "", "questions": "Question A: You mentioned the necessity of tuning the hyper-parameters such as $\\lambda$, $N$, and $K$ for each task. If we were to consider limited access to examples from a new task, have you observed the robustness of the hyper-parameters derived from the most relevant scenario? Is there a significant impact on performance if you were to employ the same set of hyper-parameters for all NLI tasks, for instance?\n\nQuestion B: In the main text, your focus seems to be on encoder/encoder-decoder architectures. I'm curious about the method's performance with decoder-only language models. Have you conducted any experiments to explore this aspect?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's acknowledgement of the effectiveness of our approach, the quality of explanations that our framework allows for and the comprehensive nature of our experiments. We address your questions and concerns below:\n\n\n**Regarding the Reasons To Reject:**\nWe agree that validating the efficacy of our approach across a broader range of domains would further strengthen our contributions. To this end, we wish to highlight some results in the appendix section (specifically App. B, Table 3 and Figure 11), where we present experimental findings of our model, proto-lm, on two biomedical datasets: Hallmarks of Cancer (Cancer type classification) and BIOSSES (Biomedical sentence similarity estimation). While these experiments might not encompass every possible domain, they serve as evidence that our approach can be effective beyond the tasks mentioned in the main text.\n\n**Response to Questions:**\n\n- Question A:\nYour point on hyperparameter transferability is indeed pertinent. We found that the hyperparameter $\\lambda$ showcases a consistent optimal value across tasks like QNLI and MNLI (cf. Appendix A and Table 2). The optimal $\\lambda$ values for WNLI are in a similar range to that of QNLI and MNLI, albeit not identical. We attribute this discrepancy to the relative sparsity of the WNLI dataset as well as the inherent dissimilarity between the tasks of WNLI and QNLI/MNLI. The hyperparameter N largely hinges on the dataset. Its optimal value would be influenced by the dataset's capacity to allow proto-lm to discern N prototypes in total (C prototypes per class). Thus, N's transferability across tasks and datasets is less predictable. In our experiments, we found that allocating a large number of prototypes (N=1000) to be an effective strategy for mitigating this unpredictability and improve overall task performance.  Our experiments did reveal that K = 1 yields the most distinct prototype, and a larger K ($\\geq$ 10) optimizes task performance. We found this behavior consistent across our experimental tasks.\n\n- Question B: Based on your suggestions, we explored our method with auto-regressive models. Alongside experiments with Bio-GPT outlined in the appendix, we also evaluated our approach using the GPT-2 (small) on SST2, QNLI, and WNLI from the GLUE benchmark. The results indicate that the proto-lm model, when backed by GPT-2, either outperforms or matches the baseline performance of a fine-tuned GPT-2, all the while offering enhanced interpretability:\n\n|                   | SST2 | QNLI| WNLI |\n| :---------------- |:------:|:------:|:------:|\n| proto-lm/GPT-2    |**93.0**$\\pm$0.05/92.49|91.18$\\pm$0.04/91.23|**56.34**$\\pm$0.01/54.93|\n\n\nWe are committed to integrating these additional findings into our revised paper. We appreciate this great suggestions which will improve our paper.\n\n**Typos Grammar Style And Presentation Improvements:**\n\nWe regret the oversights in our manuscript and are grateful for your meticulous attention to detail.\n\n* Lines 126, 291, and 750 will be corrected in our subsequent version.\n* For Line 156, your notation suggestion will be incorporated for clarity.\n* Lines 473 and 478: We will ensure that the appropriate citation format is employed.\n\n\nThank you again for your valuable insights. We are confident that, with the proposed amendments, our paper will be significantly improved in terms of clarity and depth.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors extensively evaluate the performance of the proposed methodology through a diverse set of experiments. They assess the effectiveness of the approach along with the quality of the explanations it generates. These explanations are further evaluated regarding their usefulness in identifying misclassifications and their alignment with human reasoning.\n\nFurthermore, the authors conduct a comprehensive ablation study, which delves into various aspects of tuning hyper-parameters and practical implementation of the framework. This study offers an intuitive understanding of how the method operates internally, shedding light on its inner workings and potential areas of improvement.", "reasons_to_reject": "The authors primarily focus their experiments on sentiment classification and natural language inference (NLI) tasks. This emphasis prompts an important question concerning the method's generalizability and applicability across a broader spectrum of potential use cases. While the results for sentiment classification and NLI tasks are likely valuable, the effectiveness of the proposed approach in other domains and tasks remains to be explored. Therefore, further investigation into its adaptability to diverse applications is warranted to fully understand its scope and potential impact in various fields.", "questions_to_authors": "Question A: You mentioned the necessity of tuning the hyper-parameters such as $\\lambda$, $N$, and $K$ for each task. If we were to consider limited access to examples from a new task, have you observed the robustness of the hyper-parameters derived from the most relevant scenario? Is there a significant impact on performance if you were to employ the same set of hyper-parameters for all NLI tasks, for instance?\n\nQuestion B: In the main text, your focus seems to be on encoder/encoder-decoder architectures. I'm curious about the method's performance with decoder-only language models. Have you conducted any experiments to explore this aspect?", "paper_topic_and_main_contributions": "This paper introduces a new approach to enhance the interpretability of language models by incorporating prototypical networks. The authors suggest using a pre-trained language model as a backbone and then learning a set of prototypes associated with the classification classes for a given task. These prototypes are then leveraged during inference to produce interpretable predictions. Importantly, this framework retains competitive performance compared to the original language model, making it possible to achieve explainable decisions with minimal computational overhead and without compromising efficacy.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 126: which **is** the\n\nLine 156: $Wc$ -> $w_c$ (since you are referring to the column-vector of $W_h$)\n\nLine 291: On the other **hand**\n\nLines 473 / 478: \\citet\n\nLine 750: thsi -> this"}, "extra_text": {}}, {"review_id": "FTKjkahon6", "reviewer": "Reviewer_9tN3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and feedback on our manuscript. We appreciate the reviewer's acknowledgement of our contributions. We also appreciate the reviewer's detailing of their concerns and have made every effort to address each point with clarity.\n\n**Similarity to Concept Bottleneck Models:**\n\n*Background on Concept Bottleneck Models:*\nConcept bottleneck models, as outlined by Koh et al. (2020), leverage predefined human-interpretable concepts to mold their internal representations. Essentially, they introduce an intermediate layer during training where the model is required to predict these concepts. By doing so, they ensure that the model's internal representations (at least at that layer) are aligned with human-understandable concepts, facilitating interpretability.\n\n*Contrast in Fundamental Philosophy:*\nWhile both our proto-lm and the concept bottleneck models seek to improve interpretability, their foundational philosophies diverge. Concept bottleneck models impose human-defined concepts, thereby essentially \"guiding\" the model to think in terms that humans understand. On the other hand, our proto-lm autonomously discerns and employs prototypes that emerge from the data, without explicit human-defined categories or concepts. This distinction is crucial: while one model is being \"taught\" what concepts to consider, the other is \"discovering\" patterns from the data itself.\n\n*Implementation Differences:*\nAs a consequence of the above, the practical implementation also varies significantly. Concept bottleneck models require meticulous manual annotations of concepts across datasets, which is labor-intensive and may introduce biases based on the annotators' perspectives. Conversely, proto-lm automatically pinpoints prototype vectors during the learning phase, matching them with the nearest examples from the dataset. This ensures a balance of interpretability and scalability, with less risk of human bias.\n\n*Scalability and Applicability:*\nOwing to the manual annotation requirement, concept bottleneck models can become cumbersome when introduced to large or novel datasets. This limitation hampers their scalability, especially in rapidly evolving domains. Our proto-lm, with its autonomous prototype discernment, can readily adapt to various datasets without necessitating manual input, ensuring both versatility and broader applicability.\n\n*Potential Synthesis:*\nIt's worth noting that the strengths of both models can be potentially merged. For instance, an intriguing avenue for future research could be a hybrid model where the proto-lm's discovered prototypes are further refined using human-specified concepts, marrying the best of both worlds: automated prototype detection with human-guided refinement.\n\n*Comparison with our Method:* Comparing our method with the concept bottleneck approach would mandate manual annotations for every dataset. Given the wide variety of datasets we have employed, this is currently infeasible for us. Nevertheless, we acknowledge the value in such comparisons and will consider them for potential future work.\n\n**Prototype Selection and Training:**\nWe regret any oversight in the clarity of our prototype selection process. In brief, prototype vectors are honed using CE, cohesion, and separation. Post-training, these vectors are projected onto the closest dataset example — the nearest being defined by the minimal distance between a prototype and a given example's token-level attended embedding (refer to section 2.4, appendices D and F).\n\nAll prototype vectors in our framework are autonomously learned and then automatically matched to the closest examples within our training dataset, ensuring a balance between interpretability and scalability. More on this is discussed in sections 2.3 and 2.4.\n\nWe will amend our methodology section to clarify the prototype training and projection process.\n\n**Dataset-Specific Prototypes:**\nGiven the distinct characteristics inherent in different datasets, our proto-lm is designed to discern and utilize unique prototypes tailored to each dataset. Our empirical results consistently demonstrate this adaptive behavior across diverse NLP tasks.\n\n**Completeness of Prototypes:**\nIf the reviewer is referring to the coverage of prototypes over examples in the dataset, we find that tuning the hyperparameter of K for proto-lm affects both the completeness and performance of proto-lm. We refer the reviewer to section 4.4 and Figure 6 in the paper that discuss prototypes uniqueness/coverage. In addition, we provide results below on the percentage of training data that have an *unique* prototype represent them, averaged across BERT-base and Roberta-base with N=1000. We observe a similar trend as in Figure 6. Specifically, as we increase K, more and more prototypes project onto the same example in the dataset, thus the uniqueness of prototpes decreases overall. We note that, since we have only N=1000 prototypes in the prototypical layer, the maximum coverage would be 1000 samples in the dataset, with each prototype projecting onto 1 unique sample. Initially, this constraint might seem stringent. However, across all of our experiments, we observe that learning 1000 prototypes and capturing features from an upperbound of 1000 examples is more than sufficient for achieving good performances on all tasks we experimented with (cf. Table 1 and Figure 5).\n\n|Dataset |K=1|K=2|K=5|K=10|K=100|\n| :----- |:-:|:-:|:-:|:-:|:-:|\n|IMDb (Training data size 25000) |3.56%|1.33%|0.89%|0.08%|0.00%|\n|SST5 (Training data size 8544)|12.33%|7.43%|2.11%|0.74%|0.00%|\n|RTE  (Training data size 2490)|28.43%|15.55%|4.17%|2.10%|0.08%|\n\n\n**Flexibility and Robustness of Our Approach:**\nWe recognize the pertinence of your concerns regarding the flexibility and robustness of our model. To this end, we've embarked on extensive ablation studies (sections 4.1 through 5.1) that rigorously evaluate our model's resilience under different conditions. These studies provide a comprehensive assessment of proto-lm's performance.\n\n**Comparison with Other XAI Models:** We agree on the importance of contrasting our work against established XAI techniques. As such, our studies (detailed in App. G) benchmark proto-lm against methods like LIME, Integrated Gradients, and Guided Backprop. In terms of producing relatable and understandable explanations (simulatability), proto-lm excels.\n\n**Reproducibility Concerns:** We acknowledge the inherent challenges associated with human annotation replication. However, given that the essence of interpretability is inherently human-centric, we firmly believe that studies founded on human annotations, like ours in Section 5.2 and App. G, offer invaluable insights into the real-world applicability of our findings.\n\n**Faithfulness Metrics:** While our research already integrates established metrics like COMP and SUFF (as presented in the referenced paper), we concur that a broader exploration of faithfulness metrics can further fortify our findings. We've thus extended our investigations to encompass metrics like Decision Flip - Most Informative Token (DFMIT) and Decision Flip - Fraction of Tokens (DFFOT), showcasing proto-lm's superior faithfulness in comparison to other notable XAI techniques (as demonstrated in the table below).\n\nSpeicifically, since prototypical networks rely on prototypes to make decisions AND identify prototypes as their explanations, we conduct experiments by flipping (removing) prototypes and observing changes in output. For DFMIT, we remove the *most* important prototype and observe to see if the decision changes afterwards. For DFFOT, we calculate the metric as the fraction of prototypes needed to be removed before the decision changes. We compare proto-lm (Roberta-large) against 4 other interpretability methods on IMDb:\n  - An attention based approach (Fernandes et al. 2022): Pooled attention matrix of the last layer\n  - A perturbation-based approach: LIME\n  - A gradient-based approach: Integrated Gradients\n  - A prototypical network-based approach, Prototex (Das et. all 2022).   \n\nWe report our results in the table below, where we observe proto-lm to produce more faithful explanations than other XAI methods.\n\n|                   | DFMIT $⇑$| DFFOT $↓$| COMP $⇑$ | SUFF  $↓$|\n| :---------------- |:------:|:------:|:------:|:------:|\n|Pooled Attention (Fernandes et al.)|0.052|0.387|0.131|0.111|\n|LIME|0.031|0.344|0.179|0.099|\n|Integrated Gradients|0.084|0.354|0.141|0.101|\n|Prototex (Das et al.)|0.251|0.250|0.326|0.150|\n|Proto-lm|**0.287**|**0.179**|**0.433**|**0.077**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper studies a prototypical network approach to attribute the importance of words.", "reasons_to_reject": "The approach this work proposes actually is more similar to the concept bottleneck models(https://arxiv.org/abs/2007.04612) . However, the approach proposed in the concept bottleneck is more elegant and neat. For example, in this paper, there is no clear explanation about how these prototype sentences are selected.  Prototypical Networks are initially proposed to address few-shot learning problems. So the problem of prototype selection is simple enough, which could be done by selecting samples from new classes.  In this work, these prototypes are selected automatically or by humans? Does it require different prototypes for different datasets? How about the completeness of the chosen prototypes? These concerns reflect the flexibility of the model but are not addressed in the experiments. The experiments are not strong as there are no other interpretable models as baselines to compare.  Meanwhile, \n\nI suggest some comparison with other interpretable models could be conducted, e.g. concept bottleneck,  integrated-gradient(https://arxiv.org/abs/1703.01365). Experiments based on human annotation are hard to reproduce. I suggest the authors could try some faithfulness metrics proposed in this paper(https://arxiv.org/pdf/2204.05514.pdf). And clearly explain how to select prototypes.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a framework named proto-lm to address the interpretability in large language models. For each category, several sentences are selected as prototypes. They estimate a similarity score for each input token with those tokens selected as prototypes and use the score as the attribution weights of tokens.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a prototypical-based framework that allows LLM to learn inherent interpretability by considering both token and sample-level features during the fine-tuning stage. The paper is clear, the model is well described, and experimental results show improvements in multi-class classification and NLI tasks and the quality of generated explanations. Two reviewers acknowledge the innovative aspect of the paper's approach, particularly the use of prototypical networks to improve interpretability. The method is seen as a novel way to achieve explanations for model predictions. Two reviewers raise concerns about the similarity between the proposed method and previous work, e.g. the proposed framework is similar to Friedrich 2021 (https://arxiv.org/abs/2110.02058), Koh 2020 (https://arxiv.org/abs/2007.04612), and Das 2022 (https://arxiv.org/abs/2204.05426). Clarification from similar approaches is needed.
Decision: Accept-Findings