Rather a Nurse than a Physician - Contrastive Explanations under Investigation
Authors: Oliver Eberle
Ilias Chalkidis
Laura Cabello
Stephanie Brandl
Venue: EMNLP 2023 Main (2023)
Keywords: explainability
contrastive explanations
human annotations
TL;DR: We compare contrastive/non-contrastive model-based explanations with contrastive/non-contrastive human rationales that we collected for this purpose.

Abstract: Contrastive explanations, where one decision is explained *in contrast to another*, are supposed to be closer to how humans explain a decision than non-contrastive explanations, where the decision is not necessarily referenced to an alternative. This claim has never been empirically validated. We analyze four English text-classification datasets (SST2, DynaSent, BIOS and DBpedia-Animals). We fine-tune and extract explanations from three different models (RoBERTa, GTP-2, and T5), each in three different sizes and apply three post-hoc explainability methods (LRP, GradientxInput, GradNorm). We furthermore collect and release human rationale annotations for a subset of 100 samples from the BIOS dataset for contrastive and non-contrastive settings. A cross-comparison between model-based rationales and human annotations, both in contrastive and non-contrastive settings, yields a high agreement between the two settings for models as well as for humans. Moreover, model-based explanations computed in both settings align equally well with human rationales. Thus, we empirically find that humans do not necessarily explain in a contrastive manner.

Rather a Nurse than a Physician- Contrastive Explanations under Investigation
Oliver Eberle
Ilias Chalkidis
Laura Cabello
Stephanie Brandl
Machine Learning Group, Technische Universität Berlin, Germany BIFOLD, Germany Department of Computer Science, University of Copenhagen, Denmark oliver. wewerreeereeeeeterlleeeee de, {ilias. chalkidis, lcp, brand1 ku. dk
Abstract
Non-contrastin setting
Contrastive explanations, where one decision is explained in contrast another, sup- posed to be closer to how humans explain de- cision than non-contrastive explanations, where the decision is not necessarily referenced to an alternative. This claim has never been em- pirically validated. We analyze four English text-classification datasets (SST2, DynaSent, BIOS and DBpedia-Animals We fine-tune and extract explanations from three different models (RoBERTa, GTP-2, and T5), each in three different sizes and apply three post-hoc explainability methods (LRP,, Gradientx GradNorm). We furthermore collect and re- lease human rationale annotations for subset of 100 samples from the BIOS dataset for con- trastive and non-contrastive settings. A cross- comparison between model-based rationales and human annotations, both in contrastive and non-contrastive settings, yields a high agree- ment between the two settings for models as well as for humans. Moreover, model-based explanations computed in both settings align equally well with human rationales. Thus, empirically find that humans do not necessarily explain in a contrastive manner.
Why person following diecrribdd as "Dentist??
He has 47 years of experience His specialties include Oral Maxillofacial surgery, Dr. Show affilated Bayler University Medical Center
47 years exparisnce specialties include im and Maxiliofacia Surgeni Dr. Show is affiliated with Daylor University Medical Center.
Contrastive setting
Why is the person in the following short bio descriied as a 'Dentiat' rother than "Surgenn" 7
He has 47 years of experience His specialties include Oral affilittd Baylar
Maxillofacial surgery Dr. University of Medical Center.
Ho 47 and Maxiliofacial surgery. Dr. Show is affillated with Daylor University Modical Center.
exparience. spacialtios include im
An example from the BIOS dataset of non- contrastive and contrastive human and model-base rationales. Human rationales are underlined and bold- faced, while model-based rationale attribution scores are highlighted in red (positive) or blue (negative) colors.
Figure I:
2021: Yin and Neubig, 2022) where an explanation for a model's decision provided in contrast to an alternative decision. Figure/ 1 shows an example for text classification of professions, where the con- trastive setting is phrased as: "Why is the person [...] dentist rather than surgeon?", Contrastive explanations are considered closer to how humans would argue and thus considered more valuable for humans understand the model'ss 's decision (Lipton, 1990; Miller, 2019; Jacovi et al., 2021). In partieu- lar, the latter has been shown in previous evaluation studies. So far however, has not been empirically investigated whether contrastive explanations are indeed closer to how humans would come to de- cision, i.e., whether human rationale annotations more similar to contrastive explanations than to non-contrastive explanations.
1 Introduction
In order to build reliable and trustworthy NLP ap- plications, is crucial to make models transparent and explainable. Some use cases require the ex- planations not only to be faithful to the model's inner workings but also plausible to humans. We follow the terminology from De Young et al. (2020) and define plausible explanations as model-based rationales that have high agreement with human rationales, and and faithful explanations as the input to- kens most relied upon for classification. Both qual- ities (plausibility and faithfulness) can be estimated via metrics, i.c. are not binary. Recently, various contrastive explanation approaches have been pro- posed in NLP (Jacovi et al., 2021; Paranjape et al.,
In this work, we initially compare human gaze and human rationales collected from a subset of the SST2 dataset (Socher et al., 2013; Hollenstein
Equal contribution.

---

al., 2018; Thom Jakobsen al., 2023), binary sentiment classification task, with contrastive and non-contrastive model-based explanations using the Layer-wise Relevance Propagation (LRP, Bach et al. 2015; Ali et al. 2022) framework. We find no difference between con-oontrastive and contrastive model-based rationales in this binary setting. The analysis on DynaSent (Potts et al., 2021) yields sim- ilar results. We thus further explore the potential of contrastive explanations, collecting human ratio- nale annotation for both settings, contrastive and non-contrastive, on a subset of the BIOS dataset (De-Arteaga et al., 2019) for a five-way medical occupation classification task, and compare them to model-based explanations.
and sizes (small, base, large) and for three different post-hoc explanation methods (LRP, GradientxIn- put, Gradient Norm) on four English text classifica- tion datasets. (ii) We include both human annota- tions and gaze patterns into our analysis, which pro- vide human signals at different processing levels. (iii) We release a subset of the BIOS dataset, text classification dataset for five medical professions. (iv) We further release human rationale annotations for 100 samples of this newly released dataset for both contrastive and non-contrastive settings.
We release our code on Github to foster repro- ducibility and case of use in future research."
Related Work
We find that human annotations in both settings agree on a similar level with model-based ratio- nales, which suggests that similar tokens are se- lected. Contrastive human explanations seem to be more specific (fewer words annotated), but agrec- ment between the two settings varies across classes. Based on these results, we conclude that humans do not necessarily explain in contrastive manner by default. Moreover, model explanations computed in non-contrastive and contrastive manner do not differ while both align equally well with human rationales. We observe similar findings in another single-label multi-class animal species classifica- tion dataset, DBPedia Animals.
Contrastive Explainable AI (XAI) Contrastive explanations have only recently been applied in language models. We are revising the most promi- nent recent papers but also would like to refer to earlier work in the field of computer vision (Dhu- randhar et al., 2018; Prabhushankar et al., 2020). Jacovi et al. (2021) propose framework to gen- erate contrastive explanations by projecting the la- tent input representation to a maximally contrastive space. They evaluate the usability of contrastive explanation on capturing bias on text classification benchmarks BIOS and NLI, but not the quality of the explanations per se. We use the BIOS dataset to collect human rationales in a contrastive and non-contrastive setting. Paranjape et al. (2021) ap- ply and human-evaluate contrastive explanations in commonsense reaaonnnn task. They find con- trastive explanations to be more useful humans and that model performance can be improved via conditioning predictions using contrastive expla- nations. Yin and Neubig (2022) compare three contrastive explainability methods with their orig- inal version: Gradient Norm, InputxGradient and Input Erasure. They apply their methods in differ- ent settings including user study for predicting the language model's behaviour and conclude that contrastive explanations both more intuitive and fine-grained in comparison to non-contrastive ex- planations. We use their methods in our analysis together with (a contrastive version of) LRP (Gu et al., 2018) extended to Transformer models.
Note - Human rationales 전 reasoning: As part of this work, we collect human rationales in the form of highlighting supporting evidence in the text to decide for the gold label; we show an ex- ample in Figure 1. These human rationales should be understood as proxies for how humans (anno- tators) explain (rationalize) given outcome post- hoc, which shall not be conflated with how humans reason, came to a decision, hoch In other words, human rationales can be only be seen as a filtered aftermath of human reasoning. Hence, our obser- vations are only suggestive of how humans explain decisions they are provided with, rather than how they come to make these decisions. The latter could possibly be examined by analyzing physiological signals, e.g., brain stimuli or gaze (eye-tracking). pre-hoc in relation to rationales.
Both aforementioned human evaluation studies differ to our evaluation analysis as they provide ex- planations humans to ask about the model deci-
Contributions The main contributions of this work are: (i) We provide an extensive comparison between contrastive and non-contrastive rationales provided both by humans and models for three dif- ferent model architectures (RoBERTa, GTP2, T5)
/etps::/ggithh coas contras tive-xai

---

prediction: nurse
previously worked abortion
written
experiences
both inside
outside of clinics for Socialist Worker
newspaper.
paychologist
previously worked an abortion provider
written
about
experiences both inside
outside of clinics for Socialist Worker previously
newspaper.
contrastive: nurse vs. paychologist
She
worked
abortion
provider
and
written
about
experiences
bath
inside
outside of
clinics
for
Socialist
Worker
newspaper.
An example biography for 'nurse' from the BIOS dataset highlighted with LRP relevance scores -red for positive, and blue for negative- per class (occupation) based on RoBERTa large. We further show explanations for the foil ('psychologist') ). In the last row, we present an explanation for 'nurse', ; the correct outcome, in contrast to "psychologist', the second best guess of the model.
Figure 2:
sion and the relevance and helpfulness, respectively. In contrast, we aim to analyze whether the ratio- nales provided by humans are closer to contrastive explanations than non-contrastive explanations.
Propagation" (LRP) for Transformer models. To compute explanations that accurately reflect the model predictions, the handling of specific non- linear model components is needed, which includes the layer normalization and attention head modules, that be treated via carefully detaching nodes of the computation graph as part of the forward pass. For non-linear activation functions, i.e., GeLU, we propagate relevance proportionally to the observed activations (Eberle, 2022).
Human rationales vs. model rationales Hu- man rationale annotations often serve as refer-
ence point for evaluating model explanations by directly comparing them to the human ground truth (Schmidt and Bießmann, 2019; Camburu et al., 2018; Young et al., 2020). While high agree- ment between model-based and human explana- tions does not necessarily lead to a faithful model prediction (Rudin, 2019; Atanasova et al., 2020), others argue that providing explanations plausible to humans is crucial when building trustworthy sys- tems (Miller, 2019; Jacovi et al., 2023).
Gradient Input We further compute con- trastive and non-contrastive explanations via 'Gradient x Input' (Baehrens et al., 2010; Shriku- mar et al., 2017), which can be seen special case of LRP without the use of specific propagation rules. In setting this results to no specific treat- ment of non-linear computations, i.e., detaching of non-conserving modules.
3 Methodology
Non-contrastive explanations typically compute the most relevant features for target class label L, e.g., by computing the gradients with respect to the logit (score) of the top-predicted class. Contrastive explanations can in extension be obtained by con- sidering the difference in evidence for the target class L and some foil class (F + L). To compute contrastive and non-contrastive explanations, use the following methods:
Gradient Norm In addition, computing the norm of the gradient (Li et al., 2016) directly has been also considered in the context of contrastive expla- nations by Yin and Neubig (2022).
To obtain contrastive explanations, define the evidence be explained the difference:
w(x) w(x);+
Layer-wise Relevance Propagation Since naive computation of gradients Transformer models has been shown to result in less faithful explana- tions (Ali et al., 2022), we build contrastive expla- nations the framework of "Layer-wise Relevance
where y(x) is the logit (score) of the top-predicted target label by the model and y(æ) is the score for the foil. For generative models, we select the logit of the predicted label, or foil, token.

---

4 Experiments
physician. Figure 1 depicts a specific example in both settings. We collect annotations via Prolific," 3 a croud-sourcing platform. We select annotators with fluency in English and include pre-selection annotation phase for the contrastive setting, where clear guidelines were provided. We use Prodigy." as the annotation platform, and we change partly the guidelines and the framing of the questions. as shown above, between the two (contrastive and con-contrastive) settings. For each example, we have word-level annotations from individuals (an- notators). For further details on the annotation pro- cess and the dataset, see Appendix A. We release the new version of BIOS, dubbed Medical BIOS, annotated with human rationales on HuggingFace Datasets (Lhoest et al., 2021).
4.1 Datasets
In this study, conduct experiments four single-label English classification datasets: SST2 (Socher et al., 2013), DynaSent (Potts al., 2021), BIOS (De-Arteaga et al., 2019), and DBPedia Ani- mals (Lehmann et al., 2015).
SST2 The dataset contains approximately 70,000 (68k train/1k dev/lk test) English movie reviews, each labeled with positive or negative sentiment. To analyze non-contrastive vs. contrastive expla- nations, we use the non-contrastive human ratio- nale annotations provided by Thorn Jakobsen et al. (2023). 2 The 263 samples chosen by Thorn Jakob- sen et al. belong to the development or test split from SST2, and they overlap with the samples from ZuCo, an eye-tracking dataset where reading pat- terns have been recorded from English native speak- ers reading movie reviews from SST (Hollenstein et al., 2018).
DBPedia Animals We consider a subset of the DBPpedia dataset comprising 10,000 (8k train/ Ik dev/1k test) English Wikipedia article abstracts for animal species labeled with the respective biolog- ical class out of 8 classes (amphibian, arachnid, bird, crustacean, fish, insect, mollusca, & reptile). 6
DynaSent This English sentiment analysis dataset contains approximately 122,000 sentences, each labeled as positive, neutral, negative, Thorn Jakobsen et al. (2023)² released non- contrastive annotations for 473 samples from the test set, excluding examples labeled as neutral on the premise that neutral sentiment comes in lack of context, i.e., no evidence of positive or negative sentiment which we use to compare non-contrastive and contrastive model rationales.
4.2 Examined models
We consider three publicly available pre-trained language models (PLMs) covering three different architectures: (i) encoder/(i)) decoder-only, and (iii) encoder-decoder. We use RoBERTa of Liu et al. (2019), GPT-2 of Radford et al. (2019), and T5 of Raffel et (2020) in three different sizes (small, base, and large); we thus test 9 models total.? 7 We fine-tune RoBERTa and GPT-2 using a standard classification head, while we train T5 with teacher-forcing (Williams and Zipser, 1989) as a soguence-to-sequemee model. We conduct grid search to select the optimal learning rate based on the validation performance. We use the AdamW optimizer for RoBERTa, and GPT-2 models, and Adafactor for T5, following Raffel et al. (2020). We use a batch size of 32 examples and train our classifiers up to 30 epochs using early stopping based on validation performance.
The dataset comprises English biographies labeled with occupations and binary genders. This is an occupation classification task, where bias with respect to gender can be studied. We consider subset of 10,000 biographies (8k train/Ik k dev/Ik test) targeting medical occupations (psychologist, surgeon, nurse, dentist, physician).
BIOS
We collect and relcase human rationale annota- tions for a subset of 100 biographies in two differ- ent settings: non-contrastive and contrastive. In the former, the annotators were asked to find the rationale for the question "Why is the person in the following short bio described the gold label occupation, e.g., nurse. In the latter, the question was "Why the person in the follow- ing short bio described as a L rather than F?",
L??,, where L is
http:///ww.pollfficcoo 'https://prodi. gy/
httee////ngggiieeeeoooaaattttttt cc///atasett//costtalcc/// edi edical-bis
where F (foil) is another medical occupation, e.g.,
htttps://huggingg
apediaa-dtaseed
'tttps: / http://uuggnnggg face. htt.:aacc...aa.aaaeeeaeeaaaaaa/e cto/dataset//caastalleee ai air-rationales
"We report the classification performance and the number of parameters per model in Table 2.

---

Cohen's Kappa
model rationales between the contrastive and the non-contrastive settings but we also see corre- lation 0.99 for the continuous values for both datasets. The reason for this might be that in binary classification settings, LRP already considers the only alternative when assigning importance scores, i.e., already computes evidence for one class in contrast to the only other class. For the rest of the paper, therefore focus on the other two datasets which include 5 and 8 classes, respectively.
SST2
Dynasent
contriop
1.00
0.6
contrirp
contrarp
contr.br Arp
paze
contsiop
nan mon-contr.irp
Comparison between model rationales with human annotations for binary sentiment classification on DynaSent and additionally with human gaze on SST2.
Figure 3:
Inter annotatorrrrrroooo agrcement
0.57
0.31
0:44
1.0
4.3 Aggregating rationales
0.9
For all examined datasets (Section 4.1), we con- sider the following aggregation methodology (see Figure 9 in App. A) for human and model ratio- nales, before we proceed with the analysis:
large
Human annotations are aggregated based on a word-based majority vote across all annotators.
Paaro F3
0.
0.85
0.48
0:35
Lapel eapport
2200
1250
1639
1533
2340
1000
Model explanations, computed the sub-word level via an XAI method, are aggregated per word via max-poot max-pooling (Eberle et al., 2022).
Figure 4: Upper: Cohen's Kappa scores for inter- annotator agreement for human rationale annotation (i) averaged across pairwise comparison within non- contrastive setting, (ii) averaged across pairwise compar- ison within contrastive setting, (iii) between contrastive and non-contrastive setting. Lower: Model performance scores (macro FI) for the best model (RoBERTa-large and training support (#samples) across BIOS classes.
When comparing with human rationales, model rationales and gaze are binarized based on the top- k scored tokens, where & is the number of tokens selected in the aggregated human rationales.
5
Results
We first show results for the binary sentiment clas- sification datasets, SST2 and DynaSent, before we dive into the extensive analysis of the human and model-basee rationales of the BIOS dataset.
5.2 Human rationales
Initially, we perform an analysis of the collected human rationales on the BIOS data by comparing the two settings (contrastive and non-contrastive). On average the contrastive rationales are shorter (4 vs. 8 annotated words), which is an indicator of more precise (focused) rationales. This is expected, since the annotators in the contrastive setting were asked to explain the decision for one class in con- trast to another, c.g., 'surgeon' against 'physician'. For instance, in the following example (biography) describing a 'surgeon':
5.1 Sentiment classification tasks
We show results for SST2 and DynaSent in Fig- ure 3 in the form of agreement scores computed with Cohen's 's Kappa for gaze, human annotations and the contrastive and non- contraastee LRP scores for RoBERTa-base. For SST2, we find that gaze shows higher agreement with human annotations than with model rationales (0.31 vs. 0.26) whereas the agreement between annotation and model ra- tionales is even higher (0.46). We see very high agreement (0.99) between contrastive and non-contrastive model rationales. The analysis for DynaSent shows similar results with a lower agreement between annotations and model ratio- nales (0.41). The numbers show an almost per- fect agreement on the binarized versions of the
" "After earning his medical degree, virtually all his training has been concentrated on two fields: facial plastic and reconstructive surgery, and head and neck surgery otherwise known as Otolaryngology"
the terms 'medical degree' and 'Otolaryngology'

---

were both annotated in the non-contrastive anno- tation setting (marked in underline), but not in the contrastive one (marked in bold).
RoBERTa base dentist model non-contr.
physician model non-contr.
model contr non-cona:
In Figure 4, we present the inter-annotator agree- ment measured by Cohen's Kappa within but also between contrastive and non-contrastive human ra- tionales. We observe similar results for all three scores per class with scores ranging from 0.4 to 0.73 for the comparison across contrastive and non- contrastive settings. The class physician shows lowest scores all three comparisons whereas den- tist achieves the highest agreement in all 3 com- parisons. This indicates that the agreement across settings is similar to the agreement within settings and thus the selection of tokens does not necessar- ily differ between contrastive and non-contrastive annotations.
0.49
0.44
human non-corts
0.29
0.41
0.56
0.53
0.42
0.45
GPT2-base
0.5
0.22
0.3
0.34
0.34
0.36
0.39
0.4
human por-contsr.
0.25
0.33
0.27
0.3
0.32
0.37
T5 base
0.1
0.17
0.25
0.4
0.45
0.45
0.45
human
The low agreement score for the class physician can be explained by the lack keywords the bi- ographies, similar to nurse as those professions do not necessarily imply a medical specialization and vary also across countries. The biographies with the label dentist and surgeon often include very clear keywords, some even semantically related the profession, which makes identifying them much easier for humans, as well as for models." the lower part of Figure 4, we also show label support the train data and macroFl-scores for the best-performing model, RoBERTa-large. The Fi-scores show similar distribution across classes where dentist almost reaches perfect accuracy with macroFl -coore of 0.98 and physician with the lowest score of 0.8, In other words, both humans and models face similar challenges.
Figure 5: Agreement between human rationales and brodel-based explanations computed with LRP. Upper: RoBERTa, Center: GPT-2, Lower: T5
based rationales based on the top-k attributed to- kens, where k is the total number of selected tokens in the corresponding human rationale. Considering the agreement across classes, i.e., all in Fig- ure 5, we observe that in most cases contrastive and non-contrastive model- -bsee rationales have similar agreement rate with both contrastive and non-contrastive human rationales (maximum dif- ference is 0.06). In other words, although fewer words are selected by humans in the contrastive setting, the selection of tokens does not seem be heavily influenced by the two different settings for both models and humans. The agreement is substantially lower in the class 'physician', where highly indicative words are not present, as noted earlier. Here, we also see an overall higher agree- ment between non-contrastive model rationales and human rationales (right column of left-most plots). The original claim, that human rationales more similar contrastive than to non-contrastive model rationales (left column vs. right column all sub- plots) is not visible in Figure 5.
5.3 Human vs. model- based rationales
We further proceed with analysis of model-basee rationales compared to human rationales. In Fig- ure 5, we present the agreement between human rationales and model-based rationales computed with LRP for the base version of the three dif- ferent examined models (RoBERTa, GPT-2, and T5). Since LRP provides continuous attribution scores for all tokens, in order to compare with binary human rationales, we binarize the model-
POS analysis To better understand the grammati- cal structure of human and model-based rationales, analyze the part-of-speech (POS) tags of ratio- nales in the BIOS data. While human and model- based rationales are mainly formed by nouns and adjectives, models tend to give more importance to verbs compared to annotators, who barely selected
"Inspecting the human annotations, we observe that spe- cialized words, such as 'dental', and 'surgery' are present and selected across all (100%) examples for dentists, and surgeons, respectively. Contrary for physicians, the generic words 'med- ical', and "medicine" are present in 50% of the relevant exam- ples, and selected in 60-70% of those. For nurses, the word nuussing is present only in 59% of the relevant examples, and has been selected in 100% of those.
"POS tagging is done with spaCy (Honnibal et al., 2020).

---

Model Size
RoBERTa
GPT-2
T5
contrastive roberta- noberta-base
non-contrastive roberta-boe aaarra-basee
BIOS
surgen nurse
0.06
0.03
0.02
0.07
0.03
0.02
Small (S)
0.90
0.76
0.38
0.25
Base (M)
0.88
0.74
0.48
0.01
0.03
0.01
0.01
0.03
0.01
Large (L)
0.93
0.78
0.62
0.20
PlspediaAAnimale
gpt2- baaee
gpt2-
Small (S)
0.97
0.72
0.68
surgeon nurse
0.24
0.09
0.04
0.26
0.05
0.04
Base (M)
0.99
0.70
0.79
Large (L)
0.99
0.86
0.54
0.14
0.04
0.03
0.15
0.05
0.02
Table 1: Spearman correlation between contrastive and non-contrastive model-based rationales on the BIOS (upper part) and OBpedia-Animal (lower part) datasets across all examined models.
t5-base
t5-base
0.05
surgeon nurse
0.00
0.10
0.10
0.05
0.08
0.06
0.04
them as keywords in their explanations. This behav- ior consistent across explainability methods, and both contrastive and non-contrastive explanations.
LRP
Gxl
gradNorm
LRP
Gxi
gradNorm
Figure 6: Relative frequency of gendered words among the top-5 tokens in explanations.
5.4 Model-based rationales
In Table 1, present the Spearman correlation efficients between contrastive and non-contrastive model-based explanations across all models for the full test set of the BIOS and the DBPedia- animals datasets. We observe that overall explana- tions highly correlate, in particular for RoBERTa but also for GPT-2. Large models correlate higher for both datasets, except for DBPedia-animall in T5. This finding suggests that contrastive and non- contrastive model-based explanations do not differ per in the distribution of importance score. We will further look into the selection of tokens and the sparsity the model explanation.
two classes with the highest frequency of gendered words in the explanations. Note that these classes, 'nurse' and 'surgeon', also have the highest gender disparity in the dataset (85% male surgeons and 91% female surgcons). We overall see more gen- dered words in GPT-2 compared to the other mod- els, particularly for explanations computed with LRP. The high dependency on gendered tokens for GPT-2 might be one of the reasons behind the overall lower agreement with human rationales dis- played Figure 5 compared to other models.
Degree of information in explanations To bet- ter understand the differences between contrastive and non-contrastive explanations in both humans and models, we compute entropy to assess sparsity of the respective attributions. Averaged sentence level entropy values human rationales reveal that con-contrastive explanations are less sparse, i.e., their averaged entropy is significantly higher (1.72 vs 1.14, pc0.05). This results sparser human rationales for contrastive explanations indicating that humans do indeed choose relevant tokens more selectively. When looking at model explanations the BIOS dataset, observe different entropy levels across explainability methods, but less so be- tween contrastive and non-contrastive explanations, with the exception of T5 models. These observa- tions provide additional support for findings in text generation that have reported benefits of contrastive
Does gender matter? We extract the top-5 to- kens with the highest importance scores attributed by respective explainability methods for each sam- ple and analyze the amount of gendered words on these tokens." 10 With this, we want to quantify what role gender information plays in the model expla- nations. While human-based rationales do not con- tain words with grammatical gender, we find that models do rely on these tokens when computing cxplanations. We examine the relative frequency of words after aggregating the output tokens (see Sec- tion 1.3)-related, in Figure show results for the base versions of all 3 models and all 3 explainability methods for the
'Male' 'Female'. Heatmaps
'"The gender analysis is based on publicly available lexi- con of gendered words in Fnglish. https://ee thubb...tttt.. con/e cmonsen/ gendered_ sondderedwworr. words.

---

explanations to provide more informative explana- tions for the prediction of the next token compared to non-contrastive methods (Yin and Neubig, 2022). This would also explain lower correlation coeffi- cients for T5 in Table between contrastive and non-contrastive model explanations.
on the BIOS dataset.
On the binary sentiment classification tasks, we see similar agreement scores between non- contrastive human rationales and model explana- tions than for the 5-class classification task on BIOS. The numbers need to be compared care- fully the annotation task was different across the two datasets. For the sentiment classification task, no labels were given a-priori and we only analyze the samples where the true label agrees with the label assigned by the annotator. Further- more, the sentiment classification task is much more subjective than the occupation classification task. Annotators might select tokens differently when they first had to assign label, i.e., first sess the sentence before deciding to which class most likely belongs. Including human gaze into the analysis shows lower agreement with the model explanations in comparison to the human anno- tations. Prior work has shown that human gaze correlates to a higher degree with attention mecha- nisms (Eberle et al., 2022). In general, human gaze could be considered an alternative to human ratio- nales when evaluating model explanations as they provide more information and the task, i.e., read- ing the text, might be more intuitive than assigning rationales afterwards.
Where explanations differ. We show hand- picked example of a contrastive model explana- tion (nurse vs. surgeon) from RoBERTa large in comparison to the non-contrastive explanation and the foil explanation in Figure 2. Here, we clearly see, that (i) the order of the most important tokens changes from the non-contrastive (upper) to the contrastive (lower) explanation. For instance, the word abortion gets the highest score in the con- trastive explanation whereas the word worked is considered most important in the non-contrastive explanation. We also see a more sparse distribution of the importance scores in the contrastive expla- nation than in the non-contrastive explanation. We further look into a possible link between model uncertainty, i.e., how close are the class probabili- ties between the first two classes, and difference in contrastive and non-contrastive explanations, i.e., Spearman correlation coefficient. We find that the two variables highly correlate with each other, in particular for RoBERTa where coefficients range from 0.6 0.73 for LRP. This means, that con- trastive and non-contrastive model explanations are more similar when the model is more certain about the label prediction.
When comparing model-based explanations with each other, we find them to highly correlate be- tween contrastive and non-contrastive settings. In general, our results did not show that contrastive explanations by default more class-specifee in selecting relevant tokens than non-contrastive ex- planations. Our analysis suggests that contrastive explanations are more class-specific, i.c., focus on specifc terms for classes that share a joint set of features (similar tokens) like dentist and surgeon in the BIOS dataset, similar to human rationales. In line with previous work, we have seen that non- contrastive explanations are not necessarily class discriminative and that contrastive explanations can be more class specific but overall share similar fea- tures for similar classes (Gu et al., 2018). While we have observed a strong correlation between model- based contrastive and non-contrastive explanations, qualitative analysis of text samples has in paral- lel provided sensible examples where contrastive explanations do provide more class-specins infor- mation that deviates from the relevant features lected by non non-contrastive methods.
6 Discussion and Conclusion
In this work, we have compared both human and model rationales in contrastive and non-contrastive settings on four English text classification datasets. We find that human rationale annotations agree similar level within than across contrastive and non-contrastive settings but fewer tokens are selected in the contrastive settings (on average vs. 8). This suggests that there not per se dif- ference in token selection for the two settings but tokens are selected more carefully in the contrastive setting. The agreement varies across classes, indi- cating that for more challenging labels the token selection is not straightforward for classes that share specific vocabulary.
We further compare human rationales with model-based explanations and find no difference in agreement between the contrastive and non- contrastive setting for both models and humans
Our findings suggest that contrastive explana-

---

tions in particular useful in generative models. In contrast to the limited differences observed in the context of few-label classification settings typ- ically investigated, our study provides additional evidence supporting the benefit of the contrastive setting for more complex tasks. This is further supported by findings in self-explaining language models where contrastive prompting leads to ex- planations that are preferred by humans over non- contrastive explanations (Paranjape et al., 2021).
to further evaluate their plausibility and usability, i.c,, are they useful for humans to understand the models, see Brandl al. (2022); Yin and Neubig (2022).
Ethics Statement
Broader Impact. We release the first dataset with human annotations in both contrastive and non-contrastive settings in NLP. We hope this in- centivizes other researchers to further look into contrastive XAI methods and to extend this dataset by other languages and tasks.
The subtle differences between contrastive and non-contrastive explanations may provide impor- tant signal for improving ML models. Previ- work has shown how non-contrastive expla- nations provide useful information for debugging and removing undesired model behavior (Anders et al., 2022). In extension, contrastive and non- contrastive explanations could be useful during training to improve robustness of models and avoid shortcut learning behavior by regularizing the model to focus on more class-specific features.
Annotation Process. All participants were in- formed about the goal of the study and the proce- dure at the beginning of the study. They all gave written consent to collect their annotations and de- mographic data. Participants were paid average of 12 GBP/hour. The dataset is publicly available." All answers in the study are anonymized and can- not be used to identify any individual.
Limitations
Potential risk. We do not anticipate any risks in participating in this study. We are aware of potential poor working conditions among crowd- workers"! and try to counteract by paying above- average compensation.
Our analysis is limited to English text classification datasets. In order make more general claims about contrastive explanations, an extension of our analysis to more languages and downstream tasks is needed. The tasks and datasets examined are fur- ther limited to small number of classes, nonethe- less not binary as in prior studies, which may af- feet the efficiency and inherent need for contrastive explanations, since the degree of differentiation between the classes may be too broad, e.g., den- tist and psychologist are two very different medical professions. Experimenting with datasets including hundreds of labels (Chalkidis and Søgaard, 2022; Kementchedjhieva and Chalkidis, 2023), which in many cases are very close semantically, could po- tentially lead to different results.
Acknowledgements
We thank colleagues at the CoAStaL NLP group for fruitful discussions in the beginning of the project. In particular, we would like to thank Jonas Lotz, Qinghua Zhao and Yova Kementched- jhieva for valuable comments on the manuscript. OE received funding by the German Ministry for Education and Research (under refs 011SI 8056A and 011S D11S18022 18025A) and BBDC/BZML and BIFOLD. LC, and IC are funded by the Novo Nordisk Foun- dation (grant NNF 20SA0066568), SB funded by the European Union under the Grant Agree- ment no. 10106555, FairER. Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency (REA). Neither the European Union nor REA can be held responsible for them.
Furthermore, we compare model explanations and human rationales both in a post-hoe way where first a decision has been made and evidence has been collected afterwards. This is briefly discussed in the introduction. We use a limited definition of plausible explanations, i.e., we compare binary hu- man rationale annotations with continuous model explanations which is trivial and we automat- ically filter information when binarizing the model explanations. For a complementary evalu- ation, we would also need to show the collected and computed rationales again to human annotators
"ht //ts::///wwww noemamag.ammmmmm cwtpaag.....ttttttt/oooeeeeeeooe behindd wwwtaaiiiaaliinelllllleaeee

---

References
page 120-128, New York, NY, USA. Association for Computing Machinery.
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. 2022. XAT for transformers: Better explanations through conservative propagation. In Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pages 435-451. PMLR.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani.
Eric Lehman, Caiming Xiong. Richard Socher, and Byron C. Wallace. 2020, ERASER: A benchmark evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online. Association for Computational Linguistics.
Christopher J. Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus-Rober Müller, and Sebas- tian Lapuschkin. 2022. Finding and removing clever hans: Using explanation methods to debug and im- prove deep models. Information Fusion, 77:261- 295.
Amit Dhuramdhar, Pin- Yu Chen, Ronny Luss, Chun- Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based the missing: Towards contrastive explanations with per- tinent negatives. Advances in neural information processing systems, 31.
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classifi- cation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 3256-3274, Online. Association for Computational Linguistics.
Oliver Eberle. 2022. Explainable structured machine learning. Berlin.
Ph.D. thesis, Technische Universität
Oliver Eberle, Stephanic Brandl, Jonas Pilot, and An- ders Søgaard. 2022. Do transformer models show similar attention patterns to task-specific human gaze? In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages -399,, Dublin, Ireland. Association for Computational Linguistics.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Muller, and Wo- jeiech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise rele- vance propagation. PLoS ONE, 10(7):0030140
Jindong Gu, Yinchong Yang. and Volker Tresp. 2018. Understanding individual decisions of cnns via con- trastive backpropagation. Asian Conference on Computer Vision, pages 119-134. Springer.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanahe, Katja Hansen, and Klaus-Robers Müller. 2010. How to explain individual classifica- tion decisions. J. Mach. Learn. Res., 11:1833-8811
Nora Hollenstein, Jonathan Rotsztejn, Marius Troen- dle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific data, 5(1):1-13.
Stephanic Brandl, Daniel Hershcovich, and Anders Sø- gaard. 2022. Evaluating deep taylor decomposition for reliability assessment in the wild. In Proceedings of the Sixteenth International AAAI Conference on Web and Social Media.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Oana-Mare Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. BesacBianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems 31, pages 9539-955000 9549, Curran Associates, Inc.
Aloon Jacovi, Jasmijn Bastings, Sebastian Gehrmann, Yoav Goldberg, and Katja Filippova. 2023. Diag- nosing ai explanation methods with folk concepts of behavior, In Proceedings of the 2023 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 247-247.
Ilsss Chalkidis and Anders Søgaard. 2022. Improved multi-label classification under temporal concept drift: Rethinking group-robust algorithms in label- wise setting. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 2441-2454, Dublin, Ireland. Association for Computational Lin- guistics.
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. 2021, Contrastive explanations for model interpretabilityy In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages 1597-1611, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Maria De-Artcaga, Alexey Romanov, Hanna Wal- lach, Jennifer Chayes, Christian Borgs. Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in high-stakes setting. In Proceedings of the Conference on Fair- ness, Accountability and Transparency, FAT '19,
Yova Kementchedjhieva and Ilias Chalkidis. 2023. An exploration of decoder approaches to multi- label classification for legal and biomedical text. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5828-588888888 2-885 Toronto, Canada. Association for Computational Linguistics.

---

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015. Dbpedia a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195
planations in neural networks. In 2020 IEEE Interna- tional Conference on Image Processing (ICIP), pages 3289-3293. IEFE
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame. Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh. Canwen Xu, Nicolas Patry, Angelina SiMMillan-Mjoor Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac. Thibault Goehringer. Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a umified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Cynthia Rudin. 2019. Stop explaining black box ma- chine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206 215.
Philipp Schmidt and Felix Bießmann. 2019. Quanti- rying interpretability and trust machine learning systems. CoRR, abs/1901.08555
Avanti Shrikumar, Peyton Greenside, and Anshul Kun- daje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learn- ing Volume 70, ICML'17, page 3145-3153.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 681-691, San Diego, California. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Peter Lipton. 1990. Contrastive explanation. Royal Institute of Philosophy Supplement, 27:247 266.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach.
Terne Sasha Thorn Jakobsen, Laura Cabello, and Anders Søgaard. 2023. Being right for whose right reasons? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Pupers), pages 1033-1054, Toronto, Canada. Association for Computational Linguistics.
Tim Miller. 2019, Explanation in artificial intelligence: Insights from the social sciences. Artificial intelli- gence, 1-38.
Ronald J. Williams and David Zipser. 1989. Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2):200- 280.
Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannanch Hajishirzi, and Luke Zeullmoyer. 2021. Prompting contrastive explana- tions for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages 4179 4192. Online. Association for Computational Linguistics.
Kayo Yin and Graham Neubig. 2022. Interpreting lan- guage models with contrastive explanations. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 184-198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Christopher Potts, Zhengxuan Wu, Atticus Geiger. and Douwe Kiela. 2021. DynaSent: A dynamic bench- mark for sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1thh International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 2388-2404, Online. As- sociation for Computational Linguistics.
Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib. 2020. Contrastive ex-

---

A BIOS Annotations
Your task is to find and annotate the words in the bio that answer the following question "Why the person the following short bio described as a Dentist?". In other words, which words can be used a evidence that this person is "Dentist".
We collect and release human rationale annotations for a subset of 100 biographies in two different settings: non-contrastive and contrastive. In the non-contrastive setting, the annotators were asked find the rationale for the question "Why is the person in the following short bio described where the gold label occupation, e-g.. nurse. the contrastive setting, the question was "Why the person the following short bio described as a rather than a F?" where F (foil) is another med- ical occupation, e.g., physician. Figure provides a specific example in both settings.
I??"
You should select words or phrases (multi- word expressions) that answer this specific question. In other words, your selection should be valid, i.e., the words should be re- lated to the given medical occupation and not generic ones.
You should select ALL the words, or phrases (multi-word expressions) that answer this question. other words, your annotation should be complete, and no words that are evidence of the described medical occupation should be left unannotated.
We used Prodigy, as the annotation platform (Figures 7-8) and made some adjustments to the guidelines and the phrasing of the questions. as shown above, between the two (contrastive and non-contrastive) settings.
We collect annotations via Prolific," 13 an online crowd-soureing platform. We compensated all an- notators at an hourly rate of 12f. We select annota- tors with fluency in English through pre-selection annotation phase, where clear guidelines vided. We provided introductory information and guidelines via Google Forms. The guidelines are the following:
Guidelines for constrastive rationales
You are going to annotate 30-35 short biogra- phies from people working in the medical sector. The people described in these doc- uments have one of the following medical occupations: 'Psychologist', 'Nurse', 'Physi- cian', 'Surgeon', or 'Dentist'. Each example is paired with a question.
pro-
Guidelines for non-constrastive rationales
You are going to annotate 30-35 short biogra- phies from people working in the medical sector. The people described in these doc- uments have one of the following medical occupations: 'Psychologist', 'Nurse', 'Physi- cian', 'Surgeon', 'Dentist'. Each example is paired with question.
If you don't 't feel confident about of the aforementioned medical occupations, please advice an online open dictionary, such as the Cambridge English Dictionary, and review the definition and some example sentences, e.g., for surgeon: httttp::/diitiinn httpps////iiiiiiinnnry.cc cambr i idge wwtp///iiciiooorrrgnnniiis///rss n).
If you don't feel confident about one of the aforementioned medical occupations, please advice an online open dictionary, such as the Cambridge English Dictionary, and review the definition and some example sentences, e.g.. for surgeon: (https://dictionar)) cambr dgge ottg//dictiioarry//gnnni////ssss n).
See the following question + biography pair (Test Examples in Figure 8) as an example.
In the first example, the document (bio) de- scribes them 'Surgeon' rather than 'Den- tist'.
Your task is to find and annotate the words in the bio that answer the following question "Why is the person in the following short bio described Surgeon rather than a Dentist?". Imagine you are trying to convince someone and have to find evidence that this person a Surgeon and NOT Dentist (even reality both are true).
See the following question + biography pairs (Test Examples in Figure 7) examples.
In the first example, the document (bio) de- scribes them 'Dentist'.
Phttps::/prodi.gy/ "https::/www.prooll

---

Tasd Exammll 1-
Text Exempi following descrilaed Deniaet"
"Why the loving shart described Dentier?
some relevant queetion)
Complete and virid Selection selected that relevant question)
prodige
prodigy
perve the belwing to descited on
Cbual imsenl Surgrry lather wuth
[Bat remeal onl MMechiitta comed Serain De Slmen
Festor Univensty Medtur Cower they Ingha
offlleed with Batoe Univerity Mecttal Cortor: Bightss
X
(a) Invalid- Partial Annotation
(b) Valid Complete Annotation
Figure 7: Example 1 presented as part of the guidelines for the non- conncrttreee setting.
Test Example 2- "Why is the person in the follown short desoribed so Surgeon Dentist?
"Why person following ahort bed Suroasn
Precise Selection seleoted words that relevant contrnst
rather than Dentise"
accupation B)
(E0 Valid and Preciss Sk action cae selected words that ralevarc: the queatian.)
prodigy
prodigy
Frsence
Whyh perse in sabwving soort so rootteed en "harpeon" "Deetar"
persec the folowing dowrreed "Srgaaer
They 12 war of oupar Thair ceciaabls inclads (Doi caeeal and MaoliiadiiSeggess eseci Schoi Melical
Fil:
Trey foor 47 youn regeeisee thei ramiaa induule Dut
Universtt Nedical Center Drglen.
x
(a) Invalid Annotation
(b) Valid Annotation
Figure 8: Example presented as part of the guidelines for the contrastive setting.
You should select ALL the words, or phrases (multi-word expressions) that answer this question. In other words, your annotation should be complete, and no words that are evidence of the described medical occupation should be left unannotated.
Aggregating Human Annolations to Rationales
A1: She. attended the nursing school
Shec atlendod the. nursing. school
[ Shee attended the, nursing. school
AA:
She attended the nursing school
HR
Aggregating Model XAI Attributions to Rationates
You should select ONLY words or phrases (multi-word expressions) that answer why this person is occupation A, e.g., "Surgeon", and not any words that answer the contrast occu- pation B, e.g.. "Dentist". In other words, your selection should be precise, i.e., the words should be related to the given medical occupa- tion and not the one in contrast.
SW:
She, attend. #ee, the. nurs, ifing school 2.5
MS:
13 0.5 5.4 34 23
AMS I 2.5
13
0.5 54 23
AMR:
Figure 9: Depiction of aggregation methodology for hu- man and model rationales. Notation: A, for the ith anno- tator, AA for the aggregated annotation (rationale), SW for sub-words, MS S for model XAI attribution scores, AMS for word-level aggregated MS, and AM for aggregated model rationale based top-k words.
For the contrastive setting, which we believe is more difficult to understand at first, we included a pre-selection process. We therefore conducted a pilot annotation project for 5 straightforward exam- ples. We selected the annotators based on two cri- teria: (a) manual inspection of their annotations to assess, if they follow the guidelines, (b) computing wiir inter-annotator agreement and excluding annotators with low scores (<0.5 Cohen's Kappa).
of approx. 35 examples each, in the contrastive setting, approx. 100 annotated examples in total. Overall, we have word-level annotations from 3 in- dividuals (annotators) for each example per setting.
Aggregating Rationales In Figure 9, we present an example of the aggregation methodology for human and model rationales.
The selected annotators annotate final subset

---

Model
Classification Performance
Family
Size
Alias
#Params
SST2
DynaSent
BIOS
ABimals
S
MiniLM22-L6xH776
30M
N/A
N/A
0.872
0.976
RoBERTa
M
roberta-base
125M
0.929
0.879
0.880
0.982
L
roberta-large
355M
N/A
N/A
0.892
0.988
S
distil-gpt2
82M
0.867
0.983
GPT-2
M
gpt2
124M
N/A
N/A
0.869
0.983
L
gpt2-M
355M
0.881
0.992
S
t5-v1_1-small
61M
0.886
0.985
T5
M
t5-v1_1 -base
223M
N/A
N/A
0.897
0.989
L
15-vi_1-large
750M
0.887
0.989
Table 2: Test Results (Micro-Fl) for all models (RoBERTa, GPT-2, T5) and all sizes (Small, Base, Large) across all datasets. We also report the number of parameters per model (#Params). Best scores for each model per dataset are underlined.
Additional Results
In Table 2, we report the classification performance for all examined models across all datasets, along- side other model details.


Reviews:
[{"review_id": "iAOLk3JdXz", "reviewer": "Reviewer_beUE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive assessment of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a carefully conducted study that provides new insights on free-form rationales and explanations. Contrary to the common assumptions in the field that contrastive explanations are preferable over non-contrastive explanations, this paper finds that the benefits of contrastive explanations are rather mixed.", "reasons_to_reject": "- I don't have any significant reason to list for rejection; While I personally don't find the conclusion of this paper particularly surprising or ground-breaking per say, the scope and the substance of the work presented here seems solid for acceptance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an analytic investigation into the effectiveness and the practical relevance of \"contrastive explanations\" compared to non-contrastive explanations. While the recent literature has assumed that the contrastive explanations are probably more useful and natural for humans, the empirical findings of this paper reveals that this is not necessarily the case. While the findings and conclusions are not ground-breaking per say, this paper presents a carefully conducted study that other researchers might find insightful.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GnXbV4Qnmx", "reviewer": "Reviewer_vveg", "summary": "", "questions": "A.\tIn Tables 1 and 2, I can see that T5-large has a higher correlation than T5-small on the BIOS dataset, but on the DBpedia-Animals, the situation is quite the opposite. Why does T5 show a different correlation in the model size between the two datasets? Could you somehow give an explanation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we address them below. \n\n> In contrastive settings, the degree of difference between positive entities and negative ones might be a quite sensitive factor for models to provide different rationales. However, I cannot find such an analysis in the paper. Apart from the comparison between contrastive and non-contrastive rationales, I would prefer to see more discussions about the rationale difference between simple contrastive settings, e.g., nurse vs. teacher (easy to distinguish), and hard ones, e.g., nurse vs. doctor (hard to distinguish).\n\nWe agree that contrastive explanations are likely most useful for complex tasks, this motivated us initially to look into medical professions only, instead of all professions originally presented in BIOS. In our analysis, we could see that contrastive and non-contrastive explanations are more similar for confident decisions, i.e., when the probability for the true label is clearly higher than for all other labels which is in line with the reviewer's intuition. We will further highlight this point in the revised version.\n\n> For complex tasks, contrastive settings might show benefits and provide more information for models. I am curious about how the model rationale would change when the label prediction changes by shifting non-contrastive settings to contrastive ones. I suggest adding such case studies, where in one case, the model has different label predictions between contrastive and non-contrastive settings, while in another case, the model gives the same label prediction between two settings. Then, analyze the difference of their rationale changes.\n\nWe compute all explanations post-hoc which means they have no effect on the model prediction and do not provide any information to the model. It would indeed be interesting to see how contrastive rationales influence the outcome of self-explainable model architectures that rely on rationales for the model prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper draws attention to the comparison between contrastive and non-contrastive rationales provided by humans and models, which provides empirical support for the exploration of using contrastive settings to improve models.\n2.\tThe analysis was conducted based on three model architectures and three post-hoc explanation methods on four datasets, which shows the generality of conclusions. \n", "reasons_to_reject": "1.\tIn contrastive settings, the degree of difference between positive entities and negative ones might be a quite sensitive factor for models to provide different rationales. However, I cannot find such an analysis in the paper. Apart from the comparison between contrastive and non-contrastive rationales, I would prefer to see more discussions about the rationale difference between simple contrastive settings, e.g., nurse vs. teacher (easy to distinguish), and hard ones, e.g., nurse vs. doctor (hard to distinguish). \n2.\tFor complex tasks, contrastive settings might show benefits and provide more information for models. I am curious about how the model rationale would change when the label prediction changes by shifting non-contrastive settings to contrastive ones. I suggest adding such case studies, where in one case, the model has different label predictions between contrastive and non-contrastive settings, while in another case, the model gives the same label prediction between two settings. Then, analyze the difference of their rationale changes.\n", "questions_to_authors": "A.\tIn Tables 1 and 2, I can see that T5-large has a higher correlation than T5-small on the BIOS dataset, but on the DBpedia-Animals, the situation is quite the opposite. Why does T5 show a different correlation in the model size between the two datasets? Could you somehow give an explanation?", "paper_topic_and_main_contributions": "The paper mainly analyzes the difference between contrastive and non-contrastive rationales provided by humans and models, using three different model architectures and three different post-hoc explanation methods, on four text classification datasets. The authors find that human annotations agree on a similar level with model-based rationales, in both contrastive and non-contrastive settings. In addition, contrastive human rationales are more specific with fewer words annotated than non-contrastive ones. For model explanations, the rationale difference between the two settings is relatively small except for generation models like T5.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y4tojXcNMm", "reviewer": "Reviewer_i43o", "summary": "", "questions": "1. Can you provide more details on the criteria used to select the three models and three post-hoc explainability methods used in the study?\n2. Have you considered using other models or methods to evaluate the quality of model explanations? If so, why were they not included in the study?\n3. Can you provide a more detailed analysis of the differences between contrastive and non-contrastive explanations, and how they impact the quality of model explanations?\n4. How do you define \"plausible explanations,\" and how did you ensure consistency in the evaluation of model explanations?\n5. Have you considered the impact of different factors (e.g., dataset size, model size, explainability method) on the quality of model explanations? If so, can you provide a more detailed analysis of these factors?\n6. How do you plan to extend this work to other languages and downstream tasks, and what challenges do you anticipate in doing so?\n7. Can you provide a more detailed analysis of the collected human rationale annotations, and how they compare to model explanations?\n8. How do you plan to make the collected human rationale annotations available to the broader research community, and what potential applications do you see for this dataset?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we address your questions below, which also address/cover your comments. \n\n> 1. Can you provide more details on the criteria used to select the three models and three post-hoc explainability methods used in the study?\n\nWe selected these models to cover all widely used transformer-based model architectures (encoder-only, decoder-only, encoder-decoder) in 3 different sizes. We favored variety in these dimensions over different pre-trained models, e.g., BERT and RoBERTa, in similar settings (architecture, size).\n\n> 2. Have you considered using other models or methods to evaluate the quality of model explanations? If so, why were they not included in the study?\n\nAs mentioned above, we cover a wide range of model architectures, in different model sizes, and many post-hoc XAI methods. We believe that our current metric to evaluate explanations is in line with the literature but are happy to extend this if you have any suggestions.\n\n> 3. Can you provide a more detailed analysis of the differences between contrastive and non-contrastive explanations, and how they impact the quality of model explanations?\n\nThis depends on how one defines the quality of model explanations. In this paper, we are interested in the plausibility of explanations, i.e., how well they align with human rationales. We find that they agree on a similar level and discuss this in Section 6 of our paper in more detail.\n\n> 4. How do you define \"plausible explanations,\" and how did you ensure consistency in the evaluation of model explanations?\n\nWe define plausibility as the level of similarity between model explanations and rationales provided by humans and compute the F1 score between the two as has been described in DeYoung et al. (2020). We optimize models under similar settings following a grid search over the main hyper-parameters to guarantee a fair ground for comparison, and we apply all XAI methods in a similar fashion. \n\n> 5. Have you considered the impact of different factors (e.g., dataset size, model size, explainability method) on the quality of model explanations? If so, can you provide a more detailed analysis of these factors?\n\nWe looked into the human-model alignment, i.e., plausibility, for different model sizes and could not find any consistency, e.g., larger/smaller size influencing plausibility. This can however be due to additional confounding factors such as the model classification performance, or how XAI methods are affected by being applied in larger models. We believe that a detailed analysis of all those factors exceeds the scope of the paper and could be included in follow-up work.\n\n> 6. How do you plan to extend this work to other languages and downstream tasks, and what challenges do you anticipate in doing so?\n\nAs NLP research is increasingly moving towards generative instruction-finetuned models, we believe that this should be the main focus for future work.\n\n> 7. Can you provide a more detailed analysis of the collected human rationale annotations, and how they compare to model explanations?\n\nWe provide a detailed description of the rationale annotations in the appendix but are happy to complete it in case something is missing. Figure 3 and Figure 5 show results on the comparison between model rationales and human rationales for all 4 datasets, and we comment on those in the respective sections (5.2 and 5.3).\n\n> 8. How do you plan to make the collected human rationale annotations available to the broader research community, and what potential applications do you see for this dataset?\n\nAnnotations as well as the code for our experiments will be made available upon publication, most likely via HuggingFace for ease of use. We believe the collected rationales can be useful for future work in contrastive explanations as such a dataset does not exist yet but also for investigating bias where gendered attributes are relevant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include its thorough analysis of contrastive explanations in NLP models, its use of multiple datasets and models, and its collection of human rationale annotations for contrastive and non-contrastive settings. The paper also provides valuable insights into how humans and models explain decisions, and highlights the importance of making NLP models more transparent and explainable.\n\nIf this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be:\n\n1. Providing a better understanding of how humans and models explain decisions, which can inform the development of more transparent and trustworthy NLP applications.\n2. Releasing a new dataset with human annotations in both contrastive and non-contrastive settings, which can incentivize other researchers to further look into contrastive XAI methods and to extend this dataset by other languages and tasks.\n3. Demonstrating the effectiveness of multiple explainability methods and model sizes, which can help guide future research in this area.\n4. Highlighting the importance of collecting human rationale annotations for evaluating model explanations, which can improve the interpretability and accountability of NLP models.", "reasons_to_reject": "1. The study only uses three models and three post-hoc explainability methods, which may not be representative of the full range of models and methods used in NLP research.\n2. The study does not provide a detailed analysis of the differences between contrastive and non-contrastive explanations, which may limit the usefulness of the findings for practitioners.\n3. The study does not provide a clear definition of what constitutes a \"plausible explanation,\" which may lead to inconsistencies in the evaluation of model explanations.\n4. The study does not provide a detailed analysis of the impact of different factors (e.g., dataset size, model size, explainability method) on the quality of model explanations, which may limit the usefulness of the findings for future research.\n5. The release of the labeled subset seems to be too small, i.e., 100. Can you elaborate it?\n", "questions_to_authors": "1. Can you provide more details on the criteria used to select the three models and three post-hoc explainability methods used in the study?\n2. Have you considered using other models or methods to evaluate the quality of model explanations? If so, why were they not included in the study?\n3. Can you provide a more detailed analysis of the differences between contrastive and non-contrastive explanations, and how they impact the quality of model explanations?\n4. How do you define \"plausible explanations,\" and how did you ensure consistency in the evaluation of model explanations?\n5. Have you considered the impact of different factors (e.g., dataset size, model size, explainability method) on the quality of model explanations? If so, can you provide a more detailed analysis of these factors?\n6. How do you plan to extend this work to other languages and downstream tasks, and what challenges do you anticipate in doing so?\n7. Can you provide a more detailed analysis of the collected human rationale annotations, and how they compare to model explanations?\n8. How do you plan to make the collected human rationale annotations available to the broader research community, and what potential applications do you see for this dataset?", "paper_topic_and_main_contributions": "This paper is about contrastive explanations in NLP models, and it makes several contributions towards understanding how humans and models explain decisions. The paper addresses the problem of making NLP models more transparent and explainable, which is crucial for building reliable and trustworthy NLP applications. \n\nThe main contributions of the paper include:\n\n1. Analyzing four English text classification datasets and fine-tuning three different models (RoBERTa, GTP-2, and T5) in three different sizes, and applying three post-hoc explainability methods (LRP, GradientxInput, GradNorm).\n2. Collecting and releasing human rationale annotations for 100 samples from the BIOS dataset for contrastive and non-contrastive settings.\n3. Conducting a cross-comparison between model-based rationales and human annotations, both in contrastive and non-contrastive settings, and finding a high agreement between the two settings for models as well as for humans.\n4. Showing that humans do not necessarily explain in a contrastive manner, and that model explanations computed in both settings align equally well with human rationales.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study comparing human and model rationales in both contrastive and non-contrastive settings. The results show that model-based rationales computed in both settings align equally well with human rationales, contradicting a claim in literature that contrastive explanations are more natural for humans than non-contrastive explanations. Overall, the reviewers were impressed by the carefully conducted study on several datasets, architectures, and post-hoc explanation methods. The insights from the study are informative to the community, whereas the dataset used in this paper will be publicly released as a unified benchmark fostering future research. However, the reviewers posted some questions about the justification for the conducted experimental settings and speculated results for alternative settings which the authors answered in the rebuttal.
Decision: Accept-Main