KNOW How to Make Up Your Mind! Adversarially Detecting and Alleviating Inconsistencies in Natural Language Explanations
Year: 2023

Abstract:
While recent works have been considerably improving the quality of the natural language explanations (NLEs) generated by a model to justify its predictions, there is very limited research in detecting and alleviating inconsistencies among generated NLEs. In this work, we leverage external knowledge bases to significantly improve on an existing adversarial attack for detecting inconsistent NLEs. We apply our attack to high-performing NLE models and show that models with higher NLE quality do not necessarily generate fewer inconsistencies. Moreover, we propose an offthe-shelf mitigation method to alleviate inconsistencies by grounding the model into external background knowledge. Our method decreases the inconsistencies of previous highperforming NLE models as detected by our attack.

heading: 1 Introduction
text: The accurate yet black-box nature of deep neural networks has accelerated studies on explainable AI. The advent of human-written natural language explanations (NLEs) datasets (Wiegreffe and Marasovic, 2021) has paved the way for the development of models that provide NLEs for their predictions. However, by introducing an adversarial attack, which we hereafter refer to as eIA (explanation Inconsistency Attack), Camburu et al. (2020) found that an early NLE model (Camburu et al., 2018) was prone to generate inconsistent NLEs (In-NLEs). More precisely, two logically contradictory NLEs generated by a model for two instances that have the same context are considered to form an inconsistency. For example, assume a self-driving car stops in a given traffic environment (the context). If the passenger asks the car Q1:“Why did you stop?”, and it provides NLE1: “Because the traffic light is red.”, and, for the same context, if the passenger instead asks Q2: “Why did you decide to stop here?” and the car provides
NLE2: “Because the traffic light is green”, then NLE1 and NLE2 form an inconsistency.
A model that generates In-NLEs is undesirable, as it either has a faulty decision-making process (e.g., the traffic light was green, so the car should not have stopped), or it generates NLEs that are not faithfully describing its decision-making process (e.g., the car stopped for a red traffic light, but states that it was green) (Camburu et al., 2020). While recent high-performing NLE models have largely improved in terms of the quality (plausibility) of the generated NLEs, to our knowledge, these models have not been tested against generating inconsistent NLEs.
In this work, we first propose a fast, efficient, and task-generalizable adversarial attack that utilizes external knowledge bases. Through experiments on two datasets and four models, we verify the increased efficiency of our approach over the eIA attack, the only inconsistency attack for NLE models, to our knowledge. We also show that the highperforming NLE models are still prone to generating significantly many In-NLEs and, surprisingly, that a higher NLE quality does not necessarily imply fewer inconsistencies. Second, we propose a simple yet efficient off-the-shelf method for alleviating inconsistencies that grounds any NLE model into background knowledge, leading to fewer inconsistencies. The code for this paper is available at https://github.com/MJ-Jang/eKnowIA.

heading: 2 Inconsistency Attack
text: We propose eKnowIA (explanations Knowledgegrounded Inconsistency Attack), which detects more In-NLEs in a faster and more general manner than eIA.

heading: 2.1 Original eIA Attack
text: Setting. Given an instance x, Camburu et al. (2020) divide it into: the context part xc that remains fixed, and the variable part xv that is
540
changed during the attack. For example, xc and xv would be a premise and a hypothesis, respectively, for natural language inference (NLI — detailed below). Let em(x) denote the NLE generated by a model m for the input x = (xc, x̂v). The objective is to find x̂v such that em(x) and em((xc, x̂v)) are logically contradictory (see examples in Table 10). Steps. The eIA attack has the following steps:
1. Train a neural model to act as a reverse explainer, called REVEXPL, that takes xc and em(x) as input and generates xv, i.e., REVEXPL(xc; em(x)) = xv. 2. For each generated NLE em(x): (a) Automatically create a set of statements Ie that are inconsistent with em(x).
(b) For each ê ∈ Ie, generate a variable part x̂v = REVEXPL(xc; ê).
(c) Query m on x̂ = (xc, x̂v) to get em(x̂). (d) Check whether em(x̂) is indeed incon-
sistent with em(x) by checking whether em(x̂) is included in Ie.
Creating Ie. Camburu et al. (2020) used simple elimination of negation (removing “not” or “n’t”) and a task-specific template-based approach for this step. For the template-based approach, they manually create a set of label-specific templates for NLEs such that introducing the instance-specific terms of an NLE from one template into any template from another label creates an inconsistency. They illustrate this process only on the e-SNLI dataset (Camburu et al., 2018), leaving room to question how easily it generalizes to other datasets. e-SNLI contains NLEs for the SNLI dataset (Bowman et al., 2015), where the NLI task consists in identifying whether a premise and a hypothesis are in a relation of entailment (if the premise entails the hypothesis), contradiction (if the hypothesis contradicts the premise), or neutral (if neither entailment nor contradiction hold). Examples of their templates are: “<X> is <Y>” (for entailment) and “<X> cannot be <Y>” (for contradiction). Based on the templates, for a em(x) of “A dog is an animal.”, an inconsistent statement of “A dog cannot be an animal” is obtained (<X> = “A dog”, <Y> = “an animal”). They manually identified an average of 10 templates per label.

heading: 2.2 Our eKnowIA Attack
text: The template-based approach in eIA has two major drawbacks: (1) requires substantial human effort to find an exhaustive set of templates for each
dataset, (2) many different ways of obtaining inconsistencies (e.g., using antonyms) are not taken into account. Moreover, even their negation rule can also be improved. To alleviate these drawbacks, we adopt three rules. Negation. We remove and add negation tokens to negated and non-negated sentences, respectively. To avoid grammatical errors, we add one negation per sentence only if the sentence belongs to one of the following two templates:
• <A> is <B>, <A> are <B> (add “not”), • <A> has <B>, <A> have <B> (add “does/do
not” only if <B> is a noun). Antonym replacement for adjectives/adverbs. We replace adjectives/adverbs with their antonyms from ConceptNet (Speer et al., 2017) (using the NLTK POS tagger). Only one adjective or adverb at a time is replaced for each NLE, to avoid deteriorating the contradictory meaning. Employing other abundant thesauruses could improve our approach, which we leave as future work. Unrelated noun replacement. We replace a noun with an unrelated one, e.g., “human” with “plant”. This is only applied to the noun that is the last word of the sentence, to reduce the possibility of false inconsistencies as the part-of-speech (POS) tagger occasionally made incorrect predictions for words in the middle of a sentence. To get unrelated nouns, we use the DistinctFrom and Antonym relations in ConceptNet. However, we noticed that ConceptNet contains noisy triplets where the subject and object are not antonyms, such as “man” and “people” for “person”1. To avoid these, we created a list (see Table 8 in the appendix) of triplets from ConceptNet to be ignored, by manually investigating a random subset of 3000 detected inconsistencies. While this involved human effort, we highlight that this is due to the nature of ConceptNet and other knowledge bases with more accurate instances may be used instead. However, since we only found eight noisy triplets, we decided to keep ConceptNet, which otherwise worked well for our datasets. Finally, we also noticed that our rules may not lead to In-NLEs if both the context and variable part contain negations. Examples are in Table 9 in the appendix. We filter out such pairs.

heading: 2.3 Experiments
text: Datasets. We consider two tasks: NLI (with the e-SNLI dataset described in Sec. 2.1) and com-
1A pair of words with opposite meanings (from Wikepedia).
monsense question answering (CQA). The CosE 1.0 dataset (Rajani et al., 2019) contains CQA instances formed of a question, three answer candidates, and an NLE for the correct answer. The objective of the Cos-E (Rajani et al., 2019) dataset is to select an answer among the three candidates given a question and to generate an NLE to support the answer. Following Camburu et al. (2020), we set the premise as context and the hypothesis as the variable part for e-SNLI. For Cos-E, to avoid omitting the correct answer, we set the question and the correct answer as the context, and the remaining two answer candidates as the variable part. Just like eIA, our attack is solely intended for detecting In-NLEs and not as a label attack (which may or may not happen). Evaluation metrics. Let Ie be generated at Step 2a for each instance in a test setDtest, and let Is ⊆ Ie be the set of detected In-NLEs (after Step 2d). For each instance, our attack can identify multiple inconsistencies (via multiple variable parts). We, therefore, use two evaluation metrics: hit-rate (Hr) and success-rate (Sr): Sr = Nc/|Dtest| andHr = |Is|/|Ie|, where Nc is the number of unique instances for which the attack identified at least one inconsistency. Intuitively, Sr denotes the ratio of the test instances where the attack is successful, whileHr denotes the ratio of detected In-NLEs to that of the proposed In-NLEs. Models. We consider the following highperforming NLE models, with their implemen-
tation detailed in Appendix A.1: NILE (Kumar and Talukdar, 2020) for NLI, CAGE (Rajani et al., 2019) for CQA, and WT5-base (220M parameters) (Narang et al., 2020) for both tasks. WT5 models with more parameters (e.g., WT5-11B) would require considerably more computing while providing relatively small gains in NLE quality (32.4 for WT5-base vs. 33.7 for WT5-11B (Narang et al., 2020)). Therefore, they are not considered here due to limited computing resources. Implementation details are given in Appendix A.1.

heading: 2.4 Results
text: eKnowIA vs. eIA. We compare eKnowIA with eIA only on the WT5-base model, since eIA requires a prohibiting amount of time. As in Camburu et al. (2020), we manually verified the naturalness of adversarial hypotheses on 50 random samples for each model. Sentences that go against common sense are considered unnatural. Minor grammatical errors and typos are ignored. We observe that 81.5% of the adversarial hypotheses were natural, on average, for each model. Details are in Appendix A.4. The results are summarized in Table 2. The e-SNLI results are adjusted to reflect the proportion of natural adversarial hypotheses by multiplying the number of detected pairs of In-NLEs for each model with the estimated naturalness ratio. For Cos-E, an unnatural variable part would consist of stop words or a repetition of another answer candidate. We automatically found 2 out of 22 examples to be unnatural, which were removed. We observe that eIA generates a tremendous amount of inconsistent candidates (Ie), e.g., 24M for e-SNLI, thus being extremely slow (e.g., 10 days vs. 40 min for eKnowIA), while also obtaining lower Sr and Hr than eKnowIA (e.g., 2.19% vs. 12.88% Sr). eKnowIA on NLE models. The results of eKnowIA applied to NILE, CAGE, and WT5 are in
the upper lines of each block in Table 1. All models are vulnerable to the inconsistency attack. Also, a better NLE quality may not necessarily guarantee fewer inconsistencies. For example, WT5-base has a better NLE quality than CAGE on Cos-E (0.55 vs. 0.43 e-ViL score; see below), but eKnowIA detected more inconsistencies for WT5-base than for CAGE (0.95 vs. 0.42 success rate). Examples of generated In-NLEs are in Table 3. More examples are in Tables 10–12 in Appendix A.7. We observe that the In-NLEs usually contradict common sense, which is aligned with previous studies showing that language models, used as pre-trained components in the NLE models, often suffer from factual incorrectness (Mielke et al., 2020; Zhang et al., 2021).

heading: 3 Our KNOW Method for Alleviating Inconsistencies
text: Our approach for alleviating inconsistencies in NLE models consists of two steps: (1) extraction of knowledge related to the input and (2) knowledge injection. Extracting related knowledge. We leverage a knowledge extraction heuristic proposed by Xu et al. (2021) as follows:
1. Extract entities from an input’s context part. 2. Find all knowledge triplets that contain the
entities. 3. For each entity, calculate a weight sj for each
extracted triplet as:
sj = wj ×N/Nrj and N = ∑K j=1Nrj ,
where wj is the weight of the j-th triplet predefined by the knowledge base (e.g., ConceptNet), Nrj is the number of extracted triplets of the relation rj for the given instance, and K is the total number of triplets containing the entity for the given instance. 4. For each entity, extract the triplet with the highest score. Grounding with the extracted knowledge. After extracting the triplet with the highest weight per entity in an instance, we transform each of them into natural language and concatenate them to the instance. We use “Context:” as a separator between the input and the triplets. We leverage the templates that transform a relation into free-text (e.g., IsA to “is a”) from Petroni et al. (2019).

heading: 3.1 Experiments
text: We apply our KNOW approach to NILE, CAGE, and WT5-base, and name them KnowNILE, KnowCAGE, and KnowWT5-base, respectively.
Inconsistencies. The results in Table 1 show that grounding in commonsense knowledge diminishes the number of In-NLEs for all models and tasks. The KNOW models defended against 58% of the examples attacked by eKnowIA. Also, we observed that, among the inconsistent examples of KNOW models, 20% of them on average were newly introduced instances. Examples that failed to be defended, as well as newly introduced In-NLEs are provided in Tables 15-16 in Appendix A.7. Successfully defended examples are provided in Table 4. More successfully defended examples, nondefended examples, and newly attacked examples can be found in Tables 13-14 in Appendix A.7.
First, we highlight that a successfully defended example means that our eKNowIE attack did not find an adversarial instance together with which the KNOW model would form a pair of In-NLEs, while our attack did find at least one such adversarial instance for the original model. Second, we notice that even when the selected knowledge might not be the exact knowledge needed to label an instance correctly, the model can still benefit from this additional knowledge. For example, in the first sample in Table 14 in Appendix A.7, the most proper knowledge triplet would be {dog, DistinctFrom, bird}. However, despite the indirect knowledge given,2 i.e., {dog, DistinctFrom, cat}, the model is able to defend the In-NLE by inferring that dogs are different from other animals. To examine whether the improved consistency of the KNOW models stems from knowledge leakage (using the same knowledge triplets in the mitigation method as in the attack), we calculate the overlap of triplets. On the e-SNLI dataset, we find that only 0.3% of knowledge triplets are reused for the attack on the KNOW models, and no overlap was found for the Cos-E dataset. This indicates that the leakage is not significant. NLE quality. To evaluate the quality of generated NLEs, we conducted a human evaluation using Amazon MTurk, as automatic evaluation metrics only weakly reflect human judgements (Kayser et al., 2021). We follow the setup from Kayser et al. (2021): we asked annotators (three per instance) to judge whether the generated NLEs justify the answer with four options: {no, weak no, weak yes, yes} and calculated the e-ViL score by mapping them to {0, 1/3, 2/3, 1}, respectively. Details of the human evaluation are in Appendix A.5. In Ta-
2ConceptNet does not contain {dog, DistinctFrom, bird}.
ble 1, the KNOW models show similar NLE quality to their original counterparts, suggesting that our KNOW method preserves NLE quality while decreasing inconsistencies. Similar results are observed on the automatic evaluation of NLEs (see Appendix A.6).

heading: 4 Related Work
text: A growing number of works focus on building NLE models in different areas such as natural language inference (Camburu et al., 2018), question answering (Narang et al., 2020), visual-textual reasoning (Hendricks et al., 2018; Kayser et al., 2021; Majumder et al., 2022), medical imaging (Kayser et al., 2022), self-driving cars (Kim et al., 2018), and offensiveness classification (Sap et al., 2019). Most commonly, the performance of these models is assessed only in terms of how plausible the reasons provided by their NLEs are. To our knowledge, Camburu et al. (2020) is the only work to investigate inconsistencies in NLEs. We improve their adversarial attack as well as bring an approach to alleviate inconsistencies. Works have also been conducted to analyse and make dialogue models generate responses consistent with the dialogue history (Zhang et al., 2018; Welleck et al., 2019; Li et al., 2020). However, these works are difficult to be applied to NLE models, in part because they require specific auxiliary datasets, such as pairs of inconsistent sentences. Other works investigated the logical consistency of a model’s predictions (Elazar et al., 2021; Mitchell et al., 2022; Kumar and Joshi, 2022; Lin and Ng, 2022), but would not have straightforward extensions for investigating NLEs inconsistencies. Besides consistency, NLEs can also be assessed for their faithfulness w.r.t. the decision-making process of the model that they aim to explain (Wiegreffe et al., 2021; Atanasova et al., 2023).

heading: 5 Summary and Outlook
text: We proposed the eKnowIA attack, which is more generalizable, successful, and faster than the previous eIA attack in detecting In-NLEs. Our experiments show that current NLE models generate a significant number of In-NLEs, and that higher NLE quality does not necessarily imply fewer inconsistencies. We also introduced a simple but efficient method that grounds a model into relevant knowledge, decreasing the number of In-NLEs. Our work paves the way for further work on detecting and alleviating inconsistencies in NLE models.

heading: Limitations
text: Our eKnowIA attack contains logical rules designed specifically for the English language. While these rules may apply or be adapted to other languages with simple morphology, there could be languages in which completely new rules may be needed. Both our attack and the KNOW method rely on knowledge bases, which may sometimes be noisy. We employed manual efforts to eliminate (a small number of) noisy triples from ConceptNet. Our attack also relies on a manual annotation to ensure that the adversarial inputs are natural (estimated to be the case 81.5% of the time). Finally, we were not able to test our methods on instances with long text, as we are not aware of datasets with NLEs for long text inputs or long NLEs.

heading: Acknowledgements
text: This work was partially supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1, by the AXA Research Fund, and by the EU TAILOR grant 952215. Oana-Maria Camburu was supported by a Leverhulme Early Career Fellowship. We also acknowledge the use of Oxford’s ARC facility, of the EPSRC-funded Tier 2 facility JADE II (EP/ T022205/1), and of GPU computing support by Scan Computers International Ltd.

heading: A Appendix
text: A.1 Implementation Details
We implemented the WT5-base model based on the HuggingFace transformers package3 and replicated performance close to the reported results (see Section A.3). For the other models, we used the implementations provided by the respective authors. A single Titan X GPU was used.

heading: A.2 Training REVEXPL
text: We adopted T5-base (Raffel et al., 2020) for training the reverse explainer (REVEXPL). We trained the model for 30 epochs with a batch size of 8. For efficient training, early stopping was applied if the validation loss increases for 10 consecutive logging steps, which were set to 30,000 iterations. The dropout ratio was set to 0.1. We used the AdamW optimiser (Loshchilov and Hutter, 2018) with learning rate 1e−4 and epsilon 1e−8. We also used gradient clipping to a maximum norm of 1.0 and a linear learning rate schedule decaying from 5e−5.
For Cos-E, we used 10% of the training data as the validation set, and the original validation set as the test set.

heading: A.3 WT5-base Performance Replication
text: This section describes the performance of our trained WT5-base model. We report the accuracy for measuring the performance on the natural language inference (NLI) and CQA tasks. To automatically evaluate the quality of generated NLEs, we use the BLEU score (Papineni et al., 2002), ROUGE (Lin, 2004), Meteor (Banerjee and Lavie, 2005), and the BERT score (Zhang et al., 2020), which are widely used automatic evaluation metrics. The results are summarised in Table 5. In terms of accuracy and BLEU score, our replication performs better than originally reported for Cos-E, but produced slightly lower results for e-SNLI.
3https://github.com/huggingface/transformers

heading: A.4 Naturalness Evaluation of the Generated Variable Parts
text: It could be unfair to consider that a model generates inconsistent NLEs if the adversarial variable parts are unnatural. Hence, we manually evaluated 50 random samples of generated adversarial variable parts for each model (or all samples when there were less than 50 pairs of inconsistencies found).
On e-SNLI, we observe that, on average, 81.5% (± 1.91) of the reverse variable parts were natural instances, i.e., semantically valid and not contradicting commonsense. The specific figures for each e-SNLI model were 80%, 80%, 84%, and 82% for KnowNILE, NILE, WT5, and KnowWT5, respectively. We adapted the results in Table 1 to reflect the number of inconsistencies caused only by natural variable parts.
For the Cos-E dataset, we considered that the variable parts (the two incorrect answer choices) are unnatural if (1) the answer choices are stopwords of the NLTK package or (2) the correct answer is repeated. We observed only one unnatural case for KnowWT5 and WT5, respectively, and none for the other two models. We eliminated the two cases from the counts.

heading: A.5 Design of Human Evaluation Process for Assessing NLE Quality
text: For the human evaluation, we sampled 200 generated NLEs for each model. Three Anglophone annotators are employed per instance. We selected annotators with a Lifetime HITs acceptance rate of at least 98% and an accepted number of HITs greater than 1,000. However, it is widely known that the quality of MTurk annotation is not guaranteed even for Master workers (Rouse, 2019). When we used the e-ViL evaluation framework off-theshelf (Kayser et al., 2021), we found that many workers do annotations without due consideration by simply checking “yes” in most cases. We also initially obtained an inter-annotator agreement captured by Fleiss’s Kappa (K) of only 0.06 on average for Cos-E, which casted doubt on the quality of the evaluation. This prompted us to add a quality control measure to the evaluation framework. We carefully collected trusted examples where the quality of the NLEs is objectively “yes” or “no”. For each HIT consisting of 10 examples, we incorporated in random locations two trusted examples with the correct answers being “yes” and “no”, respectively. After annotation, we discarded the HITs where
the annotators gave a wrong answer for any of the trusted examples (we consider correct a “weak yes” answer for a “yes” trusted example and a “weak no” for a “no” trusted example). We repeated this process until the number of rejected HITs was fewer than 15% of the total HITs. We achieved an increased K value of 0.46 and 0.34 for e-SNLI and Cos-E, respectively, from 0.35 and 0.06 (without trusted examples). Similar levels of K as ours were obtained in other studies, such as (Marasović et al., 2022; Yordanov et al., 2022).

heading: A.6 Quality Evaluation on the Generated NLEs
text: Table 6 shows the detailed results of human evaluation on the quality of generated NLEs. In addition to the e-ViL score, we followed the evaluation method of Marasović et al. (2020) by merging weak no and weak yes to no and yes, respectively, and reporting the ratios of w/yes and w/no. Also, the results of the automatic evaluation metrics are provided in Table 7. The results show that all the Know-models show similar or better results than their original counterparts.

heading: A.7 Examples
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss our limitations in the "Limitations" section.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1. Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Sections 3 and 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A.1.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Sections 2 and 4.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 2.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Appendix A.1 A. 4.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Appendix A.5.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A.5.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A.5.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 4.1. and Appendix A.5.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix A.5.

heading: Future Work
text: Employing other abundant thesauruses could improve our approach, which we leave as future work. Unrelated noun replacement. We replace a noun with an unrelated one, e. g. , “human” with “plant”. This is only applied to the noun that is the last word of the sentence, to reduce the possibility of false inconsistencies as the part-of-speech (POS) tagger occasionally made incorrect predictions for words in the middle of a sentence. To get unrelated nouns, we use the DistinctFrom and Antonym relations in ConceptNet. However, we noticed that ConceptNet contains noisy triplets where the subject and object are not antonyms, such as “man” and “people” for “person”1. To avoid these, we created a list (see Table 8 in the appendix) of triplets from ConceptNet to be ignored, by manually investigating a random subset of 3000 detected inconsistencies. While this involved human effort, we highlight that this is due to the nature of ConceptNet and other knowledge bases with more accurate instances may be used instead. However, since we only found eight noisy triplets, we decided to keep ConceptNet, which otherwise worked well for our datasets. Finally, we also noticed that our rules may not lead to In-NLEs if both the context and variable part contain negations. Examples are in Table 9 in the appendix. We filter out such pairs