How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks
Year: 2023

Abstract:
Natural Language Processing (NLP) models based on Machine Learning (ML) are susceptible to adversarial attacks – malicious algorithms that imperceptibly modify input text to force models into making incorrect predictions. However, evaluations of these attacks ignore the property of imperceptibility or study it under limited settings. This entails that adversarial perturbations would not pass any human quality gate and do not represent real threats to human-checked NLP systems. To bypass this limitation and enable proper assessment (and later, improvement) of NLP model robustness, we have surveyed 378 human participants about the perceptibility of text adversarial examples produced by state-of-the-art methods. Our results underline that existing text attacks are impractical in real-world scenarios where humans are involved. This contrasts with previous smaller-scale human studies, which reported overly optimistic conclusions regarding attack success. Through our work, we hope to position human perceptibility as a first-class success criterion for text attacks, and provide guidance for research to build effective attack algorithms and, in turn, design appropriate defence mechanisms.

heading: 1 Introduction
text: Like many other machine learning models, Natural Language Processing (NLP) models are susceptible to adversarial attacks. In NLP, these attacks aim to cause failures (e.g. incorrect decisions) in the model by slightly perturbing the input text in such a way that its original meaning is preserved.
Research has reported on the potential of adversarial attacks to affect real-world models interacting with human users, such as Google’s Perspective and Facebook’s fastText (Li et al., 2019)) More generally, these attacks cover various learning tasks including classification and seq2seq (fake news (Li et al., 2020), toxic content (Li et al., 2019), spam messages (Kuchipudi et al., 2020)), style transfer
(Qi et al., 2021) and machine translation (Michel et al., 2019)).
It is critical to properly assess model robustness against adversarial attacks to design relevant defence mechanisms. This is why research has investigated different attack algorithms based on paraphrasing (Iyyer et al., 2018), character-level (Gao et al., 2018; Pruthi et al., 2019) and word-level (Garg and Ramakrishnan, 2020; Ren et al., 2019) perturbations, and made these algorithms available in standardized libraries (Morris et al., 2020b; Zeng et al., 2021).
For the many NLP systems that interact with humans, we argue that effective adversarial attacks should produce text that is both valid and natural. Validity refers to the property that humans perceive the same semantic properties of interest1 for an adversarial text as for the original text it was produced from. Naturalness refers to the perception that an adversarial text was produced by humans. Adversarial texts that are invalid and/or unnatural can still cause failed NLP model decisions, however, their ultimate effect on humans is negligible because they would fail to convey the intended meaning (e.g. hate speech that is not perceived as hateful) or they would be suspected to be computer-generated (e.g., a phishing email using awkward vocabulary and grammar).
Unfortunately, the scientific literature on adversarial text attacks has neglected (and sometimes ignored) the inclusion of human perception as an essential evaluation criterion – see Table 1. We found that (i) 3 studies do not include humans at all in their evaluation; (ii) merely 12 studies consider naturalness, and they only do so under limited settings. Indeed, these studies involve a single attack, one or two naturalness criteria, less than 10 participants, and they disregard the impact of parameters and factors like perturbation size and language pro-
1In the case of classification tasks, these semantics properties boil down to the class labels.
8822
ficiency. Instead, the studies rely on automated metrics (i.e cosine distance to measure semantic similarity), but these are not suitable proxies for human perception (Morris et al., 2020a).
The absence of systematic analysis of adversarial texts as perceived by humans risks leading to overestimation of their semantic quality and, in turn, to fallacious model robustness assessment and misguidance during the design of defences. This was hinted in the seminal work from Morris et al. (2020a), where a 10-participant survey on one dataset and two attacks revealed a discrepancy between the human-perceived naturalness of adversarial examples.
Therefore, in this paper, we present the first extensive study that evaluates the human-perceived validity and naturalness of adversarial texts. We surveyed 378 participants in assessing, based on five criteria, over 3000 texts (original and adversarial) coming from three datasets and produced by nine state-of-the-art attacks.
Our investigations first reveal that the participants would classify 28.14% of adversarial examples into a different class than the original example. This means that the adversarial perturbations change human understanding of the modified text and, thus, fail to achieve their purpose. Irrespective of the classification task, participants detect 60.3% of adversarial examples as computer-altered; they can even identify 52.38% of the exact altered word. These findings contrast the overly optimistic conclusions regarding attack success rates from previous small-scale human studies. Our results underline that existing attacks are not effective in real-world scenarios where humans interact with NLP systems. Through our work, we hope to position human perception as a first-class success criterion for text attacks, and provide guidance for research to build effective attack algorithms and, in turn, design appropriate defence mechanisms.

heading: 2 Motivation
text: Consider the example of fake news shown in Figure 1b. (“Original”). Ali et al. (2021) have shown that this example is detected by existing fake news detectors based on NLP machine learning models. However, the same authors have also revealed that, if one changes specific words to produce a new sentence (“Adversarial”), the same detector would fail to recognize the modified sentence as fake news. This means that fake news could ultimately reach
human eyes and propagate. Fortunately, fake news – like hate speech, spam, phishing, and many other malicious text contents – ultimately targets human eyes and has not only to bypass automated quality gates (such as detectors) but also fool human understanding and judgment. Indeed, to achieve their goal of propagating erroneous information, adversarial fake news should still relay wrong information – they should be “valid” fake news – and be perceived as a text seemingly written by humans – i.e. they should be “natural”. The fake news example from Figure 1 is unnatural because it uses irrelevant proper nouns like "Slut Tower" or "Donald Hobo" that do not exist in reality, and this makes the fake news ineffective. We, therefore, argue that invalid and/or unnatural examples do not constitute relevant threats.
Thus, the goal of adversarial text attacks becomes to produce examples that change model decision and are perceived by humans as valid and natural. Our study aims to assess, using human evaluators, whether state-of-the-art text adversarial attacks meet this goal. The answer to this question remains unknown today because, as revealed by our survey of existing attacks (see Table 1), only six papers cover both validity and naturalness, five of them do so with less than 10 human participants, and Textbugger (Li et al., 2019) that has the largest number of participants assesses naturalness only at word level, not sentence level. Nevertheless, all these papers evaluate the effectiveness of the specific attack they introduce (rarely with another baseline) and there is a lack of standardized studies considering them all.
For our study, the validity and naturalness requirements led us to consider word-based attacks. Indeed, character-based attacks are easily
detectable by humans and are even reversible with spelling and grammar check methods (Sakaguchi et al., 2017). In word-based attacks, the size of the perturbation δ is typically defined as the number of modified words.

heading: 3 Research questions and metrics

heading: 3.1 Research questions
text: Our study firstly investigates the validity of adversarial examples as perceived by humans.
RQ1 (Validity): Are adversarial examples valid according to human perception?
Validity is the ability of the adversarial example to preserve the class label given to the original text (Chen et al., 2022). Figure 1a) illustrates a case of an invalid adversarial example, which changes the positive sentiment of the original example. Thus, we aim to compare the label that human participants would give to an adversarial example with the label of the original example. To determine the original label, we use as a reference the “ground truth” label indicated in the original datasets used in our experiments – that is, we assume that this original label is the most likely to be given by human evaluators. To validate this assumption, our study
also confronts participants to original examples and checks if they correctly classify these examples (Section 5.1). A statistical difference between humans’ accuracy on adversarial examples compared to original examples would indicate that a significant portion of adversarial examples is invalid.
In addition to validity, we study next the degree to which adversarial texts are natural.
RQ2 (Naturalness): Are adversarial examples natural?
To answer this question, we measure the ability of humans to suspect that a piece of text has been computer altered (with adversarial perturbations). An adversarial example is thus evaluated as less natural, the more it raises suspicion (to have been altered) among the participants.
The suspicion that a text seems computer-altered might arise from different sources, for example the use of specific words, typos, lack of semantic coherence etc. Thus, in addition to evaluating suspiciousness, we refine our analysis in order to unveil some reasons why humans may found an adversarial text to be suspicious. We investigate three additional naturalness criteria:
• Detectability is the degree to which humans
can recognize which words of a given adversarial sentence we altered. High detectability would indicate that the choice of words significantly affect the naturalness of these examples (or lack thereof). We assess detectability in two settings: wherein humans do not know how many words have been altered (unknown |δ|)) and wherein they know the exact number of altered words (known |δ|).
• Grammaticality is the degree to which an adversarial text respects the rules of grammar. The presence of grammar errors in a text might raise the suspicion of human evaluators. However, grammar errors may also occur in original (human-written) text. Therefore, we study both the total number of grammar errors in adversarial examples (“error presence”), and the number of introduced errors compared to original texts (“error introduction”). The latter is a better evaluator for the quality of generated adversarial text. A high relative amount of grammar errors could explain the suspiciousness of the adversarial examples (or lack thereof).
• Meaningfulness is the degree to which the adversarial text clearly communicates a message that is understandable by the reader. We assess the meaningfulness of adversarial text first in isolation (“clarity”)), and then check whether humans believe the meaning of the original text has been preserved under the adversarial perturbation (“preservation”). We hypothesize that adversarial texts with significantly altered meanings are more suspicious.
Finally, because the perturbation size is known to impact success rate and human perceptibility of adversarial attacks in other domains (Simonetto et al., 2021; Dyrmishi et al., 2022), we investigate the relationship between the number of altered words and validity/naturalness.
RQ3: How does perturbation size impact the validity and naturalness of adversarial examples?
Although there is a general acceptance that lower perturbation sizes are preferred, the actual magnitude of the effect that perturbation size causes on text perception has not been studied before.

heading: 3.2 Reported metrics
text: Throughout our study, we compute different metrics for each attack separately and all attacks altogether.
Validity: the percentage of human-assigned labels to adversarial text that match the ground truth provided with the datasets.
Suspiciousness: the percentage of adversarial texts recognized as “computer altered".
Detectability: the percentage of perturbed words in an adversarial text that are detected as modified.
Grammaticality: the percentage of adversarial texts where human evaluators detected present errors (errors introduced by the attack), did not detect or were not sure.
Meaningfulness: the average value of clarity of meaning and meaning preservation, as measured on a 1-4 Likert scale (the Likert scale options are given in Figure 2).

heading: 3.3 Statistical tests
text: To assess the significance of differences we observe, we rely on different statistical tests chosen based on the concerned metrics.
• Proportion tests are used for validity and suspicion, because they are measured as proportions.
• Mann Whitney U tests are used for detectability, grammaticality and meaningfulness because their data are ordinal and may not follow a normal distribution (which this test does not assume). We compute the standardized Z value because our data samples are larger than 30, and the test statistic U is roughly normally distributed.
• Pearson correlation tests are used to assess the existence of linear correlations between the perturbation size and validity/naturalness.
We perform all these tests with a significance level of α = 0.01.

heading: 4 Study design

heading: 4.1 Adversarial texts
text: To generate the adversarial texts presented to participants, we used the TextAttack library (Morris et al., 2020b), which is regularly kept up to date with state-of-the-art attacks, including word-based ones.

heading: 4.1.1 Attacks
text: In total, we used nine word-based attacks from the library. Three of them( BERTAttack (Li et al., 2020), BAE(Garg and Ramakrishnan, 2020), CLARE(Li et al., 2021)) belong to the family of attacks that uses masked language models to introduce perturbations to the original text. Three others (FGA(Jia et al., 2019), IGA(Wang et al., 2019), PSO(Zang et al., 2020)) use evolutionary algorithms to evolve the original text towards an adversarial one. The remaining three (Kuleshov(Kuleshov et al., 2018), PWWS(Ren et al., 2019), TextFooler(Jin et al., 2020)) use greedy search strategies. For all the attacks, we used the default parameters provided by the original authors. We excluded only Hotflip attack because it was not compatible with the latest Bert-based models and Alzantot attack, for which we used its improved and faster version FGA. You can refer to Table 1 for details related to the human study performed by the original authors.

heading: 4.2 Datasets
text: We attacked models trained on three sentiment analysis datasets: IMDB movie reviews (Maas et al., 2011), Rotten Tomatoes movie reviews (Pang and Lee, 2005) and Yelp polarity service reviews (Zhang et al., 2015). We reuse the already available DistilBERT models in the TextAttack library that are trained on these three datasets. Sentiment analysis is a relevant task to assess validity and naturalness, and is easily understandable by any participant, even without domain knowledge. We limited the study to only one task to avoid the extra burden of switching between tasks for the participants. We include this choice in the section Limitations as a study with diverse tasks and datasets would be interesting (i.e datasets with more formal language).
On each dataset, we ran the selected nine wordlevel attacks, which resulted in 25 283 successful adversarial examples in total.

heading: 4.3 Questionnaire
text: We collected the data using an online questionnaire with three parts, presented in Figure 2. The beginning of the questionnaire contains the description of computer-altered text as "a text altered automatically by a program by replacing some words with others". We do not use the term “adversarial examples” to make the questionnaire accessible to
non-technical audiences and avoid biases. We do not provide any hints to participants about the word replacement strategy (i.e. synonym replacement). In addition to this explanation, we clarify to the participants the intended use of the data collected from this study.
The first part of the questionnaire shows examples in isolation and without extra information. It contains questions about validity, suspiciousness, detectability (unlimited choices), grammaticality (presence of grammar errors), and meaningfulness (clarity). We display only one text at a time, and each participant receives five random adversarial texts shuffled with five random original texts. We exclude the five original texts used as the initial point for the adversarial generation process, to ensure that participants do not look at two versions of the same text. Question number 5 on detectability will appear only if the participant answers "computer altered" to question 4.
The second part focuses on detectability (exact number). Adversarial examples and their exact number n of perturbed words are shown, and participants have to choose the n words they believe have been altered. Each participant evaluates four adversarial examples they did not see in the first questionnaire part.
The third part shows original and adversarial examples together. It contains questions about
grammaticality (errors introduction) and meaning (preservation). Each participant sees the same four adversarial examples (s)he had in the second part and their corresponding original examples.
For each participant, we have (randomly) selected the displayed adversarial examples in order to ensure a balance between the different attacks and perturbation sizes. Each participant sees nine adversarial examples in total (one per attack) with different perturbation sizes (chosen uniformly). More details about this distribution are presented in Appendix A.1.

heading: 4.4 Participants
text: In total, 378 adults answered our questionnaire. Among them, 178 were recruited by advertising on private and public communication channels (i.e. LinkedIn, university networks). The rest were recruited through the Prolific crowdsourcing platform. Prolific participants had 80% minimum approval rate and were paid £3 per questionnaire, with an average reward of £9.89/h. All valid Prolific submissions passed two attention checks. For a real-world representation of the population, we advertised the study to targeted English language proficiency levels. As a result, 59 participants had limited working proficiency, 183 had professional proficiency, and 136 were native/bilingual.
You can find the complete dataset with the generated adversarial sentences and the answers from the questionnaire in this link2.

heading: 5 Results and Analysis

heading: 5.1 RQ1: Validity
text: To 71.86% of all adversarial examples, participants have associated the correct class label (according to the dataset ground truth). This contrasts with original examples, which human participants label correctly with 88.78%. This difference is statistically significant (left-tailed proportion test with Z = −12.79, p = 9.92e− 38).
Table 2 shows the detailed human accuracy numbers for each attack separately. Five of the nine attacks exhibit a statistical difference to original examples (the four others have over 80% of correctly labelled adversarial examples, without significant difference with the original examples). Humans have (almost) the same accuracy as random for two of these attacks, ranging between 50 and 60%.
2https://figshare.com/articles/dataset/ACL_ 2023_Human_Study_Adversarial_Text_7z/23035472
Insight 1: Five out of nine adversarial attacks generate a significant portion (>25%) of adversarial examples that humans would interpret with the wrong label. These examples would not achieve their intended goal in human-checked NLP systems.

heading: 5.2 RQ2: Naturalness
text: We report below our results for the different naturalness criteria. The detailed results, globally and for each attack, are shown in Table 3.

heading: 5.2.1 Suspiciousness
text: Humans perceive 60.33% of adversarial examples as being computer altered. This is significantly more than the 21.43% of the original examples that raised suspicion (right-tailed proportion test of Z = 23.63, p = 9.53e−124 ). This latter percentage indicates the level of suspiciousness that attacks should target to be considered natural. A per-attack analysis (see Table 3) reveals that all attacks produce a significant number of examples perceived unnatural, from 46.55% (FGA) to 68.5% (PSO).
Insight 2: Humans suspect that the majority of the examples (60.33%) produced by adversarial text attacks have been altered by a computer. This demonstrates a lack of naturalness in these examples.

heading: 5.2.2 Detectability
text: When humans are not aware of the perturbation size, they can detect only 45.28% of the altered words in examples they found to be computer altered. This percentage increases to 52.38%, when
the actual perturbation size is known (with statistical significant according to a Mann-Whitney U Test with Z = −73.49, p = 4.4e−8). These conclusions remain valid for all attacks taken individually, with a detection rate ranging from 30.3% to 53.2% (δ unknown) and from 39.4% to 65.9% (δ known).
Insight 3: Humans can detect almost half (45.28%) of the perturbed words in adversarial text. This indicates that the perturbations introduced by attacks are not imperceptible.

heading: 5.2.3 Grammaticality
text: Humans perceive grammar errors in 38.9% of adversarial texts and claim that 40.6% of adversarial texts contain errors not present in their original counterparts. Surprisingly, however, humans are more likely to report grammar errors in examples they perceive as original, than in those they deem computer-altered (73.0% versus 44.6%)(4. There is thus no positive correlation between grammaticality and naturalness.
One possible explanation is that human perception of grammar mistakes significantly differs from automated grammar checks. Indeed, the LanguageTool grammar checker (Naber et al., 2003) reports that only 17.7% adversarial examples contain errors, which is significantly less than the 40.6% that humans reported. This teaches us that automated grammar checks cannot substitute for human studies to assess grammaticality.
Humans report varying rates of grammar errors across different attacks. The rates are highest for CLARE (53.8%) which is significantly more than the lowest rate (BERTAttack, 23.7%). Human perception of the grammaticality of the different at-
tacks changes drastically when they also see the corresponding original examples (e.g. BERTAttack has the highest error rate with 55.4%, and CLARE has the lowest with 16.4%), indicating again that this criterion is not relevant to explain naturalness.
Please note that the grammar error presence and introduction are studied in two different settings (ref. section 3.1 and 4.3 ) with different sets of texts, hence can not be compared against each other. We can only comment on the results separately.
Insight 4: Humans perceive grammar errors in 40% of adversarial examples. However, there is no positive correlation between perceived grammaticality and naturalness.

heading: 5.2.4 Meaning
text: Humans give an average rating of 2.60 (on a 1-4 Likert scale) to the meaning clarity of adversarial texts. This is less than original texts, which receives an average rating of 3.44 (with statistical significance based on Mann Whitney U test, with Z = −412.10, p = 1.43e−142). Furthermore, participants have mixed opinions regarding meaning preservation from original texts to adversarial texts (average rating of 2.11) on a 1-4 scale.
To check whether lack of clarity indicates a lack of perceived naturalness, we show in Table 5, for each rating, the percentage of adversarial texts with this rating that humans perceived as computer al-
tered. We observe a decreasing monotonic relation between rating and suspiciousness. This indicates that the more an adversarial text lacks clarity, the more humans are likely to consider it unnatural.
All attacks have an average clarity score ranging from 2.26 (PWWS) to 3.06 (FGA), which tends to confirm the link between naturalness and meaning clarity. Meaning preservation ranges from 1.7 to 2.67. Interestingly, the attacks with a higher preservation rating (FGA, IGA, TextFooler) tends to have a higher validity score (reported in Table2), though Kuleshov is an exception.
Insight 5: Humans find adversarial text less clear than original texts, while clarity is an important factor for perceived naturalness. Moreover, attacks that preserve the original meaning tend to produce more valid examples.

heading: 5.3 RQ3: How does perturbation size impact the validity and naturalness of adversarial examples?
text: Pearson correlation tests have revealed that perturbation size does not affect validity and detectability, but correlates with suspiciousness, grammaticality and meaning clarity. Figure 3 shows the graphs where a correlation was established (the others are in Appendix A.2). Thus, adversarial examples are perceived as less natural as more word have been altered (positive correlation). On the contrary, fewer grammatical errors are reported by humans for higher perturbations. We performed an automated check with Language Tool, which gave the opposite results, more grammatical errors are present for larger perturbations. This again demonstrates the mismatch between human perception or knowledge of grammar errors and a predefined set of rules from automatic checkers. However, as a reminder, error presence is not the most relevant metric when evaluating adversarial text. Error introduction should be considered more important. Finally, adversarial examples with larger perturbation size have less clear meaning and preserve less original text’s meaning.
Insight 6: The perturbation size negatively affects suspiciousness and meaning, and has no impact on validity or detectability.

heading: 6 Misc. results
text: We conducted an analysis to check whether human perception of naturalness and validity is related to their language proficiency. We found out that language proficiency only affects some aspects of naturalness and not validity results. People with professional proficiency are more suspicious, they achieve a higher accuracy at detecting adversarial text compared to the other two groups(64.6% vs 54.8% and 57.0%). Regarding grammaticality, people with higher proficiency level report more added errors to the original examples by adversarial attacks. Lastly, for the meaning preservation there is a statistical difference only between two proficiencies, natives give a lower score compared to limited working proficiency. For detailed results, refer to Table 8 in Appendix .

heading: 7 Discussion and conclusion
text: Our study unveils that a significant portion of adversarial examples produced by state-of-the-art text attacks would not pass human quality gates. These examples are either invalid (labelled differently from intended) or unnatural (perceived as computer altered). This means that the practical success rate of these attacks in systems interacting with humans would be lower than reported in purely model-focused evaluations.
Through our investigations, we discovered that validity is related to the meaning preservation of the original text by adversarial perturbations. As for naturalness, it appears that the detectability of (at least one) altered words, as well as meaning clarity are strong factors determining the suspiciousness of a text to have been computer-altered. The (perceived) presence of grammar errors is not a relevant criterion to determine naturalness. However, grammaticality may still make sense in contexts where exchanged texts rarely contain grammar mistakes (e.g. in professional or formal environments).
More generally, the relevant criteria to evaluate the quality of adversarial examples depend on the considered use case and threat model. Our goal, therefore, is not to qualify an existing attack as “worse than claimed”, but rather to raise awareness that different threat scenarios may require different evaluation criteria. We, therefore, encourage re-
searchers in adversarial attacks to precisely specify which systems and assumptions their study targets, and to justify the choice of evaluation criteria accordingly.
In particular, we corroborate previous studies that discourage the use of automated checks to replace human validation (Morris et al., 2020a). Our study has revealed that human perception of grammaticality does not match the results of grammarchecking tools. We thus argue that humans play an essential role in the evaluation of adversarial text attacks unless these attacks target specific systems that do not involve or impact humans at all.
Interestingly, none of the existing attacks dominate on all criteria. A careful observation of Tables 2 and 3 reveals that six attacks (over nine) lie on the Pareto front (considering our evaluation criteria as objectives). This implies that different attacks fit better in different threat models.
Ultimately, we believe that our results shape relevant directions for future research on designing adversarial text. These directions include further understanding the human factors that impact the (im)perceptibility of adversarial examples, and the elaboration of new attacks optimizing these factors (in addition to model failure). The design of relevant attacks constitutes a critical step towards safer NLP models, because understanding systems’ security threats paves the way for building appropriate defence mechanisms.
Limitations
• Our study focuses on word replacement attacks. While these attacks are the most common in the literature, the human perception of attacks that rely on insertion or deletion can differ from our conclusions.
• While we evaluated three datasets and over
3000 sentences, they all target the sentiment analysis classification task. Muennighoff et al. (2022) have recently released a large-scale benchmark that covers dozens of text-related tasks and datasets that can further validate our study. It would be especially interesting to consider datasets that use more formal language (i.e. journalistic).
• The texts we consider in this study have a maximum length of 50 words. While this allows the evaluation of a higher number of texts, the human perception of perturbations in longer texts might differ.
• We considered a uniform distribution of generated adversarial texts per bin for each attack. However, their real distribution in the wild might differ from our assumed one.
• All our texts and speakers revolve around the English language, while the problems that text adversarial attacks raise (such as fake news and misinformation) are global. Languages where grammar is more fluid, that allow more freedom in the positioning of the words or where subtle changes in tone significantly impact the semantics can open vulnerabilities and hence require further studies.
Ethical considerations
This study investigates perception of humans on adversarial examples, which are modified texts that aim to change the decision of a NLP model. While these examples can be used by malicious actors, our goal is to understand the threat they bring and take informed decisions on preparing effective defences against these threats.
The texts shown to participants of this study were collected from open platforms, and it may
contain inappropriate language. To mitigate this issue, we asked only participants 18+ years old to take the survey.

heading: Acknowledgements
text: Salijona Dyrmishi’s work is supported by the Luxembourg National Research Funds (FNR) AFR Grant 14585105.

heading: A Appendices
text: A.1 Distribution of texts to participants in the study
This study was designed to take into consideration the level of perturbation caused to a text. As such, we use the concept of perturbation bins, which are bins of 5% for the perturbation size. As the maximum perturbation we study is 40%, in total there are 8 bins. FGA and IGA attacks set a maximum perturbation size of 20%, therefore we do not consider higher perturbations for them.
Dataset generation: We split the dataset mentioned in Section 4.1 in two parts: original and adversarial, where the original counterpart of adversarial examples in the adversarial dataset do not intersect with the original sentences in the original dataset. The split is done by randomly selecting first randomly 50 texts for each attack and perturbation bin combination (9x8). In the cases where the attack has generated less than 50 texts in a bin, we take all of those. In total, there were 3168 texts that were added to adversarial dataset. To build the original dataset, we select from the dataset in Section 4.1 the original texts that are not counterparts of the texts in adversarial dataset. Finally, the adversarial dataset was further split in two parts by selecting randomly the examples.
Survey population: We populate the survey step by step starting from Part 1.
Part 1: Original and Adversarial text We select 5 original texts randomly from original dataset. For adversarial texts, we randomly select 5 attack - perturbation bin combinations from all possible combinations. After that, we choose 5 random texts from these 5 attack-bins from one of two sub-adversarial dataset.
Part 2: We select for each of the 4 attacks not present in Part 1 a random perturbation bin. A random text is then picked for the given attack-bin combination from the sub-dataset of the adversarial dataset that was not picked in Part 1.
Part3: The same adversarial texts as in Part 2, joined with their original counterparts.
The full distribution of the texts to participants is illustrated in Figure 4. The distribution of answers per attack and bin is given in Table 6 and 7 .
A.2 Effect of perturbation size Conducting Pearson correlation tests, we found that perturbation size does not affect validity, detectability (unknown and known perturbation size)
and grammar errors introduced by perturbations. Figure 5 shows visually the relationship as well as test statistics.
A.3 Language proficiency effect Table 8 shows the effect of language proficiency in the evaluated metrics for naturality and validity.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Unnumbered section named "Limitations"

heading: 3 A2. Did you discuss any potential risks of your work?
text: Unnumbered section named "Ethical considerations"

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Sec. 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sec 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sec 4.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? There was no intended use described with the original artifacts

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Unnumbered section named "Ethical considerations"

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1, Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sec 1, Sec 4, Appendix
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 4, Ethical considerations

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 4
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 4

heading: Future Work
text: Ultimately, we believe that our results shape relevant directions for future research on designing adversarial text. These directions include further understanding the human factors that impact the (im)perceptibility of adversarial examples, and the elaboration of new attacks optimizing these factors (in addition to model failure). The design of relevant attacks constitutes a critical step towards safer NLP models, because understanding systems’ security threats paves the way for building appropriate defence mechanisms. Limitations
• Our study focuses on word replacement attacks