Evaluating the Validity of Word-level Adversarial Attacks with Large Language Models (2024)

Abstract: Deep neural networks exhibit vulnerability to word-level adversarial attacks in natural language processing. Most of these attack methods adopt synonymous substitutions to perturb original samples for crafting adversarial examples while attempting to maintain semantic consistency with the originals. Some of them claim that they could achieve over 90% attack success rate, thereby raising serious safety concerns. However, our investigation reveals that many purportedly successful adversarial examples are actually invalid due to significant changes in semantic meanings compared to their originals. Even when equipped with semantic constraints such as BERTScore, existing attack methods can generate up to 87.9% invalid adversarial examples. Building on this insight, we first curate a 13K dataset for adversarial validity evaluation with the help of GPT-4. Then, an open-source large language model is fine-tuned to offer an interpretable validity score for assessing the semantic consistency between original and adversarial examples. Finally, this validity score can serve as a guide for existing adversarial attack methods to generate valid adversarial examples. Comprehensive experiments demonstrate the effectiveness of our method in evaluating and refining the quality of adversarial examples.

heading: 1 Introduction
text: Despite the success of Deep Neural Networks (DNNs) in various Natural Language Processing (NLP) tasks such as Text Classification, there is a line of research showing the vulnerability of DNNs to adversarial attacks (Alzantot et al., 2018; Ren et al., 2019; Li et al., 2020; Jin et al., 2020). These attacks involve introducing human imperceptible perturbations to original sentences, known as adversarial examples, in order to mislead victim models
† Equal contribution. ∗ Corresponding author.
into making incorrect predictions. According to the definition of a successful adversarial attack, the generated adversarial example must satisfy two re-
4902
quirements. (1) Misclassification: victim model’s prediction should be changed. (2) Semantic Consistency: the semantic meaning should remain sufficiently close to its original sentence.
The most effective adversarial attack against text classification DNNs is word-level attack, which substitutes a few words to generate perturbed adversarial examples, e.g., synonym substitutions (Garg and Ramakrishnan, 2020; Li et al., 2020, 2021; Ren et al., 2019; Alzantot et al., 2018; Yoo and Qi, 2021a). However, according to investigations in this paper, we find that these attack methods often fail to guarantee high semantic consistency and may even result in semantic flipping. As shown in Figure 1, two adversarial examples xvalid and xinvalid are generated over original sample xori with synonymous substitutions. Both of them have altered victim model’s predictions from to , meeting the first requirement, i.e., Misclassification. For the second requirement, i.e., Semantic Consistency, xvalid successfully retains its semantic meaning to xori ( → ), while xinvalid’s meaning significantly changed ( → )1, leaving it an invalid adversarial example and a failed attack.
Numerous sentence embedding tools, semantic metrics and constraints, like grammar checkers, WordNet (Pedersen and Kolhatkar, 2009), USE (Cer et al., 2018), BERTScore (Zhang et al., 2019) and so on have been applied to adversarial attacks, in order to help word-level attack methods satisfy the semantic consistency requirement. However, as illustrated in Table 1, even equipped with constraints and having good performance in semantic metrics (e.g., in terms of BERTScore, all attack methods are ≥ 94%), these attack methods can still generate massive invalid adversarial examples (e.g., the real ASR (ASRh) of BAE (Garg and Ramakrishnan, 2020) is only 11.7% instead of so-called 99.6%) according to the evaluations of human. This observation not only reveals limitations of current adversarial evaluations, but also indicates that existing constraints are not sufficient to help generate valid adversarial examples.
At the same time, we find that the validity results offered by GPT-4 (Achiam et al., 2023) (ASRg) are highly consistent with human evaluations, which suggests the impressive ability of large language models (LLMs) in semantic understanding and evaluation. Based on this finding, we propose a
1That means the ground truth label of xinvalid is actually , and the victim model made a correct prediction of it.
novel method named as Adversarial Validity evaluation with Large Language Models (AVLLM) to address the issue of invalid adversarial examples. Specifically, we first collect a 13K word-level adversarial validity evaluation dataset with rich annotations by GPT-4. Then, with recent advancements of open-source LLMs, we fine-tune a lightweight LLM to provide a validity score for assessing the semantic consistency between original and adversarial examples. Also, Chain-of-Thought (Wei et al., 2022) (CoT) is leveraged to enable our AVLLM offer detailed explanations of semantic differences, making the validity score interpretable. In addition, the lightness of AVLLM makes it possible to be integrated into most existing adversarial attack methods, serving as a plug-and-play module to help generate valid adversarial examples. Extensive experiments across various datasets demonstrate AVLLM’s superior performance in evaluating and refining the quality of adversarial examples.
In conclusion, our contributions are three-fold:
1) Our investigations reveal that many popular adversarial attack methods have serious issues in generating invalid adversarial examples, and existing semantic constraints or sentence embedding tools are not sufficient to help with it.
2) A 13K dataset is curated by leveraging the impressive semantic understanding ability of LLMs for word-level adversarial validity evaluation, which can facilitate the research of textual adversarial attack.
3) We propose AVLLM, as a interpretable metric for adversarial validity evaluations, and as a plug-and-play module to help generate valid adversarial examples. Extensive experiments show its effectiveness in evaluating and refining the quality of adversarial examples.

heading: 2 Related Works
text: Adversarial Attack According to the perturbation grains, adversarial attacks can be categorized into three levels: character-level, word-level, and sentence-level. Character-level attacks often utilize spelling errors to mislead victim models (Gao et al., 2018; Eger et al., 2019; Li et al., 2019). Sentencelevel attacks often use generative adversarial networks or text paraphrase technologies to directly generate adversarial examples (Iyyer et al., 2018; Maheshwary et al., 2021; Lei et al., 2022). However, both character and sentence level attacks can easily cause grammar errors and largely modify the text structure, which make them challenging to maintain the quality of adversarial examples. Most recent research attempts to develop word-level attacks which often use gradient information (Guo et al., 2021; Yuan et al., 2021) or substitution strategies (Alzantot et al., 2018; Garg and Ramakrishnan, 2020; Li et al., 2020; Ren et al., 2019). However, our investigation reveals that word-level methods can generate massive invalid adversarial examples due to semantic consistency issue, which has not been well discussed in previous works.
Adversarial Defense To defend against adversarial attacks, various defense methods have been proposed including data augmentation (Ng et al., 2020; Si et al., 2021; Kober et al., 2021), adversarial training (Wang and Bansal, 2018; Shafahi et al., 2019; Zhu et al., 2020; ?), and reconstruction-based methods (Jones et al., 2020; Xu et al., 2022; Wang
et al., 2023). There is also a line of research that focuses on detection of adversarial examples (Mosca et al., 2022; Huber et al., 2022; Raina and Gales, 2022) Notably, the proposed AVLLM aims at evaluating the validity of adversarial examples and refining their quality, instead of defending against adversarial attacks.
Evaluation of Adversarial Examples Existing word-level adversarial attack methods evaluate the quality of their generated adversarial examples by synonyms and semantic constraints, such as (1) substitution limitations by NLTK (Bird and Loper, 2004), WordNet (Pedersen and Kolhatkar, 2009) and counter-fitting (Mrkšić et al., 2016), (2) sentence similarity scores by modification rate, ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019), (3) embedding distance by USE (Cer et al., 2018) and SimCSE (Gao et al., 2021). However, our investigation reveals that these automatic evaluation metrics are not sufficient to reflect the semantic consistency between original and adversarial examples. Previous studies also conduct human evaluations to assess the quality of adversarial examples (Alzantot et al., 2018; Jin et al., 2020) focusing more on the grammar, fluency and readability instead of semantic consistency which is our focus. A recent line of research attempts to evaluate the semantic consistency by human judges (Morris et al., 2020a; Herel et al., 2022; Dyrmishi et al., 2023) which may introduce human bias and can be considerably costly. In contrast to evaluation by combinations of embedding distance and seman-
tic metrics (Morris et al., 2020a; Chiang and Lee, 2023), we propose to leverage impressive semantic understanding ability of LLMs to provide a interpretable metric for the adversarial validity.

heading: 3 Method

heading: 3.1 Preliminary
text: Given a text classification dataset, each sample is a pair of input text x ∈ X and its corresponding label y ∈ Y , where X and Y denote the input and output space, respectively. We consider a well-trained victim model f that maps X to Y . For an original sample x correctly predicted by the victim model, i.e., f(x) = y, an adversarial attack method may perturb x to generate an adversarial example xadv, which should satisfy the following conditions:
f(xadv) ̸= f(x) and C(xadv, x) ≥ ϵ, (1)
where C is a semantic consistency function (e.g., cosine similarity), and ϵ denotes the threshold differentiating between the original sample and the adversarial example. There are two conditions in Eq.(1): (1) Misclassification: f(xadv) ̸= f(x), and (2) Semantic Consistency: C(xadv, x) > ϵ. Adversarial examples typically satisfy the first condition, but they often fall short of meeting the second condition, which we refer to as “Validity” in this paper.
As listed in Table 2, a generated adversarial example should first maintain its semantic meaning with the original sample, thereby having the same semantic label, i.e., yadv = y. Subsequently, the victim model should make a false prediction on the adversarial example, i.e., f(xadv) ̸= f(x). Once these criteria are met, this adversarial example xadv can be deemed as a successful adversarial example.
3.2 AVLLM
Towards judging whether the adversarial example satisfies yadv = y, prior works often use various semantic metrics mostly relying on embedding distance and similarity (e.g., USE), which are not sufficient as the semantic consistency function C. In this paper, we propose to leverage LLMs to provide a score with an explanation for assessing the validity of adversarial examples. The obtained score can then serve as the semantic consistency function. Figure 2 shows that our method can be divided into 3 parts: pipeline of AVLLM, serving as a validity metric, and serving as a plug-and-play patch.
Note that for integrating into existing word-level attack process, it is necessary to train an opensource, lightweight and specific AVLLM, other than using a closed-source, compute-heavy, and general LLM (such as GPT-4). Compared with calling GPT4 API, adopting AVLLM can save costs, and boost the inference speed due to its fewer parameters.
3.3 Pipeline of AVLLM
Dataset Construction To facilitate research on evaluating the validity of adversarial examples, we collect a dataset consisting of about 13K samples with semantic consistency explanations annotated by GPT-4. The dataset statistics are shown in Table 3. Specifically, we first sample a total of 15K data from 5 popular text classification datasets, including AGNEWS2, IMDB (Maas et al., 2011a), MR (Zhang et al., 2015a), SST-2 (Socher et al., 2013), and YELP3. Considering that the attack and annotation costs are related to the length of sentence, we sampled from these datasets inversely proportional to the length. Then, these samples are uniformly distributed to 3 adversarial attack methods, including TextFooler (Jin et al., 2020), BERT-Attack (Li et al., 2020), and PWWS (Ren et al., 2019) for attacking. We choose BERT (Devlin et al., 2019)4 as the victim model. Through pilot experiments, we observe that GPT-45 is able to provide the semantic consistency score that is highly consistent with human evaluations. Specifically, the score consistency rate between GPT-4 and human with a tolerance of ±1, is 0.99. The Mann–Whitney U test (McKnight and Najab, 2010) also confirms the differences in these scores are not statistically significant with p < 0.05. Thus, we instruct GPT-4 to annotate each adversarial example with a score ranging from 1 to 5, and an explanation as illustrated in Figure 2(a). After filtering out invalid ones, we finally obtain a 13.7K adversarial validity evaluation dataset split into training and validation sets with 12K and 1.7K samples, respectively. More details are in Appendix A.
Fine-tune Recent advancements in open-source LLMs (Touvron et al., 2023; Li et al., 2023; Biderman et al., 2023; Zhang et al., 2024) make it possible to leverage a lightweight LLM for adver-
2http://groups.di.unipi.it/~gulli/AG_corpus_ of_news_articles.html
3https://www.yelp.com/dataset 4https://huggingface.co/google-bert/
bert-base-uncased 5GPT-4-1106-preview is used in this paper.
sarial validity evaluation, which should be more efficient and cost-effective than using GPT-4. In this paper, we choose TinyLlama (Zhang et al., 2024), a compact and open-source 1.1B LLM as our AVLLM. However, smaller LLMs often struggle with complex tasks requiring CoT reasoning, which indicates that TinyLlama cannot provide corresponding explanation e for the validity score VS.
To enhance the interpretability of the score, we propose to fine-tune it on the adversarial validity evaluation dataset for specialized use. We optimize the supervised fine-tuning objective as follows:
L = E log PAVLLM ([e,VS] | [Inst, x, xadv]) , (2)
where Inst is the template shown in Table 11. After fine-tuning, AVLLM is expected to generate an accurate validity score VS to assess the semantic consistency with a detailed explanation. The explanation can not only boost the score accuracy thanks to CoT reasoning, but also help understand the semantic differences between the original and adversarial examples. The score is serving as the semantic consistency function C:
C(xadv, x) = VS(xadv, x). (3)
3.4 AVLLM as a Validity Metric Through our pilot experiments, we find that many purportedly successful adversarial examples are
Algorithm 1 AVLLM as a patch. 1: Input: Text-label pair (x, y), victim model f , semantic
consistency function C, semantic difference threshold ϵ 2: Output: A valid adversarial example xadv 3: Initialization: xadv = x 4: W← ∅ 5: for 1 ≤ i ≤ |x| do 6: w ← important word ranking 7: W←W ∪ {w} 8: end for 9: for each word wj in W do
10: Initiate the set of CANDIDATES by extracting synonyms using different methods 11: for each ck in CANDIDATES do 12: x′ ← Replace wj with ck in xadv 13: if C(xadv, x) ≥ ϵ then 14: if f(xadv) ̸= y then return xadv 15: else 16: Adjusting the ranking of W 17: end if 18: end if 19: end for 20: end for 21: return NONE
not valid due to the semantic consistency issue. To evaluate the real performance of attack methods, we propose to use validity score as an evaluation metric. This evaluation is conducted after the adversarial examples are generated. The revised attack success rate (ASRϵ) is calculated as follows:
ASRϵ = |S ∧ {xadv ∈ S : C(xadv, x) ≥ ϵ}| |{(x, y) ∈ X × Y : f(x) = y}| , (4)
where S is the set of successful adversarial examples before filtering. ϵ is the semantic difference threshold which can be empirically set according to the validity score distribution. The adversarial examples that do not meet this threshold are considered as invalid and failed.
3.5 AVLLM as a Plug-and-play Patch To guide existing attack methods effectively searching valid adversarial examples, we propose to integrate AVLLM into the adversarial attack process, serving as a plug-and-play patch. As outlined in Algorithm 1, most word-level adversarial attack methods can be summarized into two main steps: (1) Word Importance Ranking (L. 3-8). The attack method adopts various ranking strategies to build its word substitution candidate set. (2) Generating Adversarial Example (L. 9-17). The attack method attempts to craft an adversarial example by replacing the word with the highest importance score with its candidate. Unlike existing approaches, the crafted adversarial example is not directly tested by the victim model. Instead, we take a simple
yet effective modification to the attack process (L. 13). The adversarial example is first evaluated by our semantic consistency function with the help of AVLLM, and only if it meets the semantic difference threshold, next steps will be proceeded. The impact of this modification is highlighted. The rejection of invalid adversarial example is expected to adjust the search direction6, thus guiding attack method to generate valid and high-quality ones.

heading: 4 Experiment

heading: 4.1 Evaluation Metrics
text: There are several metrics used in this paper: (1) Original Attack Successful Rate (ASRo) is the number of adversarial examples against the number of correctly predicted samples, without any postrevision. (2) Revised ASRs: ASRh, ASRg and ASRϵ represent original ASRs after filtering out invalid adversarial examples by human, GPT-4, and AVLLM with the threshold ϵ, respectively. (3) USE (Cer et al., 2018), BERTScore (B.S.) (Zhang et al., 2019), SimCSE (Sim.) (Gao et al., 2021) and language modeling Perplexity (PPL) calculated by GPT-2 (Radford et al., 2019) represent semantic metrics which are often used to measure the sentence similarity between original and adversarial examples. (4) Validity Score (VS) generated by
6The detailed adjustment is up to specific attack method.
AVLLM represents the average validity score of adversarial examples. Please refer to Appendix B for more experimental settings.

heading: 4.2 Validity Evaluation Results
text: The goal of validity evaluation is to assess the validity of generated adversarial examples and rebenchmark these attack methods. From results in Table 4, we have two main observations. (1) Same with prior works, ASRo is very high. But there is a significant decrease (ASRϵ) when filtering out invalid examples by AVLLM, even with a tolerant threshold ϵ = 2. This indicates that real attack success rates of these attack methods can be much lower than expected. (2) Existing metrics, such as B.S. and USE, are very high, ranging from 93.5 to 99. However, our VS scores remain below 3 out of 5 in average, as highlighted in gray background, indicating the low validity of generated adversarial examples. This discrepancy suggests that current sentence similarity metrics are not sufficient to effectively measure the semantic consistency between original samples and adversarial examples.

heading: 4.3 Online Attack Results via Patch
text: To test the performance of AVLLM as a patch being integrated into existing attack methods, we conduct experiments evaluating the attack performance after patching, with ASR and various se-
mantic consistency metrics. From Table 5, we find that: (1) Though the ASR after patching shows a decline, it overpasses offline results in Table 4, which indicates our AVLLM can effectively guide the search. (2) Our method consistently boosts nearly all semantic metrics. Meanwhile, as the constraint strength increases (↑ ϵ), the semantic consistency between adversarial examples and original samples becomes better. This shows the effectiveness of our method in helping attack methods generate valid adversarial examples. (3) In terms of ∇PPL, adversarial examples after patching are more grammatically coherent and semantically continuous, suggesting AVLLM as a patch is able to improve the quality of adversarial examples. (4) We also use grammar checker7 to detect grammar errors additionally introduced by adversarial attacks (∇Grammar). The results suggest our method can limit these caused grammar errors in generating adversarial examples, helping maintain the semantic consistency and fluency of the text.

heading: 5 Analysis
text: In this section, we aim to take comprehensive analysis and engage in thorough discussions to study the effectiveness of our method in evaluating and refining the quality of adversarial examples.

heading: 5.1 Credibility of Validity Score
text: We take consistency evaluations in Table 6 to verify the credibility of the proposed validity score. We
7https://github.com/jxmorris12/language_tool_ python
find that exact match rates may not be very convincing, however, note that ϵ representing for the threshold of a valid or an acceptable adversarial example can be empirically tuned8. With a tolerance range of 1, the consistency rates for both GPT-4 and AVLLM compared to human is 0.99, suggesting the high credibility of the proposed validity score. Surprisingly, AVLLM exhibits a higher alignment with human, which indicates the success of our specialized fine-tuning.

heading: 5.2 Comparison with Existing Constraints
text: Table 7 compares the performance of our Validity Score constraint (VS) with existing semantic constraints. The results show that our validity constraint is more effective in helping attack methods generate adversarial examples with high semantic consistency. Besides, we find that for all semantic constraints, the performance gap (ASRo − ASRg) is much lower as the constraints become tighter. In
8That is the reason why this paper often shows experimental results with different ϵ values.
terms of real attack performance (ASRg), our validity score constraint achieves the best performance when the threshold is set to 3. Even we adopt a lightweight LLM as AVLLM, it still brings much computational overhead. We will explore more efficient LLM technologies, such as quantization and parallelism, to improve its speed in the future.

heading: 5.3 Suggestions for Adversarial Attacks
text: In this section, we aim to provide valuable suggestions for both the evaluation of adversarial attacks and designing of attack methods.
Evaluation Evaluation is known to be timeconsuming since word-level attack methods would search the whole space of substitution combinations to find a successful adversarial example, resulting in thousands queries for long sentence. According to Figure 3, most valid adversarial examples can be found within 152 queries if we consider the minimum semantic threshold is 3. Similarly, the modification rate of most valid adversarial examples is 0.31. This finding suggests that the evaluation can save lots of time by limiting the number of queries and the modification rate with appropriate values. Please refer to Appendix C for results of other datasets and attack methods.
Design of Attack Method The used constraints and sources of synonyms are crucial in designing an effective and semantic consistent adversarial at-
tack method. As shown in Table 8, we find that (1) Masked language modeling (MLM) based attack method (e.g., BAE) demonstrates limitations in searching valid adversarial examples that satisfy the semantic consistency requirement, since the objective of MLM is predicting the masked word with most probable words rather than synonyms. For instance, when the attack method are attempting to find synonyms by predicting “[MASK]” in the sentence “I [MASK] this movie”, both "like" and "hate," which are antonyms, are possible predictions. (2) Using WordNet (Pedersen and Kolhatkar, 2009) (e.g., PWWS) or counterfitting (Mrkšić et al., 2016) (e.g., TextFooler) as the source of finding synonym candidates generally have higher validity scores, thus being recommended. (3) More constraints often lead to better performance. Typically, adversarial examples generated by A2T who uses 4 unique constraints has
the highest validity score. This indicates developing more constraints is necessary for designing more powerful attack methods.

heading: 5.4 Case Study
text: The case study on the quality of the generated adversarial examples is shown in Table 9. Compared with directly generated adversarial examples (w/o AVLLM), with the help of AVLLM, the generated adversarial examples tend to have higher semantic consistency with the original samples, which indicates that the quality of adversarial examples can be improved with the proposed patch.

heading: 6 Conclusion
text: In this paper, our investigation first reveals a prevalent issue in word-level adversarial attack methods, wherein many of purportedly successful adversarial examples are actually invalid due to the semantic consistency issue. Building on this finding, we advocate using LLMs to offer an interpretable validity score for assessing the semantic consistency between original sample and adversarial example. To this end, we construct a 13K dataset for adversar-
ial validity evaluation with the help of GPT-4, and then fine-tune a lightweight LLM as AVLLM for saving costs and boosting the inference speed. The proposed AVLLM can not only serve as a validity metric to assess the semantic consistency of adversarial examples, but also serve as a plug-and-play patch to help existing attack methods generate highquality and valid adversarial examples. Through extensive experiments and analysis, our method demonstrates its efficacy and provides valuable insights for advancing research in adversarial attacks.
Limitations
• This paper focuses on word-level adversarial attacks. While these attacks are the most common in this field, the semantic consistency issues of other types of attacks such as sentence-level attack are valuable to study. Besides, the studied types of word-level attack algorithms are basically limited to synonyms substitutions, other types of substitutions like insertion and deletion are not well included.
• We only consider TinyLlama (Zhang et al., 2024) as AVLLM. It is necessary to explore more open-source LLMs to validate the effectiveness of our method. Also for the victim models, currently popular large-scale models like LLaMA (Touvron et al., 2023) are not being tested, while they are important to study especially in the era of LLM.
• According to our experiments in Table 7, our method may bring much computational overhead to the attack process. Recent LLM technologies for efficiency, such as quantization and parallelism will be explored in the future.
Ethics Statement
In our experiment, we utilized models and dataset that are entirely open-source except GPT-4, ensuring transparency and accessibility. Furthermore, our AVLLM incorporates enhanced adversarial attack capabilities, which enables the craft of valid adversarial examples that are more challenge to detect. The adversarial examples with high semantic consistency may bring additional risks to existing DNNs based applications.

heading: Acknowledgments
text: We thank the anonymous reviewers for their insightful and valuable comments. We also extend our gratitude to Yiqiang Li, and the ENLP Laboratory for their assistance and support.

heading: B Detailed Experiment Settings
text: B.1 Datasets
Experiments are conducted on five benchmark classification datasets from phase-level to documentlevel tasks, including AGNEWS (Zhang et al., 2015b), IMDB (Maas et al., 2011b), MR (Pang and Lee, 2005), SST2 (Socher et al., 2013) and YELP (Zhang et al., 2015a).
B.2 Victim Models
Three different types of DNNs are adopted as victim models, including long-short term memory (LSTM) (Hochreiter and Schmidhuber, 1997), BERTBASE (Devlin et al., 2019), and ALBERTBASE-V2 (Lan et al., 2019). LSTM consists of 2 layers of 300-dimensional memory cells with the 300-dimensional pre-trained GloVe embeddings (Pennington et al., 2014). BERTBASE and ALBERTBASE-V2 contains 12 layers of 768- dimensional transformer blocks and one linear layer for classification, but ALBERTBASE-V2 has less parameter than BERTBASE.
B.3 Attack Methods
Five strong word-level adversarial attack methods are employed as attackers. Yoo and Qi (2021b) propose A2T which first use gradient-based word importance ordering and DistilBERT (Sanh et al., 2019) semantic textual similarity constraint. Garg and Ramakrishnan (2020) first compute token importance and then replaces and inserts tokens in the original text by masking a portion of the text
9https://textattack.readthedocs.io/en/latest/ 3recipes/models.html
10https://huggingface.co/textattack
and leveraging the BERT-MLM to generate substitutions. Ren et al. (2019) propose PWWS which considers the word saliency to determine the word modification order for greedy attack. Jia et al. (2019) develop an attack algorithm that exploits population-based gradient-free optimization via genetic algorithms. Jin et al. (2020) first identify the important words and then replace them with the semantically similar and grammatically correct words, named TextFooler.
B.4 Inference Settings
Given that high temperature brings a better alignment with humans, for the stage of using GPT-4 for inference and collecting samples, we use a temperature of 0.95 and a top-p of 1; For the hyperparameters for TinyLlama, the temperature we use is 0.9 and the top-p is 0.95.
B.5 Finetune Settings
For the fine-tuning protocol, we adhered to standard hyperparameters known to yield effective results in model training. We set a training batch size of 16 per device, leveraging gradient accumulation over one step to enhance memory efficiency. A cosine decay strategy was selected for the learning rate scheduler, balancing the need for learning rate reduction over time while maintaining model adaptability. To monitor progress and ensure data integrity, logging and model saving were scheduled at intervals of every 5 and 500 steps, respectively. The initial learning rate was carefully chosen at 5e−5, with the training extending over 6 epochs to thoroughly imbibe the nuances of our 13K dataset into TinyLlama. Additionally, we utilized half-precision floating-point (fp16) training to quicken the training phase without compromising the model’s learning capacity significantly. DeepSpeed optimization was also employed to further enhance training efficiency, reducing computational demand while maintaining high performance.

heading: C Additional Experimental Results
text: C.1 Win-Rate
To evaluate which adversarial attack method have the best performance in remaining semantic similarity. We conduct a comprehensive experiment which evaluate the result of different adversarial attack methods. The first step, for each dataset, we randomly sample 100 test data from the orig-
inal datasets, and then we generated adversarial examples by different adversarial attacks, we use our fine-tuned tiny-llama to score them, in last, we get a score from 1 to 5 for each samples, and we draw win-rate heatmaps Figure 6 to analyze the performance of different adversarial attacks. We can know that, the BAE is the worst adversarial attack method, and the Faster-Alzantot and A2T are strong adversarial attack methods.
C.2 VScore V.S. # Query
Figures 7 and 8 offer deeper insights into how different attack strategies and datasets impact the query number-quality relationship in adversarial text generation. In addition, Figure 7 shows that VScore falls below 2 when queries exceed 274, highlighting the need to assess query volume’s effect on adversarial text generation efficacy. This underscores the importance of detailed analyses across various methods and datasets due to the complexity of adversarial attacks.
Table 10 examines VScore-query number correlations, revealing a nuanced relationship varying by context. In datasets like AGNEWS and IMDB,
a strong correlation between VScore and query volume is evident, indicating that query number markedly affects ASR in these cases.
C.3 VScore V.S. Modification Rate or Number
Figure 10 shows the overall relationship between score and modification rate or number. Furthermore, Figure 11 and 12 reveals a notable trend specific to the AGNEWS, IMDB, and YELP datasets. We observe a negative correlation between VScore and both the modification rate and the number of modifications. This indicates that, as the rate of modification or the total number of modifications in adversarial texts increases, the VScore correspondingly decreases.
This trend underscores the necessity of adopting dataset-specific strategies when launching adversarial attacks. The variation in correlation across different datasets suggests that a uniform threshold for modification rate or number may not be universally effective. Instead, attackers should consider setting distinct thresholds tailored to the characteristics and vulnerabilities of each target dataset.
C.4 VScore V.S. Sentence Similarity Metrics We examined metrics like BERTScore, USE, CSESIM, and MiniLM for sentence similarity assessment to gauge their effectiveness in constraining adversarial texts. Analyzing VScoresentence similarity correlations across granularities (Table 10) revealed generally weak correlations, indicating the limited efficacy of current metrics in capturing the relationship between original and adversarial examples. This suggests a gap in existing metrics’ ability to closely measure adversarial texts against originals. Notably, AGNEWS and IMDB showed stronger correlations, highlighting variable metric performance across datasets.

heading: D Human Annotation Details
text: The annotation is conducted by 5 authors of this paper independently. These annotators are proficient in this domain, with different genders, ages, and educational backgrounds to ensure diversity. They are required to give objective judgments without considering answer lengths to minimize human bias. The prompts for human annotators are the same for GPT-4, which is detailed in Appendix E.

heading: E Prompt Design and Case Study
text: In Table 11, we detail the dataset construction using a prompt-response format. Figure 13 presents a comprehensive prompt, including one-shot and test samples for reference.
As shown in Figure 14, we present a selection of case studies derived from the results of AVLLM, showcasing a range of scores from 1 to 5.

heading: Future Work
text: Recent LLM technologies for efficiency, such as quantization and parallelism will be explored in the future