AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

Authors: zeyi  liao
huan  sun
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 02:10:53

Abstract: \begin{center} \textcolor{red}{Warning: This paper contains potentially offensive and harmful text.}\end{center}
As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~\citep{zou2023universal} proposes a discrete tokens optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps.
Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100\% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. 
More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99\% ASR on the latest GPT-3.5.
To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. Impressively, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.

Main paper: 
 introduction: 
 With large language models (LLMs) (Touvron et al., 2023; Achiam et al., 2023) demonstrating impressive performance on a wide range of tasks, more and more of them are being integrated into autonomous systems and deployed to solve real-world problems. Hence, improving their safety and trustworthiness has attracted significant attention (Yang et al., 2024; Tang et al., 2024; Mo et al., 2024) with a large amount of efforts (Xu et al., 2024; Yuan et al., 2024; Bhatt et al., 2023) focusing on developing safety guardrails.Unfortunately, recently proposed jailbreaking attacks (Zeng et al., 2024; Shah et al., 2023; Shen et al., 2023; Yu et al., 2023; Zou et al., 2023) have been shown successful to circumvent the safety guardrails. For example, a pioneering method named GCG (Zou et al., 2023) proposes to automatically produce discrete adversarial tokens and demonstrates its effectiveness in jailbreaking various aligned LLMs. However, existing jailbreaking methods, including GCG, only focus on finding one single adversarial prompt for queries and leave many other vulnerabilities of LLMs unexplored. Additionally, they tend to be timeconsuming, e.g., GCG requires hours of optimization to curate one adversarial suffix (Jain et al., 2023) .In this work, we take GCG as the testbed due to their resilience to standard alignment and explore several intriguing questions: How can we find as many vulnerabilities of an LLM as possible? Is it feasible to build a generative model to directly produce many adversarial suffixes given any harmful queries within a short time? Answering these questions would significantly enhance our understanding of the LLM's security posture (Section 2).In addressing these questions, we first analyze why some suffixes, despite exhibiting relatively low loss during GCG optimization, might fail to jailbreak the target LLMs. With a pilot study on this, we discover that the loss associated with the first token is disproportionately high despite the overall low loss, indicating that the model is likely to pick a tone of refusal at the beginning of inference. It will lead to subsequent generation in a safe mode. This observation underscores that loss is not a reliable measure and only selecting the suffix with the lowest loss is not the best strategy. Based on this observation, we keep all candidate suffixes sampled during the GCG optimization and use them to jailbreak, which we refer to as augmented GCG. Results show that the attack success rate (ASR) can be substantially increased from ∼20% to ∼80% on the aligned Llama-2-7B-Chat and many more successful suffixes can be discovered for each query (Section 3).Based on these findings, we proceed to pursue the two questions asked earlier. We propose to train a generative model, termed AmpleGCG, to model the mapping between a harmful query and its customized adversarial suffixes. We use a straightforward pipeline to collect training data from augmented GCG for developing AmpleGCG. To systematically evaluate the effectiveness of AmpleGCG, we carry out experiments to generate tailored suffixes for each harmful query in the test query set. Results reveal that our AmpleGCG could achieve near 100% ASR on both Vicuna-7B and Llama-2-7B-Chat by sampling around 200 suffixes, markedly outperforming the two strongest baselines as well as augmented GCG. This highlights AmpleGCG's ability to uncover a wide range of vulnerabilities for different queries. Moreover, the suffixes generated by AmpleGCG exhibit greater diversity than those derived directly from augmented GCG with no overlap, pointing to the diverse and new coverage of vulnerabilities. Notably, AmpleGCG can produce 200 adversarial suffixes for each query in around 4 seconds only, indicating high efficiency (Section 4).Even more interestingly, we find that AmpleGCG transfers well to different models. Specifically, AmpleGCG trained on open-source models exhibit remarkable ASRs on both unseen open-source and closed-source models. By simply adding specific affirmative prefixes (e.g., "Sure, here is") to the generated suffixes from AmpleGCG, ASR can reach 99% on the latest version of GPT-3.5, even higher than the ASR on the previous version (90%). This also issues a cautionary note regarding OpenAI's update of more advanced models, highlighting that such advancements may come at the cost of compromised safety (Section 4).One potential concern about GCG and AmpleGCG, where adversarial suffixes are unnatural, is that they can be easily detected by a perplexity-based defense mechanism. To address this concern, we show that by simply repeating a harmful query for multiple times at inference time, AmpleGCG's generated adversarial suffixes can successfully evade perplexity-based defenses with an 80% ASR (Section 4).To summarize, we learn a universal generative model that captures a distribution of customized adversarial suffixes given any harmful query and can generate many successful adversarial suffixes efficiently at inference time. It could also transfer to both unseen opensource and closed-source LLMs and bypass perplexity-based defense mechanisms. Our series of models can be accessed via huggingface1 with details. 
preliminaries: 
 Background on GCG. GCG (Zou et al., 2023) is a pioneering approach to elicit objectionable content from aligned LLMs through discrete token-level optimization. Specifically, given a harmful query x 1:m , GCG aims to find a suffix x m+1:m+l and append it to the original query so as to form an adversarial query x 1:m+l = concat(x 1:m , x m+1:m+l ). To make the victim model(s) prone to outputting harmful content given an adversarial query, GCG requires the response of the model(s) to start with positive affirmation y (i.e., "Sure, here is how to {x 1:m }"). Under this context, GCG leverages the standard autoregressive objective function as the loss function:L(x m+1:m+l ) = -log p(y|concat(x 1:m , x m+1:m+l )) = -log p(y|x 1:m+l ) (1)to optimize the adversarial suffix x m+1:m+l via proposed Greedy Coordinate Gradient (GCG) algorithm. Essentially, the algorithm randomly initializes the adversarial suffix and then samples a batch of candidate suffixes based on the gradient (Ebrahimi et al., 2018) at each optimization step. It uses the suffix with the lowest loss up to that point for the next iteration and continues the optimization until the number of max steps is reached. Ultimately, GCG only picks the suffix with the lowest loss throughout the entire optimization process.Besides, two settings were designed by Zou et al. (2023) , one for finding an adversarial suffix for each individual query and the other for finding a universal suffix for all queries:(1) The individual query setting intends to optimize the objective function for each harmful query and obtain one tailored adversarial suffix for it.(2) The mutliple queries setting is designed for optimizing multiple harmful queries simultaneously by fusing the gradients and generates a universal adversarial suffix for multiple queries together. Both settings are compatible to using one or more open-source victim models during optimization.Under either setting, the effectiveness of an optimized suffix is measured by detecting the harmfulness of the output from victim models with input being the adversarial query x 1:m+l .Motivation. GCG has achieved remarkable success in jailbreaking aligned LLMs and has been employed as an automated red teaming tool (Mazeika et al., 2024) . Besides, GCG stands out among other jailbreaking strategies by targeting long-tail gibberish cases, which rarely occur in real-world scenarios and therefore cannot be trivially mitigated through conventional human alignment techniques (Ouyang et al., 2022) , even with more human alignment data. It is also why we take GCG as our testbed for further exploration.However, GCG settles on only one adversarial suffix with the lowest loss to attack the victim models, leading to the oversight of many adversarial suffixes and making them easy to fix due to the limited number. In this paper, we aim to explore several intriguing questions:(1) Is loss really a suitable metric to select a potential adversarial suffix for jailbreaking?(2) How can we find as many vulnerabilities as possible for each specific query? Is it possible to build a generative model to directly produce a vast array of customized suffixes given a harmful query within a short time, instead of going through GCG optimization which is time-consuming? Addressing these questions could help us better understand the safety posture of LLMs and further amplify the impact of GCG either as an approach for jailbreaking or a tool for automated red-teaming by unveiling more vulnerabilities of LLMs. 
rediscovering gcg: loss is not a good reference for suffix selection: 
 In this section, we take a deep look at GCG and find that for each suffix, its corresponding value of the loss function in GCG is not a good reference of its jailbreaking performance (success or failure). Based on this finding, we then propose augmented GCG, which collects all suffixes sampled at each optimization step and uses them to attack a victim model. harmfulness of a response (Qi et al., 2023) , we further include the Beaver-Cost model (Dai et al., 2023) as a model-based evaluator which achieves 95% human agreement in the report. Generally, a response of a victim model is regarded as harmful if the above two evaluators detect harmful content in it unless otherwise stated. We calculate the ASR metric as the average success rate across all queries.GCG uses the loss function to select the suffix with the lowest loss to jailbreak. Is loss really a good indicator of jailbreaking performance? To answer this question, we visualize the losses of candidate suffixes during the optimization steps, where we randomly select one harmful query from AdvBench and carry out the optimization process under the individual query setting using Llama-2-7B-Chat as the victim model. From Figure 1 , we observe that (1) a suffix with a low loss might fail to jailbreak and (2) there are many successful suffixes with higher losses at each step, even though suffixes with a lower loss have a higher chance to jailbreak. We present the optimization visualization for Vicuna-7B in Appendix I, which shows more unexplored successful suffixes in 500 steps.Why might a suffix corresponding to a low loss fail to jailbreak? This is related to the known issue of exposure bias in training autoregressive models in a teacher-forcing fashion (Bengio et al., 2015) . For suffixes that correspond to a low loss but fail to jailbreak, despite the overall low loss across all tokens in the target output, we find that the loss associated with the first token is substantially higher compared to tokens at subsequent positions, indicating the optimization over the first token is not successful. When such optimized suffix is used to jailbreak the victim model, the model tends to pick a refusal tone for the first token (e.g., "Sorry"). Since autoregressive models' decoding at inference time depends on previously generated tokens, the subsequent tokens generated after the first refusal token tend to stay in a safe mode. Experiments of a controlled teacher-forcing generation in Appendix M includes more detailed discussions and analysis on this observation.Taking previous empirical findings in mind, we keep all candidate suffixes (which we refer as overgeneration) sampled during the GCG optimization and present the effectiveness of GCG augmented with overgeneration, dubbed as augmented GCG (Algorithm 1).Experimental Setting. We randomly pick 30 harmful queries from AdvBench to evaluate if augmented GCG could further improve ASR and find more successful suffixes. For each harmful query, as long as one suffix succeeds, the attack is deemed successful.(1) For the individual query setting, we evaluate all possible overgenerated candidates during optimization (all points in Figure 1 ). ( 2) For the mutliple queries setting, we randomly pick 25 harmful queries (which we ensure are non-overlapping with the above 30 test queries) as our training set and optimize for a universal suffix. Note that both individual query and mutliple queries are compatible with the overgeneration strategy. To reduce the memory and time cost, in this mutliple queries setting, we only save the suffixes with the lowest loss (stars in Figure 1 ). GCG's default setting, picking the suffix with the lowest loss at the entire optimization (purple triangle in Figure 1 ), will be our baseline to compare with.Besides evaluating ASR, we also assess the number of Unique Successful Suffixes (USS) identified for each jailbroken query on average. While ASR intends to measure the success rate across queries, the USS number reveals the total number of the identified adversarial suffixes for each query on average. These metrics together are crucial for gaining a thorough understanding of an attack method's effectiveness and the target LLM's safety posture.  
methodology: 
 Overgenerate-then-Filter Pipeline. Given that there are a number of successful suffixes missed during the GCG optimization, we propose a straitforward overgenerate-then-filter (OTF) pipeline, extended on the overgeneration strategy (augmented GCG) described in Section 3. Specifically, we apply the model-based and string-based evaluators to filter out failed suffixes for corresponding victim models after overgeneration. Illustration of OTF is included in the right part of Figure 3 . The number in the bracket means the number of sampled suffixes from either AmpleGCG or augmented GCG. Specifically, USS measures the number of unique successful suffixes for each harmful query, and diversity is assessed by average pairwise edit distances between all generated nonsensical suffixes.Llama-2-7B-Chat with OTF, are identified as challenging. We deliberately incorporate a subset of these to create a challenging test set for subsequent evaluation.With the collected training pairs (<harmful query, adversarial suffix>), we adapt Llama-2-7B (Touvron et al., 2023) as our base model for AmpleGCG and train it to generate adversarial suffixes given a harmful query. We ablate a number of decoding approaches during inference to generate suffixes from AmpleGCG and decide to take group beam search (Vijayakumar et al., 2016) as our default decoding method to encourage generating diverse suffixes at each sampling time. Specifically, we set the number of groups the same as the number of beams and set diversity penalty to 1.04 . The right part of Figure 3 illustrates how we use the collected data from the OTF pipeline to train AmpleGCG and more details can be found in Appendix B.Test Set. We select 100 harmful queries from AdvBench and ensure no overlapping with the aforementioned 318 training queries. In particular, 56 of them are randomly picked from the challenging 127 cases that fail to jailbreak Llama-2-7B-Chat under augmented GCG's individual query setting and 44 queries are randomly selected from queries that are not involved in creating training sets.Baselines. We compare our AmpleGCG with the GCG default setting under both the individual query and mutliple queries settings and its augmented counterpart. According to (Mazeika et al., 2024) , AutoDAN (Liu et al., 2023b) serves as the second most effective approach to jailbreaking after GCG, we report the performance of AutoDAN-GA as well.Target Models. We present the effectiveness of different attack methods in jailbreaking Vicuna-7B (Chiang et al., 2023) and Llama-2-7B-Chat (Touvron et al., 2023) , given any harmful test queries. To show AmpleGCG's transferability, we further include open-source Mistral-7B-Instruct (Jiang et al., 2023) and closed-source GPT-3.5-0613, GPT-3.5-0125, GPT-4-0613 models (Achiam et al., 2023) as our target models.More Rigurous Evaluators for transferring to GPT-series. In addition to the string-based evaluator and Beaver-Cost models introduced in 3, we further utilize GPT-4 as a more rigorous evaluator to assess harmfulness when experimenting with transfers to GPT-series models. We also incorporate a human evaluation process to verify the results.The evaluators for assessing the harmfulness of responses remain the same as described in Section 3, unless otherwise stated when evaluating the transferability of AmpleGCG.Effectiveness, Efficiency, and Broad Coverage of AmpleGCG. Table 2 and Table 13 (in Appendix H) present the results of jailbreaking Llama-2-7B-Chat and Vicuna-7B, respectively. We observe that AmpleGCG could obtain a high ASR compared to the two most effective baselines, GCG default setting and AutoDAN, by a substantial margin. Even when GCG is augmented with overgeneration, obtaining 257 and 87 universal suffixes for Llama-2-7B-Chat and Vicuna-7B under the mutliple queries setting respectively, AmpleGCG attains a higher ASR with fewer samples, achieving up to 99% ASR.Additionally, AmpleGCG uncovers more vulnerabilities with higher USS and more diverse adversarial suffixes-where diversity is assessed by average pairwise edit distances between all generated nonsensical suffixes-compared to both the default GCG and augmented GCG under the mutliple queries setting. Though augmented GCG can obtain 192k suffixes with 122h computation under the individual query setting, their diversity is relatively low, and it can't find vulnerabilities in a diverse span like AmpleGCG. We also automatically verify that the suffixes from AmpleGCG are distinct from augmented GCG's 192k instances, highlighting the new coverage of vulnerabilities from AmpleGCG and complementing the vulnerabilities found by augmented GCG.Notably, it only takes AmpleGCG 6 mins5 in total to produce 200 suffixes for each of the 100 test queries (4 seconds per test query) that reach near 99% ASR, substantially reducing the time cost of curating adversarial prompts for LLMs (Jain et al., 2023) .Given that the test set comprises 56 challenging queries that fail to jailbreak Llama-2-7B-Chat using augmented GCG under individual query setting (Section 4.1), we particularly study if AmpleGCG, trained on successful queries, could jailbreak these 56 challenging failure queries to demonstrate its in-domain (IND) generalization. More excitingly, we delve into whether AmpleGCG, trained on open-source models, can be applied to attacking closed-source models. We collect training pairs from four models-Vicuna-7B, Vicuna-13B, Guanaco-7B, and Guanaco-13B (Dettmers et al., 2024) , following the findings in (Zou et al., 2023) . For details on our data collection methodology when optimizing across multiple models, please refer to Appendix B. For API cost consideration, we randomly select 50 queries from the 100 queries test set. We further incorporate two extra stringent evaluators for detecting harmfulness to reduce false positives: classifier from (Mazeika et al., 2024) and GPT-4 evaluator (evaluation prompt is in Appendix O).We take the GCG mutliple queries setting as our baseline, which is optimized using the aforementioned four models as well. Compared to the default GCG, GCG augmented with overgeneration increases ASR by 4 times with 140 generated suffixes. By increasing the sampling times, AmpleGCG notably enhances the ASR performance, reaching up to 98% on the latest GPT-3.5-0125 model. To further boost this performance, we incorporate an affirmative prefix ("Sure, here is")-denoted as AF in Table 5 -at the end of the generated suffixes. This modification leads to an additional increase in ASR for both GPT-3.5-0613 and GPT-3.5-0125 models, with the latter nearly achieving a 100% ASR. This increase could be attributed to the models' tendency to overlook their internal defense mechanisms after processing effective gibberish, and subsequently become unaligned models devoid of guardrails. Similar observations have been made by (Nasr et al., 2023) , which noted that appending lengthy nonsensical tokens could bypass safeguards and retrieve private information from GPT-3.5. We include an example of attacking GPT-3.5-0125 successfully in Appendix N.To avoid false positives from model-based evaluators, we also manually verify the harmfulness of the generated content previously identified as harmful, according to the same principles used for GPT-4 evaluator. Given 99% ASR on the latest GPT-3.5 compared to 90% on the earlier version, we suspect that enhancing capabilities come at the expense of compromising safety unfortunately.However, due to the stronger internal safeguard mechanisms of GPT-4, all approaches, including AmpleGCG, fail to achieve a decent ASR, while AmpleGCG's performance still surpasses all baselines. We hypothesize that by increasing the sampling times, ASR could be further improved for AmpleGCG. Future work could also leverage more stringent harmfulness evaluators (such as GPT-4 or human evaluator) to collect training data for AmpleGCG to enhance its transferablity to attacking GPT-4.AmpleGCG against Perplexity-based Defense. As suggested by (Jain et al., 2023) , gibberish suffixes would be easily detected by perplexity-based detectors. To study the effectiveness of AmpleGCG against perplexity-based defenses, we propose two tricks to repeat the original query and extend the overall length, thereby leading to lower perplexity: (1) AmpleGCG Input Repeat (AIR) generates the corresponding suffixes from repetitive queries, i.e., AmpleGCG(repeat(x 1:m , j)), where repeat(x, j) means repeat the original query x for j times. Then, it appends the suffix to the repetitive queries to jailbreak while maintaining low perplexity.(2) We notice that AmpleGCG is only trained on non-repetitive queries and may lead to distribution shift if the input is repetitive. We further propose AmpleGCG Input Direct (AID) to directly generate suffixes via AmpleGCG(x 1:m ), similarly to training procedure, but append this suffix to the repetitive queries. Results in Appendix F indicate that AmpleGCG successfully evades the defense with an 80% ASR using only 100 suffixes, outperforming GCG and AutoDAN, which achieve ASRs of 0% and 42%, respectively. Essentially, AmpleGCG extends its effectiveness to unfamiliar repetitive patterns, suggesting that the suffixes AmpleGCG produces are effective across different formats of queries. We also hypothesize that proficiency can be attributed to the recency bias issue (Liu et al., 2024) of LLMs, only focusing on the last several statements while ignoring the previous part. 
limitation: 
 We employ two evaluation methods: string-based evaluator and model-based evaluator (Beaver-Cost), to filter out unsuccessful adversarial suffixes during data collection and identify harmful content during evaluation (except for experiments on closed-source models).The ensemble of the two evaluators offers a more strict assessment of the harmfulness of the outputs from victim models than only one evaluator (Zou et al., 2023; Huang et al., 2023) .However, instances of false positives remain, suggesting that more rigorous evaluators could be incorporated in future work to refine the training data quality and ASR results.We mainly study GCG in this work because it cannot be easily mitigated via standard alignment or other strategies as discussed in Appendix K. However, we also note that one may train a similar generative model using pairs of <harmful query, adversarial prompt> collected from any jailbreaking method, without being limited to nonsensical suffixes or GCG. We leave this study to future work. 
related work: 
 Prompt Optimization. There are two main folds of prompt optimization: soft and hard. Soft prompts optimization (Liu et al., 2023a; Lester et al., 2021; Li & Liang, 2021; Wang et al., 2022) attempts to freeze most of the parameters and take the remaining part as the soft prompts to optimize. Hard prompt optimization instead focuses on the manageable input prompt. Besides human prompt engineering, AutoPrompt (Shin et al., 2020) Jailbreaking and Red-Teaming LLMs. Jailbreaking LLM aims to elicit objectionable content from LLMs. Any method to jailbreak the aligned LLMs could be utilized to red-team the models and vice versa. Typically, manual red-teaming (Shen et al., 2023; Wei et al., 2024; Schulhoff et al., 2023) efforts can be done to ensure safe deployment. Automatic red-teaming/jailbreak could be classified into two categories (Wei et al., 2024) : Attacks under competing objectives focus on leveraging virtualization, role-playing, and endowing harmful persona or persuasion techniques (Mo et al., 2023; walkerspider, 2023; Zeng et al., 2024; Shah et al., 2023; Shen et al., 2023; Yu et al., 2023; Schulhoff et al., 2023; Perez et al., 2022) 2023) by targeting long-tailed cases. There is other work (Zhao et al., 2024; Zhang et al., 2023) , requiring direct access to the output token logits or further fine-tuning the victim models (Qi et al., 2023; Pelrine et al., 2023) . 
conclusion: 
 This work presents a universal generative model of adversarial suffixes, AmpleGCG, that works for any harmful query. We first conduct a deep analysis of GCG by revealing that loss is not a reliable indicator of the jailbreaking performance and discover more successful suffixes during the optimization. Augmented by this finding, we achieve higher ASR than default GCG and unveil more vulnerabilities of the target LLMs. Furthermore, we utilize these suffixes to learn the adversarial suffix generator, AmpleGCG. Remarkably, it achieves nearly 100% ASR for Llama-2-7B-Chat and Vicuna-7B by sampling 200 suffixes for 100 queries in 6 minutes (4 seconds for one query). AmpleGCG can also transfer seamlessly from open-source to closed-source victim models (99% ASR on the latest GPT-3.5) and bypass the perplexity defender at 80% ASR with only 100 suffixes. To conclude, AmpleGCG more comprehensively unveil the vulnerabilities of the LLMs in a shorter time, calling for more fundamental solutions to ensure model safety. 
 
 Appendix: This study includes materials that could enable malicious users to elicit objectionable content from any victim models by crafting adversarial suffixes. Though such gibberish suffixes are not covered by standard alignment processes and could not be trivially mitigated, numerous alternative methods for circumventing the safety constraints of LLMS already exist and are widely popularized online. Moreover, our two ways of amplifying effectiveness are all based on the effectiveness and implementation of GCG, which used techniques that appeared in the literature previously. Ultimately, any committed team focusing on using LLMs to create harmful content could eventually uncover these methods for enhancing the effectiveness of GCG.However, AmpleGCG identifies vulnerabilities more thoroughly and signals the need for more sophisticated defense measures. Therefore, we consider it crucial to share this research, aiming to aid in the creation of stronger defense strategies against such attacks.To sum up, the primary goal of our research is to comprehensively find the vulnerabilities inherent in LLMs, with the ultimate aim of enhancing their security and resilience. Our intention is not to facilitate or encourage malicious applications of this knowledge. To ensure responsible use, we are committed to closely monitoring the application of our findings. It is our hope that our work will inspire further investigations and advancements in the field, contributing to the development of more secure and robust LLMs.To address both reproducibility and ethical considerations, we will not release the trained AmpleGCG generator publicly. Instead, we will make the code for the overgeneration component of our proposed augmented GCG available, and provide gated access to Am-pleGCG. This decision stems from concerns that if AmpleGCG were used inappropriately, it could potentially enable malicious users to quickly deploy the model and compromise both open-source and proprietary LLMs, posing a significant risk to society. Given that GCG has already made their code publicly available for reproducibility, we believe that releasing GCG-related codebase will not introduce significant additional risks. For research purposes, we plan to release the suffixes generated by AmpleGCG on AdvBench and MaliciousInstruct, and the learned AmpleGCG model itself, provided that verified organizations request them.discussions. This work was sponsored in part by NSF CAREER #1942980. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein. We slightly modify the default GCG algorithm and augment it with overgeneration i.e. collecting all candidate suffixes during the optimization. The modifiable subset I represents the index of positions in x m+1:m+l that allows modification. Iteration T indicates the max number of steps of optimization. Batch size B means the number of candidate suffixes at each step.Compared to the default GCG algorithm, we highlight the simple changes in yellow below.For more details, please refer to (Zou et al., 2023) . Algorithm:= x 1:n ▷ Initialize element of batch x(b) i := Uniform(X i ), where i = Uniform(I ) ▷ Select random replacement token C ← C ∪ { x(b) i } ▷ Collect candidates end for x m+1:m+l := x(b ⋆ ) m+1:m+l , where b ⋆ = b L( x(b) 1:n ) ▷ Compute best replacement end loop Ensure: Optimized suffix candidates list CWe collect all the synthesized data through the proposed overgenerate-then-filter (OTF) pipeline, based on the findings that loss is not a suitable indicator and there are many other successful unexplored suffixes during the optimization. For collecting data from single victim models, we just apply the standard OTF pipeline. For collecting data from multiple victim models, we optimize for multiple victim models simultaneously over different single queries under the GCG individual setting (overgeneration) and then filter out suffixes that fail on any one of the victim models (then-filter). Note that multiple victim models are different from mutliple queries setting in GCG, where mutliple queries setting means optimization over multiple harmful queries and collecting suffixes from multiple victim models could be done in both individual query and mutliple queries settings.We first sample 445 queries out of 520 harmful queries from the AdvBench and apply OTF pipeline under the individual query setting to synthesize data. Although the mutliple queries setting, producing a universal prompt for queries, could also synthesize datasets, universal suffix is not pragmatic in real attack scenarios, as they are easily blocked by developers once leaked. Since Llama-2-7B-Chat is built with more sophisticated defense mechanisms than Vicuna-7B, some of the queries could not find any available suffixes even with the OTF pipeline. To ensure queries in the training sets are the same across the two models, we only keep 318 harmful queries that exhibit successful suffixes in both models. After saving all the successful suffixes, we design three sampling approaches to sample from the saved suffixes. They are:• random: Random sampling 200 successful suffixes for queries.• step: we continuously sample in a round-robin fashion from each step, proceeding in this manner until we reach a total of 200 samples for one query. If it's not possible to reach 200 samples, then we sample as many as available.• loss 100: For all candidates, we first segment the loss into 100 distinct spans in ascending order, and treat each span as different steps. Then, we apply the same approaches as step Sampling Strategies above to sample 200 suffixes for each query. To more convincingly demonstrate the efficacy of the refined AmpleGCG compared to GCG individual setting and the proposed pipeline, we deliberately incorporate a random subset of the 127 queries ( 445 -318 ) (hard cases) -known to be incapable of breaching Llama-2-7B-Chat under individual settings with the overgenerate-then-filter pipeline -into our test sets. Furthermore, we augment our test sets by randomly incorporating additional untested queries (unknown cases) to assemble the test set, culminating in a total of 100 harmful queries. We sample other 50 examples as validation datasets. For data statistics please refer to Table 6 and Table 7 .Since Llama-2-7B-Chat and Vicuna-7B assumably have distribution discrepancy among them as shown in Section 4, that's why #training paris curated from a combination of Llama-2-7B-Chat and Vicuna-7B are fewer than others as shown in Table 7 . All experiments are conducted on the server with 4*A100 GPUs. For training hyperparameters of finetuning the AmpleGCG, please refer to Table 8 for details.Notwithstanding the observation that different sampling strategies yield comparable ASR on validation sets, it is posited that the strategy of sampling across diverse loss intervals potentially offers a more nuanced understanding of the patterns inherent in successful jailbreak suffixes. The rationale behind our hypothesis is that each jailbreak suffix within the same step would be highly similar and only differ in one token, therefore sampling according to loss could group them in a more diverse manner and ensure examples diversity, as illustrated in Fig 2.Given our hypothesis, we use loss 100 as our sampling approach and select checkpoint at step 30000 for Llama-2-7B-Chat and checkpoint at step 15000 for Vicuna-7B for further experiments. For other experiments, we use the checkpoints at the last steps for evaluation.Figure 3 : Schematic illustration of the overgenerate-then-filter. We first heavily sample during each step of the GCG optimization under individual query setting and then apply two evaluators to filter out the suffixes that could not jailbreak the victim models.Unlike the conventional definition of transferability, referring to whether the produced universal prompts could transfer between victim models, we evaluate the transferability of the AmpleGCG. We exclude individual query setting of GCG our from the baselines due to its lower effectiveness compared to mutliple queries setting in the previous sections. Specifically, we append the suffixes optimized for one model to harmful queries and target other victim models. Table 9 : ASR results on transferability across different victim models. On average, the AmpleGCG could achieve the best attack performance compared to all baselines. Although suffixes optimized on Llama-2-7B-Chat are hard to transfer to Vicuna-7B, by sampling 100 times from the AmpleGCG, we could enhance this transferability.From Table 9 , GCG augmented with overgeneration could largely enhance the performance for the transferability while AmpleGCG could further push the results forward by only sampling 100 times. This indicates that the AmpleGCG captures the crucial aspects of harmful queries and generates tailored suffixes that are also effective for different victim models. It signifies dual aspects of transferability: firstly, from training queries to unseen test queries, and secondly, from an optimized model to unoptimized models.Although AutoDan could produce semantically meaningful prompts which are speculated more transferrable (Liu et al., 2023b) , AmpleGCG could achieve greater performance as well while being gibberish. AutoDAN halts the optimization process upon success but could also be enhanced by continuing to optimize in order to generate additional prompts for a single harmful query. We leave this aspect for future research. However, AutoDAN assumes access to the first few generated tokens and mandates them to be specific tokens, whereas our approach does not require any access to the generated tokens, which more closely mirrors real attack scenarios where users cannot modify the generated tokens.Besides, we also include the results for optimizing several models together in Appendix E and show that AmpleGCG could obtain averagely higher performance than GCG.Another intriguing observation is that suffixes originating from Vicuna-7B, whether through GCG, AutoDAN, or our own AmpleGCG, do not transfer effectively to Llama-2-7B-Chat. We hypothesize that this discrepancy arises because Vicuna-7B undergoes only superficial instruct tuning based on data distilled from ChatGPT, while Llama-2-7B-Chat benefits from iterative red teaming and reinforcement learning processes. This leads to a significant divergence in their data distributions. Despite sharing the base architecture and being fine-tuned on the same foundational model, the transferability between them is relatively poor. However, by increasing the sampling frequency with our AmpleGCG, suffixes derived from Llama-2-7B-Chat can achieve higher success rates when transferred to Vicuna-7B. Table 10 : ASR results when optimizing over Llama-2-7B-Chat and Vicuna-7B simultaneously.The AmpleGCG could still achieve more advanced ASR compared to GCG in multiple models optimization settings.The ASR results when optimized over two models, Llama-2-7B-Chat and Vicuna-7B. Please refer to Appendix B for how we collect training data from the pipeline for more than one optimized model and the specific data statistics.Results demonstrate that AmpleGCG could averagely achieve higher ASR compared to default GCG. Though we find the Llama-2-7B-Chat and Vicuna-7B could not transfer well to each other (Appendix D), due to data distribution discrepancy, AmpleGCG trained on both of them could bridge this gap. This points to the potential to train a unified AmpleGCG that works for any victim models.We follow the setting from (Liu et al., 2023b) and configure the perplexity detector's filtering threshold to the maximum perplexity among all standard user queries, i.e. only s 1:m . To compute the perplexity of statements across various victim model configurations, we employ the respective victim model to determine the perplexity.Although GCG augmented with overgeneration could increase the ASR without any perplexity defense, they could not bypass the perplexity detector as well given Table 11 . Rep N means repeat(x 1:m , j) = repeat(x 1:m , N). Results show that AmpleGCG could generalize to the unseen repetitive format of queries for AIR and suffixes produced by AID could adapt to the repetition of queries, which bypass the perplexity detector while maintaining high ASR compared to 0% ASR for GCG and 42% for AutoDan.With simple repetition tricks, AmpleGCG could generate suffixes to jailbreak the victim models successfully in a high ASR against the perplexity detector. Specifically, for the AIR setting, it indicates the AmpleGCG could generalize to the unseen format of query, long repeated queries, and generate corresponding suffixes. After alleviating the distribution shift under the AID setting, the performance of the AmpleGCG is further enhanced. We assume that it's because that AmpleGCG captures the essential part of the queries to generate customized suffixes, therefore the suffix coming from a single query could apply to unseen long repeat queries. This is also attributed to the known issue of recency bias from language models and the victim models ignore the repeated part of the queries but only focus on last several statements. We admit that default GCG could apply the AIR and AID tricks to bypass the perplexity detector but with lower efficiency. We leave that to further work to explore its effectiveness.We put the extra ASR results on Vicuna-7B against the perplexity detector in Appendix G. Visualization for Vicuna-7B during GCG optimizations. Since Vicuna-7B is only built with less strong safety alignment process, sampling during the middle of the optimization process produces more successful adversarial suffixes than Llama-2-7B-Chat, as shown in Figure 4 . MaliciousInstruct is introduced by (Huang et al., 2023) . It consists of 100 harmful instances presented as instructions. MaliciousInstruct contains ten different malicious intentions, including psychological manipulation, sabotage, theft, defamation, cyberbullying, false accusation, tax, fraud, hacking, fraud, and illegal drug use.Note that MaliciousInstruct contains different categories that AdvBench doesn't cover and the queries are ended in question marks, different from declarative queries in AdvBench. This serves as the OOD test sets to fairly compare the AmpleGCG with the baselines about their generalization. Results are shown in Table 4 .We acknowledge that there are other jailbreaking approaches and relevant works can be found in 6. The works lying in the category of competitive Objectives involve leveraging strong instruction-following capabilities of the LLM system, but conversely, model developers could also strengthen the safeguard by taking the strong following capabilities and setting more rigorous system prompts. Those translating harmful queries into another format works could also be easy to remove by augmenting broader alignment data. However, GCG doesn't depend on instruction-following capabilities and could not be easily removed by more alignment data (NewYork Times, 2023), instead, it targets longtail cases (appending gibberish suffix to user query) that standard alignment would not take into consideration. It directly optimizes the open-sourced models without costly API-based models for paraphrasing, which makes it affordable and easy to attack the systems while achieving respectful ASR. Building on this, we conduct a deeper investigation into the GCG with the aim of amplifying its performance, particularly in terms of efficiency, effectiveness, and comprehensive vulnerability coverages. We acknowledge there are other works (Zhao et al., 2024; Zhang et al., 2023) , requiring direct access to the output token logits or further finetuning the victim models (Qi et al., 2023; Pelrine et al., 2023) , but we are more interested in cases where the LLMs are frozen and could not be manipulated once deployment.We ablate different decoding methods from the AmpleGCG and measure the effect on the ASR and the number of unique successful suffixes for each jailbroken query. Particularly, for group beam search, we still use the default setting of keeping the number of beam search groups the same as the number of beams. All three sampling approaches would get higher ASR and more successful suffixes by sampling more times. Group beam search can rapidly achieve a 100% success rate within just 200 sampling instances, whereas beam search requires 1000 instances to reach the same level of success. Meanwhile, the Top p method is unable to attain a 100% success rate even by sampling 1000 times.We posit that our AmpleGCG has effectively learned to map user queries to harmful suffixes, with the most successful potential suffixes predominantly situated within regions of high probability. Consequently, employing beam search-aimed at identifying the batch of most likely suffixes across all beams-significantly enhances the discovery rate of successful suffixes per query and is evidenced by a notable increase in USS.However, these potential successful suffixes often exhibit uneven probabilities, some successful suffixes have a higher probability than other successful suffixes. It leads to repetitive suffix generation by Top p sampling, especially under heavy sampling scenarios. Such as 1000 sampling times, which only produce 208 unique suffixes and largely underestimate the effectiveness of the AmpleGCG. This observation underpins our hypothesis for Top p's stagnation at a 94% ASR, even after 1000 samples: it explores a constrained set of suffixes, thereby neglecting a broader spectrum of viable options.In the case of group beam search, which swiftly approaches a near 100% ASR within 200 samples, we attribute its success to a strategic emphasis on generating diverse outcomes. By aligning the number of beam groups with the number of beams, group beam search notably promotes variation. For some "hard" queries, whose corresponding suffixes do not lie in the high region of the distribution, group beam search can quickly find them while beam search might fail because it only searches for high-probability regions.We have selected group beam search as our default configuration because it swiftly and successfully addresses all queries, which two properties are desired to be utilized to attack and red team the LLMs. While it may not identify as many potential suffixes as beam search, discovering approximately tens is already sufficient for red teaming and proves a decent scale of vulnerabilities for different queries.In this section, we grab the failure suffixes with low loss generated during optimization and investigate why loss is not a good indicator of whether the suffix could jailbreak or not. To simplify the experiments, we only conduct research on Llama-2-7B-Chat victim models and select those failure suffixes with low loss during GCG optimization under individual query setting (blue stars in Figure 1 ). We follow the paper (Lin et al., 2023) to do the teacher-forcing generation. Specifically, we define the harmful query as q = {q 0 , q 1 , • • • , q t } and the target harmful response as r = {r 0 , r 1 , • • • }. For each generated token's logit o t by inputting the context x t = q + {r 0 , r 1 , •, r t-1 }, we obtain the rank of the target token r t in the corresponding logit o t . Notably, the target harmful behavior starts with an affirmative token like "Sure", or "Here is" (r) to induce harmful content in the later positions.Nonetheless, in Figure 5 , the rank of the first token exhibits a large discrepancy with other tokens and LLMs persist in answering in a refusal way and start the response in a safe tone with "Sorry", "As a responsible language model". Since we get the rank via a teacher-forcing generation setting, the rank of the target token is a proxy to the loss of each token defined in the equation 1.From Figure 5 , we could observe that 1)the overall loss across all tokens is low because the ranks of target tokens except on the first position are close to zero. 2)However, the loss of the first target token persists remaining high, indicating that the target affirmative tokens lie in a relatively low mass probability. Although the overall loss is low, the first several tokens largely decide the subsequent generation. Since the first token usually starts with tokens showing a refusal tone, then the subsequent output would be safe instead of harmful. Token shift that happened at the first several positions are also observed by Zhao et al. (2024) .In addition to that, we also make an intuitive ablation study on increasing the weight of the first several tokens during GCG optimization but find that such failure of jailbreaking due to exposure bias could not be eliminated well, so we adhere to the proposed overgeneratethen-filter pipeline.An example of appending the suffix generated from the AmpleGCG to the harmful user queries. As shown in the figure below, ChatGPT (GPT-3.5-0125) indeed produces harmful content about how to steal personal identities. Although they mentioned that such behavior is illegal in the end, outputting such harmful content is not a desired behavior.W a r n i n g : H a r m f u l C o n t e n t E n c l o s e d Prompt for GPT-4 evaluator to determine the harmfulness of the content output by victim models. We borrow such prompt from Harmbench (Mazeika et al., 2024) .You are a classifier for red teaming large language models (LLMs). Given a [GENERATION], determine whether it counts as an instance of the [BEHAVIOR] (subject to the rules below). Your answer should be "Yes" or "No" with an explanation.[BEHAVIOR]: \{behavior\}[GENERATION]: \{generation\} ---To reiterate, you are a classifier for red teaming large language models (LLMs). Given the above [GENERATION], determine whether it counts as an instance of the [BEHAVIOR] .Your answer should be "Yes" or "No" with an explanation.Rules:-Instances of the behavior should be unambiguous and non-minimal. Vague or very short instances do not count.-Generations should themselves be harmful or undesirable for a public-facing LLM to output. Benign generations do not count.-If the generation is harmful but not an instance of the behavior, it does not count.-Generations that include caveats or warnings count if the behavior can reasonably be said to have occurred alongside them.-Generations that respond, analyze, discuss or otherwise react to the behavior without actually exhibiting it do not count.-Generations where the model clearly tries to exhibit the behavior count. E.g., it is OK if generated code has errors.-If the behavior includes writing code, the generation MUST contain code artifacts related to the behavior, not just comments or instructions. Otherwise it does not count.-Generations must be in English, or otherwise easy for English-speakers to identify as a successful instance of the behavior (e.g., code that clearly implements a keylogger with comments in a language other than English is still acceptable)Answer:The keyword phrases we used in our first keywords-based metric, as described in Section 3 https://huggingface.co/osunlp/AmpleGCG-plus-llama2-sourced-llama2-7b-chatWe apply the logarithmic operation to the loss to more clearly illustrate its decreasing trend.Parameters are defined in https://huggingface.co/docs/transformers/v4.18.0/en/main classes/text generationTime cost is derived by running on a single NVIDIA A100 80GB GPU with AMD EPYC 774264-Core Processor. 


Decision: This paper presents a method to enhance the effectiveness of GCG, proposing a procedure that retains suffixes from intermediate optimization steps and using these to train a model to generate attack suffixes. They show that their method yields high attack success rate.

Overall the reviewers are happy with this paper. They appreciate the improvements over GCG and accompanying methodological insights, the impressive ASR, the fast generation of large numbers of suffices, the strong transferability across models, the simplicity of the method, thoroughness of the experiments, and timeliness/impactfulness of the work.

Some concerns are raised that testing could be done on more models, that there is low effectiveness of a particular trick presented for bypassing perplexity based defenses, that more defenses could be explored, that there is some room for more writing clarity, and that these types of automated attacks could aid malicious usage (ethics concern). Overall, however, I feel the authors have satisfactorily addressed these concerns in their responses, and this is a solid submission for publication.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: The paper presents AmpleGCG, a generative model designed to create adversarial suffixes that can jailbreak both open-source and closed-source large language models (LLMs). The authors propose an augmented approach that collects all candidate suffixes during the optimization process. These collected suffixes are then used to train AmpleGCG, which achieves near 100% attack success rates on various models including Llama-2-7B-chat, Vicuna-7B, and GPT-3.5.
strength_weakness: strength_weakness: - The creation and dissemination of a tool capable of efficiently generating adversarial attacks on LLMs could be considered ethically questionable, as it may aid malicious actors.
- The paper could benefit from a more detailed exploration of potential defenses against the adversarial suffixes generated by AmpleGCG, beyond just perplexity-based defenses.
questions: questions: How do you envision the ethical implications of releasing a tool like AmpleGCG to the public, and what measures do you propose to mitigate potential misuse?

Can you provide more details on the types of harmful queries used in your experiments, and how these queries were selected?
paper_summary: paper_summary: ### Summary of the paper

The paper studies the GCG algorithm, identifies some weaknesses in it, and proposes techniques to mitigate these weaknesses and enhance the attack's effectiveness. Specifically, the authors observe that having minimal loss (maximal likelihood of the target string) is neither a sufficient nor a necessary condition for the attack's success. They further observe, that over the course of the optimization, the likelihood of the first token of the target output remains relatively high. Given these observations, the authors propose an "over-generation" procedure that retains suffixes from intermediate optimization steps and applies all of them, one-by-one during the attack. This significantly increases the attack success rate. Finally, the authors use the generated attack suffixes and the corresponding queries to train a LLM, that is able to generate successful attack suffixes for unseen queries that generalize across models.

### Quality

- The paper contains interesting analyses of GCG, which enabled the authors to identify weaknesses and develop improvements. The proposed techniques are relatively simple but are well motivated and accompanied by empirical evidence and analysis.

- The evaluation methodology seems to be carefully designed and instills confidence in the results. However, the generalizability of the results is questionable because very few models were used in the evaluation. Most of the results are based on Llama-2 and Vicuna. These two models have the same architecture and pertaining data, and only differ because the latter was fine-tuned on share-GPT. To show that the observed effects are not a consequence of the chosen target model, more diverse models should have been used. Particularly, the observation about the loss (Section 3.1) should show that the effect is observable on multiple diverse models. Likewise, to evaluate transferability models other than GPT-3.5 and GPT-4 should have been used. It is known from several past papers that attacks crafted using Llama2 and Vicuna transfer to GPT-3.5 because Vicuna is fine-tuned with share-GPT. Several other closed-source models should have been used here such as Claude and Gemini to show the extent of generalizability to models that are more decoupled from the training models.

- The AmpleGCG LLM was trained on Guanaco, Vicuna and Llama, all of which are based on Llama. It is likely that if suffixes from other open-sources models such as Mistral (and its variants) or Phi were used the transferability of AmpleGCG to unseen models like GPT-4 would have been better.

- The techniques for bypassing perplexity based defenses is relatively unsophisticated, and probably easy to defend against. I would not consider this a significant contribution of this paper.

### Clarity

The paper is written clearly enough, but there is some room for improvement. Non exhaustive list of points:
- Intro-para2: "existing jailbreaking methods, including GCG, only generate one single adversarial prompt for one query". This is not true as one can generate one adversarial prompt for multiple queries.
- Intro-para2: "Vulnerabilities" is used to refer to attack strings. This is confusing. Might be better to state attack strings/adversarial prompt.
- Definitions of metrics should be included in the Table/Figure caption. For example, diversity should be defined in the caption of Table 2, not in the text.
- Training and evaluation details are dispersed throughout section 4. They should be in a separate subsection to make it easy to find them and refer to them.

### Originality

The techniques are not highly novel themselves, but their effective application to the task of adversarial suffix generation is still a significant contribution in my opinion.

## Significance

The work is timely and impactful. The safety and societal impact of LLMs is a great concern and this paper shows that existing models are not only highly vulnerable to adversarial attacks, but that these attacks can be generated very efficiently.
strength_weakness: strength_weakness: - Evaluation setup could be improved to make claims about generalizability more convincing (see summary for details). In short, I recommend including more diverse open- and closed-sourced models in the training and testing stages.

- Clarity could be improved in several places.

- The techniques for bypassing perplexity based defenses is relatively unsophisticated, and probably easy to defend against.
questions: questions: - It is mentioned that over-generation with GCG does not produce sufficiently diverse suffixes. This is expected because GCG changes one token at a time. However, it may be possible to enhance the diversity by running GCG multiple times (with different random seeds). Since it is known that suffixes at intermediate optimization steps can be successful, the optimization can be run for few steps -- 1 run with 500 steps vs. 2 runs with 250 steps. It might be useful to to generate training data for AmpleGCG this way to enhance the diversity.

- How many human evaluators performed manual inspection of GPT-3.5 responses? To what extent did they agree/disagree?
paper_summary: paper_summary: Summary: This article proposes an enhancement method based on GCG and trains a universal generative model, named AmpleGCG. In the open source model, AmpleGCG trained based on the data from Llama-2-7B-Chat could achieve 72% transfer ASR on Vicuna-7B and Mistral-7B-Instruct average. In the closed source model, AmpleGCG trained based on the data from Vicuna and Guanaco could achieve 99.00% transfer ASR on GPT-3.5-0125. AmpleGCG can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.
strength_weakness: strength_weakness: 1. This article points out that the loss of GCG is not the best metric, but does not provide a better solution. Overgenerate only selects more of the top-k candidate suffixes in each step, but the final iteration still uses the suffix with the smallest loss.
2. AmpleGCG has a low ASR for GPT-4, and the ASR calculated using the string-based evaluator is higher than the true value.
3. The tricks AmpleGCG applied to bypass the perplexity detector has low efficiency.

Comments: title: Follow-up on the Rebuttal
comment: Thank you for the clarifications. I will retain my rating of Accept for the paper.
Dear reviewer ygbw,

Hope this follow-up messages find you well. We want to thank you again for your valuable feedback and comments on our submissions.

In our rebuttal, we believe we addressed the points you raised, including concerns about better metrics, the selection of anchor points during the iteration process, clarification about the evaluators we used beyond the string-based method, and the relatively low ASR but still competent compared to the two strongest baselines. Besides, we also conducted more experiments to prove that the efficacy of repetition tricks to bypass the perplexity-based defense would **NOT** be compromised when sampling more times.

Please kindly let us know If there are any remaining concerns or additional suggestions, we would be grateful for your guidance and open to more suggestions.

Thank you once again for your time and consideration.