P4: Plug-and-Play Discrete Prompting for Large Language Models Personalization (2024)

Abstract: Empowering Large Language Models (LLMs) with distinct human-like personality traits has become an innovative task for developing advanced dialog systems. Although LLMs demonstrate impressive capabilities in following instructions, directly prompting them to exhibit certain personalities through manually crafted instructions may result in sub-optimal performance. In this paper, we propose a plug-and-play prompting method to manipulate the LLMs’ personality traits. Specifically, we append discrete personalized suffixes, automatically generated through an aggregated gradient-based search method, to the user query or dialog histories and induce LLMs to respond with target personalities. In addition, due to the high redundancy of the search space, we adopt a reward-based strategy to prune the vocabulary and focus exclusively on influential tokens. Experiment results on four models ranging from 1.1B to 13B show that our method achieves 79.9% accuracy in customizing LLMs’ personalities, significantly outperforming other prompting methods (65.5%) and model editing methods. Our method also excels in generation fluency and quality with the lowest generation perplexity and the highest GPT-4 evaluation scores.

heading: 1 Introduction
text: The landscape of natural language processing (NLP) has been evolved by Large Language Models (LLMs) (OpenAI, 2023; Zhao et al., 2023). With huge amounts of unsupervised pre-training followed by supervised instruction tuning, LLMs exhibit remarkable abilities in various tasks, including interactive dialogue (Ouyang et al., 2022; Chen et al., 2023b; Chae et al., 2023). Recent works have explored the versatility of LLMs as conversational agents with predefined characteristics, highlighting
*Equal contribution. †Corresponding Author.
their potential for personalization (Shao et al., 2023; Park et al., 2023; Shanahan et al., 2023; Chen et al., 2023a). Personalization plays a vital role in human-computer interaction, as tailoring responses can foster human-like interactions and improve the user experience (Zhang et al., 2018; Wang et al., 2023b; Huang et al., 2023).
While black-box LLMs, such as GPT-4 (OpenAI, 2023) and Claude (Models, 2023), excel in role-playing and following instructions, their requirements for users to upload personal data raises privacy concerns. Open-source models demonstrate privacy and deployment friendliness, but fulfilling application requirements demands large-scale models, with parameters as large as 70B (Shen et al., 2023). This poses computational challenges in scenarios like on-device deployment (Xu et al., 2023). One possible solution is to utilize smaller, open-sourced models. However, prompt-
9129
ing these models with manual-crafted prompts still leads to sub-optimal behaviors (Shen et al., 2023). Alternatively, techniques like instruction tuning and model editing (Mao et al., 2023) train proprietary models and directly modify model parameters, potentially degrading performance on other tasks and limiting the model’s scalability (Wang et al., 2023a).
In this paper, we introduce P4 (Plug-and-Play Discrete Prompting for Large Language Models Personalization) to address the aforementioned challenges. Inspired by AutoPrompt (Shin et al., 2020), we propose a prompting method by appending a personalized suffix to the user queries or dialog histories to manipulate the personality traits of LLMs (Figure 1). These suffixes are optimized through an aggregated gradient-based search method with the target to induce the model to generate responses exhibiting the target personality traits. In addition, previous research (Zhou et al., 2023) indicates that the search space is highly redundant, with a few tokens having a disproportionate influence on the model performance. Therefore, to reduce the optimization complexity, we employ a reward-based strategy to prune the original search space and optimize the suffixes based on the most influential tokens. Overall, since the personalized suffix is a plug-andplay module, it offers the flexibility to be activated or deactivated based on actual applications, which provides users with a convenient way to control the model behaviors without explicitly changing model parameters.
Our contributions are summarised as follows:
• We introduce a novel plug-and-play prompting method to manipulate the personality traits of LLMs utilizing personalized discrete token suffixes. The suffixes are optimized through an aggregated gradient-based search method.
• In addition, to accelerate the optimization process, we employ a reward-based strategy to prune the search space and focus exclusively on the most influential tokens.
• Empirical results on four models ranging from 1.1B to 13B demonstrating the effectiveness of our method. Specifically, our method achieves 79.9% accuracy in editing the LLMs’ personality traits while ensuring high quality of generations.

heading: 2 Related Work

heading: 2.1 Personalization in NLP
text: Personalization has been well explored in the NLP communities, with tasks such as recommender system (Das et al., 2007; Xu et al., 2022), search applications (Dumais, 2016; A. Tabrizi et al., 2018; Zeng et al., 2023) and conversational agents (Liu et al., 2022; Zhang et al., 2018; Lotfi et al., 2024). In this paper, we mainly focus on the research of the personality traits of language models. Previous works have explored personality classification and personality recognition tasks (Yang et al., 2021; Flekova and Gurevych, 2015; Wen et al., 2023b). With the capabilities of LLMs growing, recent studies (Miotto et al., 2022; Tu et al., 2023; Serapio et al., 2023; Jiang et al., 2023; Mao et al., 2023) examine the personality traits of these models and attempt to edit their personality traits through direct prompting or model editing. In this paper, we utilize plug-and-play discrete token suffixes to manipulate the personality traits of LLMs.

heading: 2.2 Discrete Prompt Optimization for Language Models
text: There has been plenty of work dealing with discrete prompt optimization. One approach is to optimize discrete tokens via the continuous embedding space. (Qin et al., 2022) introduced a decoding framework using Langevin Dynamics to sample discrete tokens from continuous embeddings. (Wen et al., 2023a) learns hard prompts via continuous optimization based on gradient reprojection schemes. Another line of work directly optimizes the discrete tokens. Hopflip (Ebrahimi et al., 2018) uses the one-hot vector gradient to estimate which individual token change has the highest estimated loss. GBDA (Guo et al., 2021) optimizes a parameterized distribution of adversarial examples with gradient-based methods. AutoPrompt (Shin et al., 2020) maximizes the log-likelihood of labels with tokens swapping, measured by firstorder approximation. ARCA (Jones et al., 2023) iteratively maximizes an objective by updating a token in the prompt or output, while keeping the remaining tokens fixed. GCG (Zou et al., 2023) adopts a similar greedy coordinate gradient-based search method to jailbreak LLMs. In this paper, our optimization method follows the above gradientbased method but is optimized with aggregated gradients to search for a universal suffix.

heading: 3 Methodology
text: In this section, we first describe the task definition, and then we introduce the aggregated gradientbased search method. Finally, we present the search space prune strategy and the response initializing process. Figure 2 provides an overview of our methodology.

heading: 3.1 Task Definition
text: We focus on sequence generation tasks. Assume a pre-trained model M is represented as a function: f : X ⇒ Y that generates outputs y corresponding to the input sentence x = (x1, x2, . . . , xn), where xi, 1 ≤ i ≤ n denotes each token in the sentence. Additionally, we define a series of personality traits P = {p1, p2, . . . }. Our goal is to manipulate M to produce an output yp exhibiting targeted personality traits pi ∈ P , without explicitly altering the model’s parameters. Inspired by AutoPrompt (Shin et al., 2020) and GCG (Zou et al., 2023), we add several "trigger" words across all the prompts. Specifically, we concatenate x with a personalized suffix s, which is composed of discrete tokens s = (s1, s2, . . . , sm), and si, 1 ≤ i ≤ m denotes each token in the suffix. Subsequently, the new prompt xp = (x1, x2, . . . , xn, s1, s2, . . . , sm) is fed into the model M and generates the output yp with targeted personalities.
Following the above definitions, we formulate the probability of predicting the next token as p(xi+1|x1:i). Therefore, for a given prompt xp = [x; s] and target label yp = (y1, y2, . . . , yl), the probability of generating yp is
p(yp|[x; s]) = l−1∏
i=1
p (yi+1 | [x; s; y1:i]) (1)
where ; means concatenation. Under this formulation, the loss of the problem is the negative loglikelihood of Equation 1, i.e
L (x, s) = − log p (yp|[x; s]) (2)
in which x is fixed and the suffix s is to be optimized. Therefore, the optimization goal is to minimize Equation 2 with respect to s, i.e
minimize s
L (x, s) (3)

heading: 3.2 Aggregated Gradient-based Search
text: So far, we have shown how to reformulate a personality manipulation task into a discrete tokens
optimization task. An intuitive idea to get the suffix s is to exhaustively enumerate all possible compositions of tokens in vocabulary V and select one with the minimum loss. However, it is unrealistic to evaluate all possible suffixes. Therefore, we adopt a step-by-step gradient-based substitution method to gradually decrease the loss. Specifically, we substitute one position at a time based on a criterion that the substitution leads to the maximum decrease in loss. Note that LLMs typically generate the embedding for each token si based on their one-hot vector esi ∈ R|V | 1, therefore we can take the gradients with respect to this one-hot vector to replace the gradients for the token. Specifically, we calculate the linearized approximation for changing si in s by computing the gradient with respect to esi , i.e,
gi = ∇esiL (x, s) ∈ R |V | (4)
For each token si, we acquire the candidate substitutions Candi by selecting the top-k index in gi. Candi = Top-k [−gi] (5) Then we acquire a candidate set
Cand = Cand1 ∪ · · · ∪ Candm ∈ Rk|m| (6)
We ramdomly sample D replacements from Cand, evaluate their loss through one forward pass, and choose the one with the smallest loss to update the suffix s.
To better optimize the suffix towards certain personalities, we aggregate gradients from multiple samples to obtain the substitutions and search for a universal suffix, i.e:
gi := Σ B j=1L
( xj , s ) = ΣBj=1∇esiL(y
j |[xj ; s]) (7)
where B denotes the batch size.

heading: 3.3 Search Space Prune
text: Due to the huge search space, it can be timeconsuming to search for appropriate suffixes. However, research has shown that the vocabulary contains a large number of "non-influential" tokens, which have minor or even negative impacts on task performance. These redundant tokens significantly increase the search space and complicate the optimization process (Zhou et al., 2023). Therefore,
1Assume the id of si in the vocabulary is 100, then esi is the vector with one in the 100th position and zero in other positions.
we adopt a reward-based strategy to prune the original vocabulary V to reduce the search space. To measure the incremental impact of an individual token v in V , we define:
∆R(v) :=
∑N i=1R (xi, v)−R (xi)
N (8)
where we treat v as a discrete token concatenated to the input xi and ∆R(v) refers to the change in reward with and without the concatenation of v. N denotes the number of examples selected from the dataset. Following the definitions in Section 3.1, we define the reward as the negative loss described in Equation 2:
R(x, s) = −L (x, s) (9)
We focus on the most influential tokens and retain the tokens with the largest changes in reward. In other words, we retain tokens that can reduce the loss of the target task as much as possible for subsequential suffix optimization. We summarize our methods in Algorithm 1.

heading: 3.4 Start Reply with Predefined Personality
text: In practice, we find that it can be difficult to directly optimize s over the response with target
personalities. However, as shown in (Lin et al., 2023; Zou et al., 2023; Zhang et al., 2023), the earlier output token positions play an important role in determining the entire response trends and sentiment. Therefore, we incorporate various personality self-definition phrases at the start of the output, such as "Being someone with an extraverted personality" or "As a neurotic person"). Through this approach, we optimize a universal suffix s that can induce the model to first output the predefined personality phrases and then subsequent responses.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Dataset We use the PersonalityEdit (Mao et al., 2023) dataset for our experiments. This dataset is constructed by prompting GPT-4 to craft responses with respect to specific topics in different personalities. Specifically, the dataset mainly focuses on three of the Big Five (McCrae and John, 1992) personality traits: neuroticism, extraversion, and agreeableness. More details about the dataset can be found in Appendix A.1.
Models To evaluate the effectiveness of our method from diverse models and sizes, we conduct
experiments on four models: TinyLlama-1.1BChat-v1.0 (Zhang et al., 2024), Llama2-7B-Chat and Llama2-13B-Chat (Touvron et al., 2023), Vicuna-7B-V1.5 (Chiang et al., 2023). These models are all open-sourced LLMs that exhibit powerful abilities in interacting with humans and following instructions.
Experimental Details We train a separate suffix for each personality. Following (Zou et al., 2023), for training, we use a sample size of 512 and a top-k of 256. We initialize the suffix with 20 "!" tokens. In addition, we set the batch size B = 25 and epochs E = 10. We prune the vocabulary to 4096 tokens. We use the cross-entropy as the loss function and Adam optimizer. The total number of steps for training is 1000. For inference, we directly append the suffix optimized through the above training procedures to the prompts. Then we prompt the models to perform inference. The detailed ablation studies for the parameter choices can be found in Section 5.4.

heading: Baselines
text: 1) Direct Prompting (DP) We append the personality instruction, for example, "Respond
with an extraversion personality" to the questions to assess the LLMs’ ability to follow these instructions.
2) In-context Learning (ICL) Since LLMs have exhibited powerful in-context learning abilities, we additionally provide LLMs with a few [Problem, Answer] samples to help LLMs better simulate targeted personality.
3) Starting with certain phrase (SW) We instruct the model to output certain phrases at the beginning of the generation, such as "Starting your response with: As an extraversion person".
4) Lora Fine-tuning (FT) We fine-tune the model with training set using Lora (Hu et al., 2021), which can be considered as the upper bound.
5) Mend (Mitchell et al., 2022) is an effective model edit method that uses a single desired input-output pair to make fast, local edits to a pre-trained model’s behavior.
Metrics Following (Mao et al., 2023), we use a personality classifier to evaluate the Editing Accuracy. Specifically, we train a Roberta-base
classifier using the training set of the PersonalityEdit dataset and achieve 98% accuracy on the validation set. In addition, to evaluate the fluency of generated texts, we calculate the Perplexity under GPT-2. We also examine the diversity of generations using Dist-2 (Li et al., 2016) by sampling five times from LLMs. Considering that LLMs serve as reliable evaluators (Chiang and yi Lee, 2023; Liu et al., 2023), we prompt GPT4 to evaluate the quality and consistency of the responses, termed as Model Evaluation. Detailed prompts can be found in Appendix C.2.

heading: 4.2 Results
text: Personality Suffix Achieves the Best Prompting Results According to Table 1, P4 shows superior editing accuracy compared to other prompting methods across all models and personalities (except for the Vicuna-7B under the extraversion personality), highlighting the effectiveness of our method. Specifically, P4 achieved an average editing accuracy of 0.779, surpassing all prompting methods, including DP (0.351), ICL (0.655), and SW (0.541). The findings further indicate that manually crafted instructions for prompting LLMs with fewer than 13 billion parameters yield suboptimal outcomes, indicating significant room for optimization. In addition, for GPT-4 evaluation, P4 also achieves the best results with the highest scores, further demonstrating the effectiveness of P4 in manipulating model personalities and generating high-quality responses.
Fluency and Diversity Even though suffixes derived from discrete optimization may lack semantic meaning, prompting models to initiate responses with self-identification phrases (as discussed in Section 3.4) enables them to produce fluent continuations. As shown in Table 1, P4 consistently surpasses other prompting approaches in terms of lower perplexity, demonstrating enhanced fluency with personality suffixes. On the other hand, the generation diversity slightly decreases with the P4 method on larger models (for example, Llama2 and Vicuna), while the Dist-2 metric consistently remains above 0.9. We attribute the drop to the rigid start of model responses.
Manually-Crafted Instructions Lead to Suboptimal Results Building on the previous discussion, manually crafted instructions yield suboptimal results when compared to P4. Specifically, directly prompting LLMs to behave in specific
personalities (DP) results in the lowest performance, achieving an average editing accuracy of only 0.351. Additionally, performance does not improve as the model size increases. Incontext learning (ICL) significantly improved the accuracy with few demonstrations while suffering from high perplexity. Instructing LLMs to initiate their responses with specific phrases (SW) yields performance better than DP while achieving lower perplexity. However, since the model does not always accurately follow instructions, the SW method falls behind P4 by a large margin.
Tuning methods leads to different behaviors To avoid compromising the model’s performance on other tasks, fine-tuning a small number of parameters is an alternative approach. However, the Mend model editing method results in unsatisfactory performance, achieving an average editing accuracy of 0.325. On the contrary, the fine-tuning method yields the best performance, achieving an accuracy exceeding 0.95. However, deploying each model with one specific personality can be resource-intensive and infeasible for ondevice deployment scenarios. In addition, the tuning methods all demonstrate high perplexity, which can further hinder their applications.
5 Discussion2

heading: 5.1 Token Distribution Shift
text: To understand the mechanism underlying our method, we propose to analyze via the perspective of token distribution shift. Following the definitions in 3.1, for a query x and the context y1:i, we define Pi+1 as the probability of generating the next token. By appending the suffix s to x and providing the same context y1:i, we aim to observe the token distribution shift at each position, termed as P si+1. Analyzing the shift between two distributions across the entire vocabulary can be difficult. Following (Lin et al., 2023), we first prompt the model with s to generate the next token ysi+1 with greedy decoding. Second, by prompting the model with the same query and context without s, the tokens for the next position are ranked by their generation probability in Pi+1. The rank of ysi+1 in the sorted list is noted as η. If η > 3, we consider the token distribution shift happens due to the suffix s. We visualize the
2Unless otherwise specified, the experiments in this section are conducted on LLama-7B-Chat.
shifted tokens in Figure 3. We only display tokens relevant to personality traits (for example, token like "it", "and", etc., are discarded in the figure). The results show that tokens consistent with the target personalities are frequently shifted. This suggests that appending the suffix s causes token distributions to shift towards a specific distribution space aligned with the target personality, thereby increasing the likelihood of generating personalityrelevant tokens.

heading: 5.2 Transferability of Suffixes
text: To investigate the transferability of the personalized suffix across various model sizes, models, and even black-box LLMs, we conduct experiments under three regimes: LLAMA2-7B-CHAT → LLAMA213B-CHAT, LLAMA2-13B-CHAT → VICUNA7B-V1.5 and LLAMA2-7B-CHAT → GPT-4 . We display the results in Table 2. Within the same model scope of the Llama series, our method exhibits exceptional transferability with 0.63 editing accuracy, significantly outperforming Direct Prompting (0.41) and achieves performance comparable to in-context learning (0.643). Under the different model scopes (LLAMA2-7B-CHAT → VICUNA-7B-V1.5), our method also shows transferability, achieving an average performance of 0.657. Surprisingly, for the extraversion personality, suffixes from Llama-7B (0.72) even outperformed the original suffix (P4 with 0.65) of Vicuna-7B. However, the overall performance of the transferred suffixes still lags behind P4 suffixes. Furthermore, due to the superior capabilities of GPT-4, direct prompting already achieves
significant performance (0.873), surpassing the suffix from Llama (0.776). Nonetheless, the transfer suffix still demonstrates better performance on certain personality (such as "Ag").

heading: Persona DP ICL P4 Trans

heading: 5.3 Applications on Multi-turn Dialogs
text: Following we explore applying our method to the multi-turn empathetic dialogue generation task. Empathetic dialogue generation (Rashkin et al., 2019; Lin et al., 2019) aims to understand emotions
according to the dialog contexts and generate responses with appropriate empathy. We utilize the EmpatheticDialogues dataset (Rashkin et al., 2019) and select four representative emotions: joyful, surprised, disgusted, and sad. Following (Wang et al., 2022), we employ attention blocks followed by a Softmax layer to predict the response emotion intents. Additionally, we train separate suffixes for the aforementioned four emotions, enabling the model to generate enhanced responses with specified emotions. We utilize GPT-4 to compare the model’s original responses (w/o suffixes) and enhanced responses (with suffixes), detailed prompts are in Appendix C.3. As illustrated in Figure 5, the employment of the emotion intent prediction module and the addition of emotion suffixes enable the model to respond
with more sensible emotions. The enhanced responses significantly outperformed the original ones across all emotion categories, achieving an average enhancement of over 26.7%. The results demonstrate the potential of our methods to develop more human-like conversational agents.

heading: 5.4 Ablation Studies
text: In this section, we investigate the impact of different parameter settings (Section 4.1) in the experiments, including the pruning vocab size (Figure 4), suffix token length, sample size, and the training batch size (Figure 6).
Pruning Vocab Size To explore the effectiveness of our pruning strategy, we report the accuracy and optimization training time with different pruning vocab sizes, including 2k, 4k, 8k, 16k, and all (32k). As illustrated in Figure 4, reducing the vocabulary size to 4k offers an optimal balance between accuracy and training time. When the search space is enlarged, performance does not significantly improve and is hampered by high training overhead.
Token Length To investigate the impact of suffix token length, we conducted experiments with different lengths, including 5, 10, 15, 20, and 25. As shown in Figure 6a, a length of 20 demonstrated the highest overall accuracy. Smaller lengths yield sub-optimal results due to insufficient length for effectively manipulating token shifts, whereas larger values add the complexity of searching for a reasonable suffix. Therefore, a trade-off exists in selecting the optimal token length.
Sample Size In addition, we conduct experiments with varying sample sizes, including 128, 256, 512, and 1024. As indicated in Figure 6b, a sample size of 512 is adequate for obtaining a qualified suffix and further increasing the sample size may result in performance degradation.
Training Batch Size We further investigate the impact of the training data batch size, specifically, the volume of data required to aggregate gradients for an optimization step. As shown in Figure 6c, using too little or too much data for aggregating gradients results in unsatisfactory outcomes, with a batch size of 25 to 35 being a practical range.

heading: 6 Conclusion
text: In this paper, we propose P4 to utilize discrete token suffixes to manipulate the personality traits of model responses. We conduct experiments on four models to validate the effectiveness of our method, achieving 79.9% editing accuracy and the lowest generation perplexity, significantly outperforming other prompting methods. Additionally, we conduct further analysis experiments and explore applying our method to empathetic dialogue generation tasks to improve model response quality. Our work presents a new plug-and-play prompting technique to precisely manipulate LLMs to display specific personality traits without changing the model parameters.

heading: 7 Limitations
text: The suffixes optimized through our methods are semantically meaningless, which can be hard for humans to understand. Therefore, how to search for appropriate suffixes that are more human-readable can be a future work. In addition, we conduct experiments ranging from 1.1B to 13B. However, behaviors in models smaller or larger than this range can be different. Finally, the applications
of our method on more datasets is under-explored and we leave it to future.

heading: 8 Ethical Considerations
text: The data (Mao et al., 2023; Rashkin et al., 2019) used in our paper are all obtained from opensourced datasets. In addition, the methods used in our work may cause misuse of LLMs. For example, users can utilize suffixes to induce the model to output aggressive responses. When applying our method in real-world applications, careful considerations should be taken to prevent the harmful impact of the model.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069,62206057), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: Appendices

heading: A Datasets

heading: A.1 PersonalityEdit
text: The dataset is constructed by prompting GPT-4 with questions such as "Answer the question in acting as an individual with depression personality facet. What is your opinion of Coldplay?" ("depression corresponds to the neuroticism personality"). By prompting with explicit personality requirements, GPT-4 responds with the question. For each of the three personalities (neuroticism, extraversion, and agreeableness), the dataset contains 1600 training data, 200 validation data, and 200 test data. Examples of the dataset can be found in Table 3.

heading: A.2 EmpatheticDialogues
text: The EmpatheticDialogues (ED) dataset encompasses 25,000 multi-turn empathetic conversations, including interactions between speakers and listeners. ED contains 32 even labels that are common emotions in daily chats. (Welivita and Pu, 2020) enriches the dataset with 41 new categories of emotional and intentional labels at the utterance level, offering detailed insights into the nature of empathy within the dialogues.

heading: B Method Details
text: We summarize the algorithm in Section 3 as Algorithm 1

heading: C More Experimental Details

heading: C.1 Baseline Prompting
text: The prompt of the three baselines: Direct Prompting (DP), In-context Learning (ICL) and Start With (Sw) are shown in Table 4.

heading: C.2 GPT-4 Evaluation
text: Following (Mao et al., 2023), we utilize the 1 to 5 scores by GPT-4 to judge the relevance of the generated responses to target personality traits. The prompts for our GPT-4 Model Evaluation are shown in Table 5.

heading: C.3 Multi-turn Dialogs Model Evaluation
text: In section 5.3, we utilize GPT-4 to compare the original response with our enhanced responses, the prompts are shown in Table 6.

heading: D More Experiment Results

heading: D.1 Word Cloud
text: We display the word cloud of shifted tokens with agreeableness personality in Figure 7. We note that the shifted tokens closely relate to the target personality trait.

heading: D.2 Suffixes Display
text: We display the samples of suffixes optimized through our method in Table 7

heading: Personality Trait Facet Text
text: Algorithm 1 Aggregated Gradient-based Search Input: Data (X,Y ), batch size B, initial suffix s1:m, loss L, vocabulary V , sample size D, Epoch E k V := V ′ ▷ Prune the original search space for e = 1, . . . , E do
for i = 1, . . . ,m do Candi := Top-k(−ΣBj=1∇esiL(Y
j |[Xj ; s])) ∈ V ′ ▷ Obtain candidates for each position end for for d = 1, . . . , D do
s̃(d) := s ▷ Initialize with the last best suffix i = Uniform(m) ▷ Random Sample replacement position s̃ (d) i := Uniform(Candi) ▷ Random sample replacement token
end for s := s̃(d
⋆), where d⋆ = argmind(Σ B j=1L(Y j |[Xj ; s̃ (d) i ])) ▷ Choose the best replacement
end for Output: Optimized suffix s

heading: Future Work
text: Therefore, how to search for appropriate suffixes that are more human-readable can be a future work. In addition, we conduct experiments ranging from 1. 1B to 13B. However, behaviors in models smaller or larger than this range can be different. Finally, the applications
of our method on more datasets is under-explored and we leave it to future