Prompt Optimization via Adversarial In-Context Learning (2024)

Abstract: We propose a new method, Adversarial InContext Learning (adv-ICL1), to optimize prompts for in-context learning (ICL). Inspired by adversarial learning, adv-ICL is implemented as a two-player game between a generator and discriminator, with LLMs acting as both. In each round, given an input prefixed by task instructions and several exemplars, the generator produces an output. The discriminator then classifies the generator’s input-output pair as model-generated or real data. Based on the discriminator’s loss, a prompt modifier LLM proposes possible edits to the generator and discriminator prompts, and the edits that most improve the adversarial loss are selected. We show that applying adv-ICL results in significant improvements over state-of-theart prompt optimization techniques for both open and closed-source models on 13 generation and classification tasks including summarization, arithmetic reasoning, machine translation, data-to-text generation, and the MMLU and big-bench hard benchmarks. In addition, our method is computationally efficient, easily extensible to other LLMs and tasks, and effective in low-resource settings

heading: 1 Introduction
text: Generative Adversarial Networks (GANs) and adversarial learning (Goodfellow et al., 2014) have driven significant progress across a range of domains, including image generation (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017), domain adaptation (Ganin et al., 2016; Tzeng et al., 2017; Xie et al., 2017; Louppe et al., 2017), and enhancing model robustness (Szegedy et al., 2013; Biggio et al., 2013; Carlini & Wagner, 2017; Madry et al., 2018). At its core, adversarial learning frames training as a minimax game
∗Equal contribution. † Equal advising.
1Our codes will available at https://github.com/ zhaoyiran924/Adv-In-Context-Learning.
between a generator and a discriminator. The generator aims to generate output realistic enough that the discriminator classifies it as real (i.e., not generated), while the discriminator aims to accurately differentiate between generator output and real training samples. After each round, the parameters of both models are updated based on an adversarial loss, and the process repeats. As the generator improves, the discriminator improves alongside it, finding “weak spots" in generator output that may go undiscovered in non-adversarial training, ultimately resulting in better generator outputs.
Despite success in other domains, applying adversarial learning to pre-training LLMs is impractical due to the data and computational overheads associated with training two models. Particularly for novel tasks where data is often scarce, it is desirable to have methods that can improve model performance using limited data. In this work, we solve this problem by applying adversarial learning to in-context learning (ICL) (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; Beltagy et al., 2022; Liu et al., 2023), which has shown to be an effective method to improve model performance with few training samples. Though, effective, ICL has shown to be sensitive to changes in prompts (Deng et al., 2022; Pryzant et al., 2023). We introduce Adversarial In-Context Learning (adv-ICL), which applies insights from adversarial learning to prompt optimization for ICL. adv-ICL keeps model parameters fixed and instead updates model prompts in an adversarial manner. This alleviates compute and data requirements, while still allowing improvements in model performance.
adv-ICL uses an adversarial objective and three main modules, implemented as LLMs, to optimize a model’s prompt for a given task, as shown in Figure 1. The first module is a generator (G), which is tasked with generating realistic, task appropriate output given a task instruction and an input. The second is a discriminator (D) which has the goal
7308
of classifying its inputs as real or produced by G. Finally, there is a prompt modifier M which is responsible for updating the prompts to G and D. As in typical adversarial learning, the learning objective is set up as a minimax game between G and D. In each round, G produces an output based on an input and a prompt consisting of a task instruction and several example inputs and outputs. D then classifies the pair constructed of the original input and G’s output as generated or real. Finally, M produces a number of possible updates to G and D’s prompts, the updates that most improve the adversarial loss from D’s classification are selected, and the procedure repeats. Through this iterative update procedure adv-ICL is able to improve G’s prompt, improving task performance.
We evaluate adv-ICL on 13 tasks with various open and closed-source LLMs, finding that advICL outperforms other prompt optimization techniques by large margins across model configurations and tasks. For instance, we increase the accuracy of ChatGPT (OpenAI, 2022) from 71.0% to 74.0% on MMLU (Hendrycks et al., 2021), 79.9% to 82.3% on GSM8K (Cobbe et al., 2021), and 72.1% to 74.0% on BBH (Suzgun et al., 2022). Importantly, adv-ICL requires very few iterations and training samples, increasing performance significantly after only five rounds of training on twenty data points. Finally, adv-ICL is easy to implement, encouraging its use in real-world applications.

heading: 2 Adversarial In-Context Learning

heading: 2.1 Background: In-Context Learning
text: Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; OpenAI, 2023) have demonstrated strong downstream task performance through conditioning on a small number of demonstrations in the input prompt, a paradigm referred as in-context
learning (ICL) (Beltagy et al., 2022; Liu et al., 2023). ICL streamlines the adaptation of a generalpurpose LLM to a specific task without the need for feature engineering or additional model training. Formally, given a specific task, consider an LLM GU (the generator), driven by a prompt U = (IG, xG1 , y G 1 , · · · , xGk , yGk ), where IG is the task instruction, xGi is a sample input, and y G i is the corresponding sample output. GU ’s output for a new input x, then, is determined by the instruction and the exemplars in U , making the choice of U crucial in determining GU ’s downstream performance (Deng et al., 2022; Pryzant et al., 2023).

heading: 2.2 Adversarial Training Objective
text: adv-ICL optimizes the generator’s prompt using an adversarial approach inspired by GANs (Goodfellow et al., 2014)—in particular cGAN (Mirza & Osindero, 2014) and BiGAN (Donahue et al., 2016) where the discriminator deals with the conditional and joint distribution of an input and output. As for GANs, it is essential to optimize both the discriminator and generator in the adv-ICL framework concurrently, to make sure they reach a desired optimal state. To assess the output of GU , we employ
a discriminator, DV , which attempts to classify GU ’s output as real or generated.
Like GU , DV is an LLM driven by a prompt V = (ID, xD1 , y D 1 , z D 1 , · · · , xDk , yDk , zDk ), where ID is a task instruction, xDi a sample input, y D i its corresponding output, and zDi a label of “real” or “generated” representing whether yDi is from a real sample or generated by GU . DV uses a GANinspired loss function J , formally defined as:
J (DV , GU ) = Ex,y∼pdata log ( DV (x, y) )
+ Ex∼pdata log ( 1−DV ( x,GU (x) ))
(1) where pdata is the distribution of real data. Note that DV is designed for the binary decision problem of classifying the input as generated or real. As shown in Figure 2, in our prompt, we represent the choices as two options: “(A) real” or “(B) generated”. This allows us to evaluate the classification probability using the generation probability of option (A), where DV (x, y) = 1 indicates a real sample. Therefore, in order for GU to improve its performance, its goal is for DV to mis-classify its outputs as real as often as possible (i.e. minimizing J ). In contrast, DV ’s objective is to increase J , indicating improved classification ability. Formally, this adversarial training objective can be expressed as the following minimax game:
min U max V
J (DV , GU ) (2)
Since the discriminator is powered by a large language model with enough capacity, achieving the optimal solution for this minimax objective indicates that the generator’s output, when paired with its input, is indistinguishable from a real sample.

heading: 2.3 Adversarial In-Context Learning Optimization
text: Whereas GANs optimize model parameters with backpropagation, adv-ICL does not update GU and DV ’s parameters, instead updating their prompts in each training iteration. This requires a number of differences in our optimization process. First, we consider a setting where we have access only to model outputs and generation probabilities, making it impossible to use backpropagation to update U and V . Therefore, we employ a third LLM to serve as the prompt modifier, M . Given a prompt’s task instruction I or demonstration (x, y) as input,
M generates r possible variations. The adversarial loss is recomputed for each variation by substituting the variation into the original prompt, and the modification that improves the adversarial loss the most is returned, following Gonen et al. (2022).
We refer to our optimization algorithm as Adversarial In-Context Learning Optimization (adv-ICL; Algorithm 1), which can be seen in pseudocode form in Algorithm 1. The entire process is as follows: Given the initial generator prompt U , and discriminator prompt V , we run T training iterations. At each iteration, we first sample m pairs of data points from our training samples to compute the adversarial training loss J (GU , DV ,m). We then optimize the loss by using M to modify both the task instruction and demonstration portions of the prompts for the discriminator and generator.

heading: 2.4 Theoretical Analysis
text: In this section, we present an analysis of whether a minimax objective can achieve equilibrium in in-context learning as is possible in the original GAN scenario. Let pdata be the distribution of the training data, and pg be of the generated data from G. We assume D, G, and M are models with infinite capacity and strong enough in-context learning capabilities, where the prompts powering D and G are iteratively updated using M following algorithm 1. We further assume that: (i) M is powerful enough to modify the initial prompt of D/G, covering all possible prompt variants; (ii) There exists a prompt P for D/G that given P , D/G can achieve the globally optimal result; (iii) M can generate P by which D/G achieves the globally optimal result. We prove the following:
Proposition 1. (Motivated by (Goodfellow et al., 2014)) If G and D have enough capacity, and at each training step, the discriminator is allowed to reach its optimum D∗ given G, and pg is updated so as to improve the criterion
J (D∗, G) = Ex,y∼pdata log ( D∗(x, y) )
+ Ex∼pdata log ( 1−D∗ ( x,G(x) )) (3)
then pg converges to pdata.
The full proof for proposition 1 can be found in Appendix A.1. We conclude that with strong enough in-context learning LLMs D,G,M , advICL converges. In practice, convergence in adversarial training must be studied empirically.
Algorithm 1 Adversarial In-Context Learning Optimization
Input: U = (IG, xG1 , yG1 , · · · , xGk , yGk ), V = (ID, xD1 , yD1 , zD1 , · · · , xDk , yDk , zDk ). Input: Generator GU , Discriminator DV , Prompt Modifier M . Input: Prompts for M to sample new instructions or demonstrations Pi/Pd. Input: Training iterations T , samples used per iteration m, number of new sampled prompts r. Input: Set of limited samples S
1: for T training iterations do 2: Sample m data points from S to compute J(GU , DV ,m). 3: // Optimize the instruction ID for DV 4: Generate r new instructions: {I1, I2,...,Ir} = M(ID, Pi). 5: Substitute In to V ∀n ∈ {1, 2, ..., r} to compute the loss Jn(GU , DV ,m) 6: Jj = maxn Jn(GU , DV ,m) 7: Update ID by Ij if Jj > J . 8: // Optimize the demonstrations (xDi , y D i , z D i ) ∀i for DV
9: for i ∈ range(k) do 10: Generate r new demonstrations: {(xi1, yi1, zi1),...,(xir, yir, zir)} = M((xDi , y D i , z D i ), Pd). 11: Substitute (xin, yin, zin) to V ∀n ∈ {1, 2, ..., r} to compute the loss Jin(GU , DV ,m) 12: Jjn = maxi Jin 13: Update (xDi , y D i , z D i ) by (xij , yij , zij) if Jjn > J . 14: end for 15: // Similarly optimize U for GU so that J(GU , DV ,m) decreases. 16: ... 17: end for Output: The optimized prompt U for the Generator GU .

heading: 2.5 Zero-shot Prompt Modification
text: We leverage LLM instruction-following abilities to generate r variants of a task instruction/demonstration in a zero-shot manner. We use different prompt templates for generating instructions, openended question-answer pairs, and multiple-choice question pairs. Specifically, suppose that P is an input task instruction or demonstration (open-ended or multiple choice) to be modified and M is the prompt modifier generating the variants, with its
modifier prompt PM . Then, M is prompted to generate r variants of P : {P1, ..., Pr} = M(P, PM ). An example is shown in Figure 3, with full prompts in Appendix A.3. We also tested using successful prompts from previous optimizations as feedback for the next iteration (Appendix A.5), following Pryzant et al. (2023); Yang et al. (2023a), but found that performance fell short compared to not integrating them.

heading: 3 Experimentation

heading: 3.1 Experimental Setup
text: Datasets. We test adv-ICL on 13 traditional NLP tasks in four main categories: generation, classification, reasoning, and challenging NLP evaluation suites. For generation, we select XSUM (Narayan et al., 2018) and CNN/Daily Mail (CNN for short) (Nallapati et al., 2016) as our text summarization benchmarks; WebNLG (Gardent et al., 2017) and E2E NLG (Novikova et al., 2017) as our data-to-text generation datasets; and LIRO (RO → EN) (Dumitrescu et al., 2021) and TED Talks (IT→ JA) (Ye et al., 2018) as our machine translation benchmarks. In the classification category, we use YELP-5 (Zhang et al., 2015), COPA (Roemmele
et al., 2011) and WSC (Levesque et al., 2012). For reasoning tasks, GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021) are chosen as arithmetic reasoning benchmarks. Finally, we also evaluate our method on two challenging evaluation suites: MMLU (Hendrycks et al., 2021) and BIG-bench Hard (BBH) (Suzgun et al., 2022). Due to computational and budget limitations, except for GSM8K and SVAMP, each benchmark is evaluated on a maximum of 1,000 test samples randomly chosen from the test set. In our preliminary experiments, we found that the empirical results on the sampled test set is aligned with performance on the whole test set. The exact number of testing samples for each task is presented in Appendix A.3.
A main advantage of ICL is that it can generalize to new tasks with limited training examples, as may be the case for novel tasks. To make our method applicable in such settings, we use 20 labeled samples during training. For our baseline methods, we assume access to at most 100 labeled data samples for each benchmark except BBH, similar to previous prompt optimization works (Xu et al., 2022; Pryzant et al., 2023). For BBH, we assume access to three chain-of-thought data samples per task.
Backbone Models. We test widely-used open and closed-source LLMs as our backbone models. For open-source, we use Vicuna-13B v1.5 (Zheng et al., 2023) —a chat model fine-tuned on top of LLaMa-2 (Touvron et al., 2023b). For closedsource, we use text-davinci-002 and ChatGPT (gpt3.5-turbo-0613) (OpenAI, 2022) built on top of GPT-3 (Brown et al., 2020). For each backbone model except ChatGPT, we use the same model for the generator, discriminator, and prompt modifier in the adv-ICL setup. Since ChatGPT does not provide the probabilities of its generated tokens, which is required for computing the adversarial loss, we employ text-davinci-002 as the discriminator, and ChatGPT is the generator and the prompt modifier.
Baselines. We compare adv-ICL with six baselines: (i) Few-shot prompting, with Chain-ofThought (CoT) (Wei et al., 2022) for reasoning tasks; (ii) Utilizing ROUGE-L score (Lin, 2004) (ROUGE-L) as the criteria to optimize the instruction and demonstrations for each task on a small sampled labeled set; (iii) Similarly, using Perplexity (Perplexity) as the criteria following Gonen et al. (2022); (iv) Genetic Prompt Search (GPS) (Xu et al., 2022), a genetic optimization method based
on the log-logits or accuracy; (v) Automatic Prompt Optimization (APO) (Pryzant et al., 2023), which uses data to generate text “gradients” evaluating the current prompt, and then utilize them to signal the models to edit the prompt in the opposite semantic direction. (vi) Automatic Prompt Engineer (APE) (Zhou et al., 2022), which automatically generates instructions and selects via evaluation scores.2
We ensure that all methods use a similar number of labeled samples, while the exact number of training samples depends on the specific algorithm. For GPS and APO, we sample 32 and 50 labeled data examples for validation, following (Xu et al., 2022; Pryzant et al., 2023). For ROUGE-L and Perplexity, we sample 80 data examples for validation. For YELP, WSC, GSM8K, SVAMP, where the benchmarks do not have enough labeled examples, we sample from their limited training set instead. APO requires additional training data for error samples. For fair comparisons, we use the same training data with adv-ICL. More implementation details for baselines are presented in Appendix A.2.
Prompt Initialization. We follow prior works to employ a set of initialized prompts. For MMLU and BBH, we employ the open-sourced prompts that come with the original papers. For GSM8K and SVAMP, we follow the chain-of-thought paper Wei et al. (2022) which employs human-written prompts. For the remaining benchmarks, we utilize prompts from Super-NaturalInstructions (Wang et al., 2022), in which instructions and demonstrations are chosen by domain experts. All the initial prompts are also used for our baseline few-shot experiments. The exact number of shots used for each benchmark is presented in Appendix A.3.
Evaluation Metrics. For the generation tasks, we evaluate the performance by ROUGE-L score (Lin, 2004), following Wang et al. (2022). For classification tasks, we use accuracy as the evaluation metric. For MMLU and BBH, we follow Hendrycks et al. (2021); Suzgun et al. (2022) and report the averaged performance among tasks.
Hyperparameters. Following the hyperparameter selection results in Section 4, we set number of training iterations T = 3 and training samples per iteration m = 5 for all tasks except BBH, where we set T = 3,m = 3 given that the training set
2As APE only polishes task instruction, we compare APE with Adv-ICL on GSM8K, MMLU and WebNLG.
contains only 3 samples. In all experiments, the prompt modifier samples from r = 5 prompts.

heading: 3.2 Main Results
text: We present the main empirical results on a set of classification, generation and reasoning tasks in Table 1, MMLU in Table 2, and BBH in Figure 4.
Generation Tasks. As shown in Table 1, advICL significantly outperforms all baseline methods across all backbone models, achieving 2.3%, 2.9%, 1.2% average absolute improvements for text-davinci-002, Vicuna, and ChatGPT respectively. We observe that adv-ICL achieves the most significant improvements for Summarization and Data-to-Text. Specifically, for text-davinci-002, adv-ICL outperforms the best baseline by 3.8% on XSUM and 3.1% on WebNLG. For Vicuna v1.5, adv-ICL achieves an improvement of 5.6% on the two data-to-text generation tasks WebNLG and E2E NLG. For ChatGPT, we achieve an improvement of 3.0% on XSUM and 2.8% on the E2E NLG generation task when compared to the vanilla few-shot baseline with no prompt optimization applied. We hypothesize that ChatGPT may obtain smaller absolute improvements when compared to other prompt optimization methods due to the misalignment between the backbone models of the generator and the discriminator. However, given that ChatGPT is the most widely used LLM and undergoes constant upgrades, it should be expected that improving ChatGPT is more difficult.
Classification Tasks. For classification tasks, adv-ICL also brings significant improvements over all SOTA prompt optimization techniques across all models with 4.0%, 2.9%, 0.8% average absolute improvements respectively. The most significant performance improvement is obtained using the text-davinci-002 backbone. The 2.9% improvements with Vicuna also illustrate the effectiveness of our proposed method on open-sourced models. The improvements of the three backbone models on classification tasks are relatively balanced.
Reasoning Tasks. For reasoning tasks, we observe a 2.7% and 2.0% absolute improvement on GSM8K and SVAMP, with text-davinci-002. Likewise, significant gains are observed with ChatGPT, achieving a 2.4% increase on GSM8K and a 1.1% boost on SVAMP. In the case of Vicuna, it achieves 3.2% absolute improvement on GSM8K and 3.3% absolute improvement on SVAMP. The effective-
ness of adv-ICL for reasoning tasks, particularly when coupled with CoT prompting, where the prompt includes detailed intermediate reasoning steps, demonstrates its ability to optimize complex prompts. This hints at potential for applying advICL to more advanced prompting methods.
MMLU & BBH. We summarize the results on MMLU in Table 2. We improve the average performance from 69.8% to 73.1%, achieving performance improvements on 51 subjects out of 57 subjects with ChatGPT. For BBH, as shown in Figure 4, adv-ICL achieves an accuracy of 70.6% where the baseline method achieves an accuracy of 68.2% with ChatGPT and chain-of-thought prompting. The detailed results on MMLU and BBH are in Appendix A.5. Note that for BBH, only three data examples are provided with the dataset. Consequently, we use the same three examples as the initial data for both the generator and discriminator. Additionally, these 3 examples are the only real data examples utilized when estimating the
objective. Despite this, we achieve substantial improvements on this task. This demonstrates the broad applicability of our method. In real-world scenarios with limited access to training samples our approach can still be effectively applied.

heading: 4 Analysis
text: In this section, we examine several design choices of adv-ICL. We further discuss the necessity of the discriminator in Appendix A.4, as well as an extended set of analyses in Appendix A.5.
Optimizing Instruction / Demonstration Only. As instruction and demonstration data are both widely used in prompts, we examine the importance of optimizing both components. We use ChatGPT and compare our method with the prompt optimization method APE (Zhou et al., 2023). We measure performance on WebNLG, GSM8K (with CoT), and MMLU. As shown in Figure 5, we find that updating only the instruction or demonstrations makes the model perform suboptimally. Additionally, optimizing demonstrations is more effective than optimizing instructions for WebNLG and MMLU while the reverse is true for GSM8k. We hypothesize that this is because generated reasoning chains may contain errors and the correctness of the generated answers with respect to questions is critical for the model’s performance (Min et al., 2022). That said, adv-ICL achieves significant performance improvements for GSM8k in both cases.
Choosing Different Models for the Discriminator and Generator. Given a generator, it is also important to answer how we can select a suitable discriminator to deploy our framework. In our main
experiments, we chose the same model as the generator for all the base models, except ChatGPT. We hypothesize that since both discriminator and generator compete with each other in adv-ICL, it is essential to balance their learning. To understand more whether we can use a discriminator different from the generator, we conducted experiments in Table 7 dividing into two groups of using a stronger generator and a stronger discriminator in adv-ICL. We observe with a stronger generator the performance is likely improved contrasting to with a stronger discriminator, the performance is potentially harmed. Overall, we suggest that the discriminator and generator should be chosen such that they are on the same performing level. A significant difference in their performance can drastically lower the overall framework’s outcome.

heading: Ablation Studies on Number of Iterations and
text: Data Samples. As shown in Algorithm 1, adv-
ICL involves three main hyperparameters: the number of training iterations T , the number of data points used per iteration m, and the number of new versions sampled for each instruction/demonstration r. We fix r = 5 and analyze the best performing combination of T and m using grid search for T ∈ {1, 3, 5} and m ∈ {1, 2, 5, 10}.
We measure adv-ICL’s performance on a validation set S constructed from one representative task per category: WebNLG for generation, GSM8K for reasoning, and MMLU for classification. We use 80 data samples from both WebNLG and GSM8K3. For MMLU, we sample 16, 16, 17, 19 from the validation sets of abstract_algebra, business_ethics, econometrics, formal_logic resulting in 228 total samples in S.
We conduct experiments with ChatGPT and Vicuna as the backbone models. As shown in Table 3, we observe the best performance with T = 3 and m = 5 for both settings. This demonstrates that our method works effectively without requiring many training iterations and data samples. We further supplement S with summarization and translation tasks (80 samples each) to conduct additional hyperparameter search experiments with Vicuna, as detailed in Appendix-Table 11. The results also indicate that the optimal settings are T = 3 and m = 5. We provide an explanation of why training with too large T or m might harm model performance in Appendix A.5.
Qualitative Analysis. For an intuitive understanding of how our prompt optimization progresses, we show examples of prompts for
3GSM8K does not come with a validation set, so we sample from the training set instead.
WebNLG changing over iterations in Appendix A.5. The prompt modifier significantly alters the generator’s prompt in two iterations by simplifying the instruction and adding a more specific requirement. The demonstrations are either replaced with a completely new one or are refined.

heading: 5 Related Work
text: Adversarial Training. Adversarial training has been widely used in image generation (Goodfellow et al., 2014; Radford et al., 2015; Arjovsky et al., 2017), domain adaptation (Ganin et al., 2016; Tzeng et al., 2017; Xie et al., 2017; Louppe et al., 2017), and improving model robustness (Szegedy et al., 2013; Biggio et al., 2013; Carlini & Wagner, 2017; Madry et al., 2018). However, previous work shows that it often harms generalization (Raghunathan et al., 2019; Min et al., 2021). In NLP, there is an increasing interest in adversarial training; however, most current research focuses on its effect on generalization (Cheng et al., 2019; Wang et al., 2019; Jiang et al., 2020), and finetunes models (Jin et al., 2020; Liu et al., 2020), which is impractical for LLMs. In contrast, advICL optimizes prompts and demonstrates strong generalization under different conditions.
Prompt Optimization. In-context learning (Liu et al., 2023) has sparked interest in prompt optimization (PO) techniques (Qin & Eisner, 2021; Deng et al., 2022; Lu et al., 2022; Xu et al., 2022; Pryzant et al., 2023; Yang et al., 2023a; Wang et al., 2024) for enhancing the performance of large language models (LLMs). Previous PO works fall into two categories: (1) continuous prompts; and (2) discrete textual prompts. Notable works optimizing continuous prompts include (Qin & Eisner, 2021; Liu et al., 2021; Lester et al., 2021). However, as model sizes increase, this approach becomes more computationally expensive. For very large language models, Xu et al. (2022) propose Genetic Prompt Search (GPS), a gradient-free prompt optimization method. Additionally, Pryzant et al. (2023) introduce Automatic Prompt Optimization (APO), utilizing text “gradients" to evaluate and modify prompts. We compare adv-ICL with GPS and APO. Other techniques like Automatic Prompt Engineer (Zhou et al., 2023) optimize only task instructions. We compare this with a variant of advICL. RL-based prompt optimization baselines like (Deng et al., 2022; Lu et al., 2022) are excluded due to involving additional MLP training and lack-
ing a universal reward. Finally, PO algorithms have been recently developed to defend against jailbreaking attacks, for example, (Zou et al., 2023; Zhu et al., 2023; Zhou et al., 2024), but use different problem settings and are not directly comparable. Finally, our method is indirectly related to demonstration selection works for ICL (Li et al., 2023; Yang et al., 2023b). Since adv-ICL does not focus on selecting the optimal combination and order of demonstrations, we do not compare our method with these approaches. However, demonstration selection algorithms can be developed concurrently and integrated with our method.

heading: 6 Conclusion
text: In this work, we introduce adv-ICL, an adversarial training framework for in-context learning using large language models. Our method has demonstrated empirical success across a diverse range of tasks and outperforms previous SOTA prompt optimization methods significantly. Effective with limited data samples and a very small number of training iterations, adv-ICL holds promise for a wide array of real-world applications.

heading: Limitations
text: One limitation of our work is that adv-ICL requires the component LLMs to follow human instructions well in performing their subtasks. However, we foresee that this limitation is going to be tackled by cutting-edge LLMs in the present and near future as LLMs are going to be more powerful. Additionally, choosing a good combination of {Discriminator, Generator} may require empirical experiments. In this work, we suggest that the same model can be used as both Discriminator and Generator. This offers strong performance as observed because both models are going to learn together well. However, in reality, many closed-source models like ChatGPT can be used as the Generator, but not the Discriminator. Choosing an optimal Discriminator in these cases requires deeper understanding as well as empirical experiments. We leave this exploration for future works.

heading: Ethical Considerations
text: It is possible that this method could be used to optimize prompts for harmful purposes such as mis/disinformation generation, hatespeech, or privacy violating use cases. While this is not what our method is designed for, there is no way to prevent
this type of misuse. While our method could also improve the efficiency and efficacy of bad actors, we do not anticipate that there is anything inherent to adv-ICL allowing it to be more effective in these settings than in other, positive, settings.

heading: Acknowledgements
text: This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC2023-010-SGIL) and the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207). Do Xuan Long is supported by the A*STAR Computing and Information Science (ACIS) scholarship. We thank the anonymous reviewers for the constructive and helpful feedback.

heading: A Appendix

heading: A.1 Theoretical Proofs of the Convergence
text: In this section, we theoretically analyze whether such a minimax objective in the form of in-context learning can achieve the desired equilibrium as in the original GAN scenario. We assume access to models with infinite capacities powering the discriminator D, generator G, and prompt modifier M and that in each iteration, we sample a sufficient number of prompts from M to update both G and D. Let pdata be the distribution of the training data, and pg be the distribution of the generated data from G.
Considering a language model which can be D or G with powerful enough in-context learning capabilities, given a task, we further assume that:
1. M is powerful enough to modify the initial prompt of D/G, covering all possible prompt variants.
2. There exists a prompt P for D/G that given P , D/G can achieve the globally optimal result.
3. M can generate P by which D/G achieves the globally optimal result.
The assumption 3 is a result of assumptions 1, and 2, and the assumption about our access to infinite capacities language models. Indeed, given D/G, from assumption 2, there exists a globally optimized prompt P for it such that it can achieve the globally optimal state for a given task. Furthermore, since M is powerful enough in modifying the initial prompt (ass. 1), plus M samples a sufficiently large number of prompts for each iteration (ass. 2), M can generate P with a non-zero probability, which concludes the assumption 3.
With the above assumptions, we prove the following results.
Proposition 2. (Goodfellow et al., 2014) For G fixed, the optimal discriminator D can be described in a closed form, denoted as D∗.
Proof for Proposition 2. Adapted from (Goodfellow et al., 2014): For a fixed G, the training objective for the discriminator D is maximizing the adversarial loss J (D,G) (Equation (1))
J (D,G) = Ex,y∼pdata log ( D(x, y) )
+ Ex∼pdata log ( 1−D ( x,G(x) ))
= Ex,y∼pdata log ( D(x, y) )
+ Ex,y∼pg log ( 1−D ( x, y ))
=
∫
x pdata(x) logD(x, y)
) dx
+
∫
x pg(x) log
( 1−D ( x, y )) dx
=
∫
x pdata(x) logD(x, y)
+ pg(x) log ( 1−D ( x, y )) dx
The function y = a log(x) + b log(1 − x) for (a, b) ∈ R2and(a, b) ̸= {0, 0} achieves its maximum in [0, 1] at aa+b . Therefore, D ∗(x) has a closed form, which is D∗(x) = pdata(x)pdata(x)+pg(x) .
Proposition 3. (Motivated by (Goodfellow et al., 2014)) If G and D have enough capacity, and at
each training step, the discriminator is allowed to reach its optimum D∗ given G, and pg is updated so as to improve the criterion
J (D∗, G) = Ex,y∼pdata log ( D∗(x, y) )
+ Ex∼pdata log ( 1−D∗ ( x,G(x) )) (4)
then pg converges to pdata.
Proof for Proposition 3. At each training step, the optimal D∗ can be achieved via editing its input prompt by M . Considering the loss function J (D∗, G) as a function in pg, then J (D∗, G) is convex in pg. Since G is powerful enough that there exists a prompt P sampled by M such that G can achieve the globally optimal loss J (assumption 2), with an optimal D∗, we can obtain the corresponding best G. Furthermore, J (D∗, G) is convex in pg, plus the global optimal of G can be obtained, with a sufficiently large enough number of prompts sampled and training iterations, pg converges to pdata.

heading: A.2 Baseline Implementation
text: In this section, we present our implementation details for the baselines. First, among the benchmarks we used, the following datasets do not have any validation set with sizes larger than or equal to 80: YELP, WSC, GSM8K, SVAPM. Therefore, we randomly sample 100 data cases from their training sets, to create their validation sets.
Each baseline requires a development set to decide which prompt(s) is/are the best at each optimization iteration. For GPS and APO, we sample 32 and 50 data samples respectively from the validation set of each benchmark, following (Xu et al., 2022; Pryzant et al., 2023). For ROUGE-L and Perplexity, we sample 80 data samples, also from each validation set. Additionally, among the baselines, only APO requires training data for error messages. For a fair comparison with adv-ICL, we use the same training data samples with adv-ICL as training data for APO.
ROUGE-L & Perplexity (Gonen et al., 2022). For these baselines, we utilize ROUGE-L (Lin, 2004) or Perplexity Gonen et al. (2022) as the measurement to optimize the input instruction and demonstrations sequentially. For the instruction, we sample 15 new instructions by paraphrasing following the template:
’Write for me 15 paraphrases of the {initial_instruction}:’. We then select the version which achieves the best result on S as the final instruction. Similarly, for each demonstration, we use the template ’Write 15 paraphrases for the following example. Keep the format as

heading: Input: and Output:. End the answer by So
text: the answer is:’ to sample 15 versions of the original demonstrations, and select the best one on S sequentially until all the demonstrations are optimized. We sample 15 versions for comparisons because our proposed adv-ICL also samples a maximum of 15 versions for the instruction and each demonstration.
GPS (Xu et al., 2022). We run GPS (Xu et al., 2022) on 3 iterations to optimize the instruction and each demonstration sequentially. Denote the original instruction/demonstration to be optimized as O. In the initial step, given the original humanwritten O, we paraphrase it into 10 versions using ’Write for me 10 paraphrases of the {initial_instruction}:’ for instruction, and ’Write 10 paraphrases for the following example. Keep the format as Input: and Output:. End the answer with <END>. So the answer is:’ for demonstration. We then select the top-5 generated O to pass to the first iteration. At each iteration, for each O in the current top−5 Os, we sample 5 new Os by Sentence Continuation strategy (Schick & Schütze, 2021) via using the backbone LLM itself, and select the top−5 Os among 25 Os to the next iteration. Finally, the best-performing O on S is selected as the output instruction/demonstration of the method. It is worth noting that in the original paper from (Xu et al., 2022), top−k with k = 25 was used. However, in our reimplementation, we use k = 5 so that it can be relatively fair to compare GPS with our method (we use r = 5) and other baselines. The template for sampling new prompts via the Sentence Continuation strategy that we used is exactly the same as Xu et al. (2022) provided.
APO (Pryzant et al., 2023). Since our setting assumes that we have access to limited training data samples, we reimplemented a simplified version of the original APO in which the selection step (Pryzant et al., 2023) only be called once, and the samples that we used to train adv-ICL are returned. For simplicity, we call the original instruction/demonstration as O. We run APO to optimize
the instruction and each demonstration sequentially in a given prompt. Given an initial O, and the error samples, we use the backbone LLM to generate feedback consisting of 5 comments as the text "gradient". Integrating this gradient as feedback, we ask the LLM to generate 10 prompt samples. We further utilize the backbone LLM to generate 5 paraphrase versions of the original O, resulting in a total of 15 new Os. Finally, we select the best O evaluated on S. All the prompt templates for generating gradients, integrating feedback, and generating paraphrased prompts are adopted from (Pryzant et al., 2023). For selecting error samples, in the original implementation, Pryzant et al. (2023) compared the generated answer with the groundtruth answer, and the error samples are the ones that have the generated answer different from the ground-truth answers. This is applicable for classification and numerical question-answering tasks, but not the text generation tasks such as summarization, this strategy of selecting error samples is not suitable. Therefore, for summarization, datato-text, and translation tasks, we select one sample that the current prompt brings the lowest ROUGEL score as the sole error sample.
APE (Zhou et al., 2023). For APE, we adopt the implementation on the GitHub4 from (Zhou et al., 2023). We limit the number of instructions sampled to 15 to have fair comparisons with adv-ICL. For the training samples for each task, we use the same samples that we train adv-ICL for APE.

heading: A.3 Supplementary Experiment Details
text: In this section, we provide more details used in the experiments.
Number of Demonstrations for Few-shot Experiments. Number of demonstrations for fewshot experiments of all datasets is listed in Table 4. For generation tasks and classification tasks, We follow the expert-written prompts from SuperNaturalInstruction (Wang et al., 2022). For reasoning tasks, MMLU and BBH, we follow the standard prompts that they propose in their paper or opensource code.
Test Set Statistics. As mentioned in the main paper, we sample a subset of the test set for efficient evaluation. In Table 5, we show the exact numbers of testing samples we used for each task.
4github.com/keirp/automatic_prompt_engineer/tree/main
Prompt Modifier Prompts. Here, we also provide the prompt used in the prompt modifier. The prompt is as follows:
• Modifying instructions: Generate 5 variations of the following instruction while keeping the semantic meaning. Keep the generated instructions as declarative. Wrap each with <START> and <END>..
• Modifying open-ended QA pairs: Generate 5 variations of the following example to make them more representative. Keep the format as Input: and Output:. Wrap each with <START> and <END>..
• Modifying MCQ pairs: Generate 5 variations of the following multiple-choice question and the answer to make them more representative. Keep the format as multiple-choice question and the answer. Keep the format as Input: and Output:. Wrap each with <START> and <END>..
Extended Experimental Details. For OpenAI API models, ChatGPT (gpt-3.5-turbo-0613) with chat completion mode and text-davinci-002 with text completion mode were called at temperature 0.6. For open-source baselines, Vicuna v1.5 13B was used with a window size of 1024. We use Nucleus Sampling (Holtzman et al., 2020) as our decoding strategy for all the models with a p value of 0.9.

heading: A.4 Why the Discriminator Works?
text: We further conduct experiments (Table 6) to verify whether the prompt modifier module work as expected. Specifically, we remove the discriminator and only employ a prompt modifier to repeatedly optimize the prompt.
In most cases, removing the discriminator and relying solely on the prompt modifier under Vicuna and ChatGPT leads to a decline in performance. This observation highlights the importance of the discriminator and adversarial loss in the optimization process.

heading: A.5 Extended Experiments
text: Choosing Different Models for the Discriminator and Generator. Table 7 presents our experimental results.
Reliability of The Results. We rerun our experiments with adv-ICL three times on WebNLG, RO → EN, YELP, GSM8K. The results are presented in Table 8.
The results clearly demonstrate that adv-ICL consistently delivers stable outcomes, thereby highlighting its reliability in faithfully reproducing our experimental findings.
Providing More Feedback to the Prompt Modifier. We conducted an experiment that involved integrating the most successful prompts from previous iterations as feedback for the next iteration. In this process, we utilized previous best-performing prompts, namely P1, P2, ..., Pk, as inputs to the prompt constructor module in order to generate the (k + 1)-th prompt, denoted as {P1, ..., Pk}. The
template for optimizing task instruction is shown as follows, similar to the prompt for optimizing demonstrations. Diversify the task instruction to be
clearer. Keep the task instruction as declarative.
Task instruction: P0 Improved task instruction: P1 . . .
Task instruction: Pk−1 Improved task instruction: Pk Task instruction: Pk Improved task instruction:
We applied the method to four representative tasks WebNLG, RO → EN, YELP, GSM8K using both Vicuna and ChatGPT models. The obtained results for are illustrated in Table 9.
In the case of Vicuna, incorporating additional feedback into the prompt modifier proves effective for tasks such as translation and classification. However, this approach falls short when applied to data-to-text and reasoning tasks. On the other hand, for ChatGPT, augmenting the prompt modifier with more feedback does not yield improved performance. This can be attributed to ChatGPT’s strong zero-shot prompt capabilities, which outshine its ability to perform effectively with fewshot prompts.
Ablation Studies on Number of Generated Samples r. We investigate whether generating fewer / more samples in each prompt modification would
affect the model’s performance. Due to the limited resources, we only conducted the experiment on the WebNLG and GSM8k dataset, with r ∈ {1, 3, 5, 10, 20}. The results are shown in Figure 6. We observe that increasing r lead to comparable results.
Why Might too Many Iterations T or Samples m Harm the Performance of Models? We observed this phenomenon in the experiments and were also curious about it. We hypothesize that first, training with too many iterations can cause the model to be overfitting to the task, leading to worse performance on the test samples. Second, adv-ICL, a specialized form of in-context learning, plays a crucial role in enhancing the performance of LLMs by enabling them to learn from the training examples and generate improved prompts. While in-context learning holds great promise, it is essential to acknowledge that increasing the number of training examples does not necessarily guarantee better performance. As demonstrated by (Min et al., 2022), a critical threshold exists for the number of training examples, and surpassing this threshold leads to a decline in performance. Thus, in our specific settings, augmenting the training examples did not yield better results.
Given its inherent complexity and nondeterministic nature, we have put forward a hyperparameter tuning approach, presented in Table 3, aimed at determining these hyper-parameters for new configuration settings.
Prompt Modifier Temperature. Lastly, we examine the influence of the generation temperature for the prompt modifier. Ideally, the prompt modifier should have enough diversity to generate potential improvements for the prompts of both the generator and discriminator. Intuitively, this means we should not use greedy decoding with a temperature of 0 for the prompt modifier. As demonstrated
in Figure 7, a temperature of 0.6 works well, providing a sufficiently large search space while still generating high-quality prompts.
Here we present the detailed results of human evaluation on generated instructions and demonstrations respectively. Details are shown in Table 10. text-davinci-002 and ChatGPT achieve similar performance with the zero-shot prompt modifier, while Vicuna performs a little bit worse but also achieves an acceptable correctness (≥ 80).

heading: Using More Tasks To Select Hyperparameters.
text: Table 11 presents our hyperparameter selection experiments for Vicuna, expanding on the tasks in-
cluded in Table 3 by adding summarization and translation tasks. We observe that (T = 3, m = 5) is also the best combination.
More Qualitative Analysis. We show one example in Figure 8. We also show an additional case of qualitative analysis on Yelp. As shown in 9, the optimization follows a similar pattern with that on the data-to-text task.
Detailed Results on MMLU. In Figure 10, we show the detailed results on MMLU with ChatGPT. As shown in the graph, adv-ICL achieves significant improvements on most tasks.
Detailed Results on BBH. In Figure 11, we show the full results of ChatGPT on BIG-Bench Hard using 5-shot Chain-of-Thought prompting. The baseline achieves an average of 68.2% accuracy while adv-ICL reaches an average of accuracy of 70.6% and never performs worse than the baseline.

heading: Future Work
text: However, we foresee that this limitation is going to be tackled by cutting-edge LLMs in the present and near future as LLMs are going to be more powerful. Additionally, choosing a good combination of {Discriminator, Generator} may require empirical experiments. In this work, we suggest that the same model can be used as both Discriminator and Generator. This offers strong performance as observed because both models are going to learn together well. However, in reality, many closed-source models like ChatGPT can be used as the Generator, but not the Discriminator. Choosing an optimal Discriminator in these cases requires deeper understanding as well as empirical experiments. We leave this exploration for future works