InstOptima: Evolutionary Multi-objective Instruction Optimization via Large Language Model-based Instruction Operators
Authors: Heng Yang
Ke Li
Venue: EMNLP 2023 Findings (2023)
Keywords: instruction optimization
automated instruction generation
evolutionary multi-objective optimization
language model-based operators
TL;DR: This paper propose a evolutionary multi-objective optimization-based instruction generation method

Abstract: Instruction-based language modeling has received significant attention in pretrained language models. However, the efficiency of instruction engineering remains low and hinders the development of instruction studies. Recent studies have focused on automating instruction generation, but they primarily aim to improve performance without considering other crucial objectives that impact instruction quality, such as instruction length and perplexity. Therefore, we propose a novel approach (i.e., InstOptima) that treats instruction generation as an evolutionary multi-objective optimization problem. In contrast to text edition-based methods, our approach utilizes a large language model (LLM) to simulate instruction operators, including mutation and crossover. Furthermore, we introduce an objective-guided mechanism for these operators, allowing the LLM to comprehend the objectives and enhance the quality of the generated instructions. Experimental results demonstrate improved fine-tuning performance and the generation of a diverse set of high-quality instructions.

imstOptima:
Evolutionary Multi-objective Instruction Optimization via
Large Language Model-based Instruction Operators
Heng Yang', Ke Li' 'Department of Computer Science, University of Exeter, EX4 4QF, Exeter, UK (hy345, k k.li) Geieter..eer.ee ac..uk
Abstract
2023; Chen et al., 2023) aim to obtain large num- ber of instructions through automated instruction generation to filter nigh-perrformncc instructions. However, due to the large and non- con-differentiabee textual search space (Ishibashi et al., 2023; Cho et al., 2023), the automated instruction generation and instruction engineering methods (Brown et al., 2020; Liu al., 2023) inefficient and strug- gle to search for various high-quality instructions. Secondly, the objectives of instruction generation are not clear. Current research (Lester et al.. 2021; Gu et al., 2022; Pitis et al., 2023) regards perfor- mance (i.e., metrics) as the sole criterion for in- struction quality. However, model performance alone cannot precisely explain instruction quality. We propose to refine instruction quality by consid- ering fine-grained objectives, such as length and perplexity, Shorter instructions can lower computa- tional costs, especially for large-scale models and datasets. Lower perplexity indicates that instruc- tions are more easily understood by language mod- els. Lastly, the diversity of instructions has been neglected in existing studies, while increasing the diversity of instructions can mitigate adversarial attacks (Wan al., 2023; Zhu et al., 2023) and im- prove instruction robustness (Yu et al., 2022; Zhu et al., 2023). We aim obtain multiple alternative instructions based on multi-objecciiv optimization. which can facilitate comprehensive evaluation instructions.
iustruction-based language modeling has re- ceived significant attention in pretrained lan- guage models. However, the efficiency of in- struction engineering remains low and hinders the development of instruction studies. Re- cent studies have focused on automating in- struction generation, but they primarily aim to improve performance without considering other crucial objectives that impact instruction quality, such instruction length and perplex- ity. Therefore, we propose a novel approach (i.e., TnstOpti that treats instruction gen- eration as evolutionary multi- objective timization problem. contrast to text edition- based methods, approach utilizes large language model (LLM) to simulate instruction operators, including mutation and crossover. Furthermore, introduce objective-guided mechanism for these operators, allowing the LLM to comprehend the objectives and en- hance the quality of the generated instructions. Experimental results demonstrate improved fine-tuning performance and the generation of a diverse set of high-quality instructions.
1 Introduction
With the rapid development of language mod- els (Ouyang et al., 2022; Touvron et al., 2023; Ope- nAL, 2023), instructions (also known as prompts) play crucial role in instruction-based language modeling, and different instructions may lead to significant differences in model outputs (Zhou et al., 2022; Honovich et al., 2022; Wan et al., 2023). For instance, even slightly perturbed in- structions (e.g., synonym substitutions (Wang et al., 2021; Zhou et al., 2021) ) or adversarial attacks (Wan et al., 2023; Zhu et al., 2023)) can result in un- expectedly low performance. However, there three problems regarding instruction-bart based learn- ing that still need be addressed in existing works. Firstly, existing works (Lester et al., 2021; Gu et al., 2022; Zhou et al., 2022, 2023; Li et al.,
To address these three problems, we formulate the task as an evolutionary multi-objective op- timization problem and propose our framework called InstOpt ima. We leverage a large lan- guage model, specifically ChatGPT (OpenAT, 2023), facilitate instruction operations such mutation and crossover. Furthermore, we intro- duce an objective-guided mechanism assist the language model in generating high-quality instruc- tions. In terms optimization objectives for in- struction generation, InstOpt ima incorporates

---

three objectives: performance (metrics), length,
the problem of automated instruction generation through multi-objective optimization.
and perplexity, enabling the exploration of di- verse and high-quality set of instructions. We adopt
2.2 Evolutionary Instruction Optimization
NSGA-II (Deb et al., 2002) in InstOptime ima to obtain a Pareto front of instruction sets.
The workflow of InstOptima is illustrated in Fig. 1. We begin by initializing a parent population of instructions to start evolving. The parent popu- lation is manipulated by LLM-based operators to generate offssring. Subsequently, we employ the non-dominated sort algorithm to rank the combined population and measure the crowdness of instruc- tions. At the end of each generation, we randomly replace some Pareto-front instructions with new instructions to enhance the diversity of the popula- tion (referred to as genes in NSGA-II). We also provide the pseudo code of the InstOptinm ima in Appendix A.4.
To validate the efficacy of InstOpt ima, we conducted experiments on three generation-base classification tasks. The experimental results in- concurrently obtain
dicate that InstOptima a diverse of instructions that outperform the counterparts regarding performance.
In summary, our contributions are as follows: We simulate instruction operators based on an LLM. We also show that the srrective-guides operators help the LLM understand optimization objective values and improve instruction quality. We divide the orientation of instruction search into multiple objectives, such as performance, length, and perplexity, facilitating fine-graie grained control over instruction quality.
2.2.1 Operators for Instructions
To handle the non-differemtiabll text search space, we formulate these operators as text genera- tion task based ChatGPT. In other words, define set of fixed prompts P, P [Pam. Pac. Pem Peel, guide ChatGPT in per- forming the instructions, where Pam, Pde: Pem: Poc are the fixed prompts for the four operations:
We utilize a multi-objective optimization algo- rithm to automatically search for set of high- quality instructions, which could benefit defend- ing against adversarial attacks and improving in- struction robustness.
The codes are available at: https::///tttb.////// com/ http:/yannhnnnggg//m Int/s/Innttpiinnn ima.
Definition Mutation (Pam): This operator mu- tates the definition in an instruction. It can in- volve paraphrases and substitution of new defini- tions.
2 Proposed Method
this section, we first introduce the instruction- based text generation, followed by the details of InstOptima.
Definition Crossover (Pac): This operator com- bines the definitions of two instructions to cre- ate a new instruction. It can involve merging or exchanging parts of the definitions between the parent instructions.
2.1 Instruction-bast based Generation
In text generation-basel tasks', instructions are uti- lized to facilitate in-context learning (Brown et al., 2020) and improve language modeling. An instruc- tion (depicted in the right part of Fig. is repre- sented as = Concat e), where d and e are the definition and example of the target task, re- spectively. d and e are token sequences similar to () ~ D. where x, y, and DD denote the input, output, and task dataset, respectively. The mod- eling of a generation model f(.,-) is defined as follows:
Example Mutation (Pom): This operator per- turbs the example to introduce diversity. It can involve modifications such example substitu- tion, addition, or deletion.
Example Crossover (Pa.): This operator ran- domly selects examples from two instructions create a new instruction.
For instance, we formulate the mutation opera- tion as follows:
d ChhattrrCConnatt (Pam; d))
(2)
ÿ Ix,
(1)
where d is the definition generated based on the original instruction The new instruction is denoted as î, f Concat(d e). The other operators follow a similar formulation to mutation. Further details of the fixed prompts are available in Appendix A.5.
where ý represents the generated output given x and I. In InstOptima, we aim to address
'We validate InstOptima generation-based text classi- fication, and TnstOpt 1 ma can be casily applied to other instruction-bused modeling tasks.

---

Initialion Pogulation
<Definitio>:: Please read
senimmetts:
Population
Foreto front Offepting
-Example>
really like movie!
Sentiment positive
Predict the sentiment
Individiual Replacement
<Input>
Instruction Individuul
Instruction Individual
Initial Instiratiion
<Deffittion::
Let's classify
customers sentiments:
<Esample:
i really like
Sentiment positive
Realve
Crowdanes Evaluacion
Predict the sentiment:
Operations
<Iupor>
Definition Mutation
Offearing
Please
customee
roviews and prodict sentiments: <Example>
Noe- dominated Sorting
really like movie!
Sentiment possliive
Predict the sentineent
Iestruction
Instrationn Individual
<Inpur>
Individual
Example Mutation
Figure I: main framework of InstOptimm im (left) and instruction operation examples (right). The details the workflow that is explained in Section 2.2. The population is composed of individuals of instruction examples.
2.2.2 Optimization Objectives
These operators allow ChatGPT to autonomously decide to emphasize or down-weighe an instruction based on the current objectives F.
We consider three objectives F (m,l,r) in optimization, i.e., metrics (m), length (I), and per- plexity (r) of the instruction.
Performance: We use set of metrics, such as accuracy, f1 score, precision, and recall, ob- tained by evaluating the instruction to calculate the performance objective. The performance ob- jective is represented as the reciprocal of the sum of these metries.
Experimental Setup
We conducted comprehensive set of experiments' to validate the performance of Inst ima. The detailed experiments setups and implementations are described in Appendix A.1.
Length: The length of the instruction is mea- sured in terms of the number of characters. This measurement is fair regardless of the tokenization strategy.
3.1 Baseline Methods
We used random instruction (RanInstruct) generation (i.e., request ChatGPT generates several instructions similar to instructions gen- erated by InstOptima) and no-instruction comparison baselines. The RanInstruct generates five random instructions using the LLM evaluate the same three objectives as InstOptima. The NoInstructablates instruction in the classification-orinneed fine-tuning Flan-T5.
Perplexity: The perplexity of the instruction is measured using the RoBERTa model.
The evaluation of objectives F is shown in the pseudo-code in Appendix A.4 but not depicted in Fig. 1 for simplicity.
(NoInstruct)
2.3 Objective-Guided Instruction Operators
To enhance the performance of ChatGPT through in-context lcarning, propose simple yet effec- tive objective-feedback mechanism. Specifically, we incorporate the fitness values F (m, l,) into the fixed prompts. For example, we can append "Please refer to the objective values: (d) (d2, Fz)" to P. in instruction examples crossover.
Pllease refer to Table the actual implementations of these objective-guidee operators.
"To improve the reproducibility. we release all experimen- tal materials in the supplementary files of the submission, including source ende, experiment logs, and results, optimized instructions.

---

Table 1: The experimental performance of InstOptima. We show the ACCURACY instead of the performance objective for intuitive evaluation. The symbols " and "y indicate 'larger is better' and 'lower is better", respectively. We repeat cach experiment in five rounds and report the average results. The best results are in bold. The ACCURACY is the best accuracy the Pareto-frons while the LENGTH and PERPLEXITY are correlated with the instruction that achieves the best accuracy.
Instoptana LENOTH
RaninstrUct LENGTH
Noinstruce ACCURACY /
MODEL
DATASET
ACCURACY
/
PERPLEXITY
ACCURACY
PERPLEXITY
laptop11
14.9
1,07
82.5g0s
740.
1007,00s
53.10a
Reataurant14
849
82.3g04
328.5
1.15000
19220
9812
89.7.011
402.7
1,09 2001
88.7aas
499.7ama2
1.16amm
96.
Flant5 amall
AGNews
90.2,0
752
L.I
829900
500.049977
1.125001
74.3
SWLL
69.1
295.3
1.14
50.8
307.3
37.9
MNLT
$7.4,0
3858-176
1.12.000
40.0
519.7a688
1009
37.3g0
Laptopid
88410a
207 saa
1.0 2.06
149.7
1.10
62.3
Rurann 14
89.1
259.4..077
1.662000
W7 4g04
589.3
1.11
52850a
8812
94.5 10.1
397.8 8 c
1,08 2.01
385.6
1.12
92.6
Plan15 base
AGNewa
93.5
200.
1.15
90.
485.4
T.T62u02
RK 1ao.
SWL1
86.6103
430.9 s22
1.10
86.4
399.3
1.11
85.9
MNT.T
80.240
288.
1.11
8e07
449.1
1,20aam
24.5
1aptop14
83.2,22
$12.9
1.08-a00
83.1
877.61815
1,05.am
67.8
ChatCPT
RestaucentIn
4
1.09-
92.1
421.6507.4
1.10n0
75.2
3.2
Main Results
RQ1: Do the objective-guided operators help
instruction optimization?
The results in Table 1 show the performance of InstOptima. Overall, InstÖpt im achieves superior objectives based on various base mod- els (e.g., ChatGPT and F1anT5). For exam- ple, it outperforms all baselines on all datasets in terms of ACCURACY. However, for instruction LENGTH and PERPLEXITY, the RanInstrue sometimes achieves better objective values. On the other hand, No Instrruct performs poorly on all datasets in terms of ACCURACY, underscoring the importance of instructions in generation- fine- tuning. Morcover, the ACCURACY objective ex- hibits small intervals but relatively large variances, making it more challenging to optimize. However, existing methods that prioritize performance opti- mization struggle to handle the variances in metrics. On the other hand, the LENGTH objective is eas- ier to optimize due to its significant variations and greater significance. This is because long instruc- tions can result in up twice training times than short instructions. The PERPLEXITY metric ranges within small intervals, indicating a moderate opti- mization challenge, but significantly impacts the understanding of instruction engineers. addition to these three objectives, InstOpt ima can eas- ily accommodate additional objectives for precise control of instruction generation.
Table
The experimental performance of InstOpt ima-imma-- on FlanT5-small. The to- kens "-" " and "4" indicate worse and better objectives than InstOptima.
InstOptina ima NN
DATASET
ACCURACY
LENGTI
PERPLEXITY'
1aptop14
84.400
789.34302
1.07
Restaurant14
83.720.
455.8
1.12200s
SST2
89.6.0.
435.2
1.12
AGNeMa
86.720.
535.81004
1.26
SNLI
69.8
454.0477m
1. +
MNLI
57.3 40.5
065.62288
1.09 2002
To investigate the impact of objective-guidee operators InstÖpting ima, we conducted ab- lative experiments to assess the performance of Inst InstOptiin 1wt.ppptmmma.... which eliminates the objective guidance in the operators. The experimental re- sults on F FlanT5- small are presented in Table 2. Based on the results in Table 1 and Table 2, it is evi- dent that InstOpt innettimaNNN achieves inferior objec- tive values on most datasets, particularly in terms of ACCURACY and LENGTH. However, for the SNLI dataset, Ins imm- -N obtains better re- sults in ACCURACY and PERPLEXITY compared InstOpt i ma. These findings demonstrate the ef- fectiveness of abjective-guidee operators. Nonethe- less, the concept of objective-guidee operators is still in its early stages and warrants further investi- gation in future studies.
Overall, InstOpt ima demonstrates impres- sive performance in instruction optimization across various tasks and datasets.
3.3 Research Questions
In conclusion, the experimental results indicate that objective-gun -uiided operators obtain better per- formance across various datasets.
We further discuss our observations and analysis by answering several research questions.

---

RQ2: Does the number of evolution generations matter in InstOptima?
Pareto fronts between pairwise objectives are pre- sented in Fig.
Dempecne
200
Prroomance
0.20
Legerh
Follenaanss
0.88
0.4
Longhh
Lorech
0.35
0.5
0.4
10
Phaylinly
Purplaily
Plsploaly
Figure The trajectory plots of objective values across different datasets. We plot the trajectories of 10 addi- tional generations using red lines. In these figures, lower objective values indicate better performance.
Figure Visualizations of 2D-Pareto fronts searched by InstOptima
three datasets. The three columns from left to right indicate the results on Lapt op14, SST2 and SNLI datasets, respectively.
Generally, larger number of generations tends result in better objective values after optimiza- tion. We conducted additional training for 10 generations on the papt:p14, 4, SST2, and SNLI datasets to study the significance of number of generations. Based on the experimental results in Fig. 2., in most cases (e.g., Laptoop4 4 and SNLI datasets), observed significant trade-off among the three objectives. However, due to the small scale of the evaluation data and population size, there were large variances in the performance objective (see the left column in Fig. 2). These variances in performance interfere with the conver- gence of the other two objectives, resulting in the absence of clear descending trends for the length and perplexity objectives with an increase in gen- erations. However, this issue can be addressed by increasing the population size, number of genera- tions, and scale of training data.
Overall, there is clear trade-off between in- struction length and perplexity. However, when considering the pairs of performance-length and ceftommannc-perplexiiie there is clear trade-off observed in Fig. This could be attributed the lack of strict trade-offs and the presence of noise fitness points due to the evaluation of metrics on small datasets during optimization. It is expected that this issue can be mitigated when evaluating performance on larger datasets.
Nevertheless, Inst.Opt. consistently dis- covers high-quality instructions in most scenarios, regardless of the loose trade-offs between objective pairs such as performance-length and performance- perplexity. This demonstrates the effectiveness of InstOpt ima in obtaining diverse set of instruc- tions.
4 Conclusion
In conclusion, given the limited evaluation resources, the number of evolution generations showed limited improvement. Instead, it is im- portant to reconcile different objective values to achieve the final instruction population.
We propose a multi-objective instruction optimiza- tion framework to obtain a diversified set of in- structions. To address the challenges posed by the large and non-differentiable text search space, InstOpt ima utilizes objective-guided instruc- tion operators based on LLM, which shows impres- sive performance in instruction generation. How- ever, it important note that multi-objective instruction optimization is still in the early stages and requires further research in the future.
RQ3: Are there trade-offss between different objectives?
To analyze the relationship between different objec- tives, we plot the Pareto front (refer to Fig. 5) of in- structions into three groups. The two-dimensional

---

5 Limitations
Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NewrlPS 20: Proc. of Annual Conference on Neural Information Processing Systems.
The first limitation of Inst ima lies in the po- tential crisis of local optima in the multi-objective optimization. Inst ima initializes the instruc- tion population based on fixed manually crafted instructions, which are then mutated using LLM. Although InstOpt ima has been demonstrated to search for diversified and high-quality instruc- tions in experiments, the essence on fixed initial instructions may lead to traps in local optima dur- ing the multi-objective process. In the future, the generation of initial instruction populations, such employing randomized initial instructions, re-
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2023, Instructzero: Ef- ficient instruction optimization for black-box large language models. CoRR, fbs/2306.03082.
Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, and Jong C. Park. 2023. Discrete prompt optimization via constrained generation for zero-shot re-ranker. CoRR, abs/2305. 13729.
mains topic worth exploring.
The second limitation of InstOptima is re- lated to experimental resources. Due to resource constraints, we only utilized single-round API calls to generate new instructions using LLM. This ap- proach overlooks the contextual information that could help in understanding objective feedback in the instruction generation. We believe that continu- ous dialogue with LLM will significantly improve the quality of instruction generated by LLM. Ad- ditionally, due to the difficulty of accessing L.M,, we conducted experiments with smaller population sizes and fewer iterations, which may underesti- mate the performance of Inst Opt ima.
Hyung Won Chung, Le Hou. Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instruction-finctie language models. CoRR, abs/2210. 11416.
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2002. A fast and elitist multiobjec- tive genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput., 6(2):182-14 197.
A Acknowledgments
This work was supported in part by the UKRI Future Leaders Fellowship under Grant MR/S017062/1 and 13/001113/1; in part by NSFC under Grant 62376056 and 62076056; in part by the Royal Society under Grant IES/R2/212077; in part by the EPSRC under Grant 2404317; in part by the Kan Tong Po Fellowship (KTPIR11231017); and in part by the Amazon Re- search Award and Alan Turing Fellowship.
Yuxian Gu, Han, Zhiyuan Liu, and Minlie Huang. 2022. PPT: pre- pre-triined prompt tuning for few-shot learning. In ACL'22: Proc. of the 60th Annual Meet- ing of the Association for Computational Linguistics, pages 1410-8423, Association for Computational Linguistics.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. CoRR, abs/2205. 10782.
Yoichi Ishibashi, Danushka Bollegala, Katsuhito Su- doh, and Satoshi Nakamura. 2023. Evaluating the robustness of discrete prompts. In EACL'23: Proc. of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2376. Association for Computational Linguis- tics.
References
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 332-642. The Association for Computa- tional Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-effiiien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 3045- 3059. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda

---

Moxin Li, Wenjie Wang. Fuli Feng, Jizhi Zhang, and Tat-Seng Chua. 2023. Robust instruction optimiza- tion for large language models with distribution shifts. CoRR, abs/2305. 13954.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Reprexentations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9):195:1 195:555.
Xiaosen Wang, Yichen Yang, Yihe Deng, and Kun He. 2021. Adversarial training with fast gradient pro- jection method against synonym substitution based text attacks. In AAAI' Proc. of Thirty-iii AAAI Conference on Artificial Intelligence, pages 13997- 14005. AAAI Press
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 16692.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: thate-of-the-art natural language processing. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38 45, Association for Computa- tional Linguistics.
OpenAl. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der. Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrlPS.
Xiaoyan Yu, Qilei Yin, Zhixin Shi, and Yuru Ma. 2022. Improving the semantic consistency of textual ad- versarial attacks via prompt. In International Joint Conference Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pages 1-8, IEEE
Silviu Pitis, Michael R. Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. CoRR, abs/2304.05970.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Har- ris Papageorgiou, lon Androutsopoulos, and Suresh Manandhar. 2014. Semeval- task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, Se- mEval@ COLING 2014, Dublin, Ireland, August 23- 24, 2014, pages 27-35. The Association for Com- puter Linguisties.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference on Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages -47-
Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsich, Kai-Wei Chang, and Xuanjing Huang. 2021. Defense against synonym substitution-basse based adversarial attacks via dirichlet neighborhood ensemble. In ACI/IIC- NLP'21: Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing, pages 5482-5492, Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seatlee Washington, USA, A meeting of SIG- DAT, a Special Interest Group of the ACI. pages 1641-1642,4 ACL.
Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. CoRR, abs/2211. 1.09110.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR. abs/2302.13977.
Yuhang Zhou, Suraj Maharjan, and Beiye Liu. 2023. Scalable prompt generation for semi-supervised learning with language models. In EACL 23: Find- ings of the A Association for Compulational Linguistics, pages 758-769. Association for Computational Lin- guistics.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during in- struction tuning. CoRR, 2atp2220500044..
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang. Hao Chen, Yidong Wang, Linyi Yang, Wei Ye,

---

Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. 2023. Promptbench: Towards evaluating the robust- ness of large language models on adversarial prompts. CoRR, abs/2306.04555
A.2 Additional Experiments for Summarization
A.2.1 Generative Text Summarization
We conducted experiments for text generation task. i.c., generative summarization. To evalu- ate InstOptima, we used three subsets from The GigaWord dataset and the FlanT5- -small model our experiments. In these subsets, the training set contains 5k training examples, while the testing set and validation set each have Ik ex- amples. According to the Rouge1 metric, it is evident that InstOpt ima performs well on the GigaWord d dataset, demonstrating that it is task- agnostic method for multi-ohjective instruction op- timization.
A
Appendix
A.1
Experiment Setup
A.1.1 Datasets
We selected six datasets for three classification tasks. For the aspect-based sentiment analy- sis (ABSA) task, we used the Laptop14 4 and Restaurant. datasets (Pontiki et al., 2014). For text classification (TC) tasks, we chose the SST2 (Socher et al., 2013) and AGNews (Zhang et al., 2015) datasets. We selected the SNLI (Bow- man al., 2015) and MNLI (Wang et al., 2019) datasets for the natural language inference (NLI) task. We trained our models on the first 1000 sam- ples from the original training, validation and test- ing datasets, respectively.
A.2.2 Experiments based on Different Backbone Models
We have conducted experiments to demonstrate the relationship between the backbone model and performance. Due to resource limitations, we are currently using FlanT5 variants (small, base, and large, Llama is not implemented currently) as backbones implement InstOptima. We have generated box plot visualize the experi- mental results in Fig. 4 The figure illustrates that performance is highly dependent on the scale the backbone instruction-follow model. In other words, because the F limited capability to follow instructions, the accu- racy achieved by an instruction is low and exhibits larger variance compared the larger instruction- follow models. In this context, InstOpt ima plays a crucial role identifying instructions with optimized objectives.
A.1.2 Experimental PLMs
For the LLM to operate instructions, we select the (OaeaPT'(OpenAl,, 2023) with a temperature of 1 and maximum token length of 500.
To obtain the objective value of perfor- mance, performed instruction-based classifica- tion experiments using the FlanT5 small and FlanT5-base models (Chung et al., 2022), as well as ChatGPI, which are the latest and popular PLM/LLM for instruction learning. For the calcu- lation of semantic complexity, we employed the RoBERTa (Liu et al., 2019) model from transform- ers(Wolf et al., 2020).
model has
A.1.3 Hyper-parameter Settings
The generation size and number of generation for NSGA-II is 100 and 10, respectively. In the fine-tuning' of the PLMs (i.e., FlanT5-small and Fl -5-base,, we set the learning rate and batch size to 5e - and 16, respectively. We fine- tune the PLMs for 3 epochs with an L22 regulariza- tion parameter of 0.01.
A.3 The Visualization of Pareto-fronts
In Fig. 5, we show the visualizations of Pareto- front instructions obtained by InstOptima on the Laptop1 4, SST2 and SNLI datasets. Due to resource limitations, we only present the plots on the Laptop14 4, SST2, and SNLI datasets. We plot the first three fronts searched by NSGA- II, and the first three fronts are indicated by red, green, and blue colors, respectively.
A.1.4 Experimental Environment
The experiments are carried out on a computer run- ning the Cent OS operating system, equipped with RTX 3090 GPU and Core i-12900k pro- cessor. We use the PyTorch 2.0.0 library and transformers 4.28.0.
A.4 Multi-objective Optimization Algorithm
InstOptiman a multi-objective instruction op- timization approach that evolves a population of instructions through a series of steps. We present the pseudo-code of InstOpt ima Algorithm I.
"chatGPT-tuzbo-030 version.
*We use the Huggingface Trainer for fine-tuning, and the code is available in the supplementary materials.

---

Table 3: The experimental performance of InstOptima. We show the ACCURACY instead of the performance objective for intuitive evaluation. The symbols / and Y indicate larger better and lower is better, respectively. We repeat each experiment in five rounds and report the average results. The best results are in bold. The ACCURACY is the best accuracy in the Pareto-front, while the LENGTH and PERPLEXITY are correlated with the instruction that achieves the best accuracy.
InstOptama
Raninstruct
MoDEL
DATASET
ACCURACY
LENGTH
PERPLEXITY
ACCURACY
LENGTHY
PERPLENITY
ACCURACY
Plaanhh-mmal
GigaNord
33.7.01 .03
,86.9,913 1915
pour
11
891.61511
30,8
12
Small
Lepp
Iuee
Stal
Ra betreel hartytiias
Acctano (4)
08-
Accenory (k)
02
66
Stail
La777
Tue
Stail
Pla-Te Viraal
Fl Years
Inatptizz
1x0gr
Tlx T5 Veisen
1,339
Dau
Steat
Figure 5: Visualizations the Pareto fronts searched by InstOpt ima on three datasets. The PLM used to evaluate performance is F
Figure 4: Box plot visualizations of the performance based on different backbone models.
Firstly, the algorithm initializes a population of instructions. Then, it iteratively performs the fol- lowing steps for specified number of generations: selecting two instructions from the population, eval- uating their objectives, applying LLM-based in- struction operators to create new instructions, and adding them a temporary population. After each generation, the temporary population is combined with the original population, and selection pro- cess is applied to choose the fittest instructions. Fi- nally, the algorithm returns the evolved population of instructions as the final results.
A.5 Fixed Prompts for Instruction Operators
The prompts in green are the trigger of objective- guided instruction generation.

---

Minimization objectives the Algorithm 1: The pseudo code of Inst ima.
Input: Task dataset D. Number of generations. N, Population size M, Instruction Operators P Output: Evolved population of instructions p
p +- Initialin www.liieeoppaaaaaaoooo inn (M) )
1/ Initialize the populationn
for 1 N do
+ 0;
11 initialize the offspring population
for j + 1 to M do
1 + P
// select parent instruct fon 1/ Select random parent instruction evaluate objectives for parent Evaluate objectives for parent 2 Extract definition and example from parent
4 - random(P);
Fi < svaluatedhjeatives (I,)
F2 <- EvalusteObjectiver (I2):
(di,e) ei) 1
10
(d2: 02) - :
// Extract definition and example from parent 2
O random(P))
select random operater
if O- Pam then
13
Generate mutated definition // Combine mutated definition with example de + Chatterr(Conna((P Fi):
i Concat(d.mm. ei):
if O Pac then
16
dac + Chat GhTonaaa de, d2, // Generate crossoverd definition // Combine cressoverd definition with example
i Concut(d...
if O Pam. then
19
// Generate nutated example i Concat(d), é // Combine original definition with nutated example
cm +- Chat (CT(Cana e1, Fi)):
21
if O = Pee then
22
Generate erossoverd examplc + Concat(d,, &cc) Combine original detinition with crossovera example
Chat CCConaaaCCn ei, e,,
24
Q - @U(
// Add offspring to the population // Combine parent and offepring pepulations Select the best individuals for the
Q' Combineeoouulation (P, Q) Seleectoopllation (Q", M)
generation
27 p : 28 return P
// Set the evolved population as the final population // Return the evoived population
Table 4: The fixed prompts used to implement LL.M-based instructions. "<Input>" indicates the input of the operators. The green keywords are the triggers of objective-guides instruction generation.
OPERATORS
PROMPTS
INPUT
you be profossional prompt engineer. Now working the multi-objective evolutionary prompt optimization, need your help to design optimize the template prompt. Here give you an exumple template prompt, pleuse understand meaning the prompt and modify it. Given the minimization objectives, please be creative and output the paraphrased mutated prompt. Please remove Minimization objectives the output: <Inputs
Pam
(1,F)
want you to he professional prompt engineer. Now am working on the multi-ohjective evolutionary prompt optimization for sentiment analysis, and need your help to design and optimize the template prompt. Here I give crossover them into prompt. Given the minimization objectives, please be creative and output the generated prompt based the exumples. Please output: <Iuput>
Pac
template prompts, pleuse understand the meaning of the two prompts and
(d,.Fi,d..J)
want you be professional prompt engineer. Now am working on the multi-objectivee evolutionary prompt optimization for sentiment analysis, and need your help to design and optimize the template prompt. Here give you two groups of examples for completing the prompt, pleuse generate new examples to substitute the following examples and there are no more than two examples in the new prompt. Given the minimization objectives, please be creative and output the generated example in the same format. Please remove Minimization objectives in the output: <Input>
P
(e,F)
want you be professional prompt engincer. Now am working the multi-objective evolutionary prompt optimization for sentiment anulysis, and need your help design and optimize the template prompt. Here give you groups of examples completing the prompt, please read the examples of the two groups examples and crossoee the examples no more than two examples the new examples, Given the minimization objectives, please be creative and output the crossovered the examples. Please remove Minimization objectives the output: <Input>
P.r
new example group and there are
(e.


Reviews:
[{"review_id": "Yb73JtXHcO", "reviewer": "Reviewer_WrSn", "summary": "", "questions": "Q1: What is the performance of ChatGPT on the five datasets?\nQ2: Why not just use F1 as performance objective in Section 2.2.2?\nQ3: Is the EvaluateObjectives() function in algorithm 1 evaluated on the validation set? More details need to be provided.\nQ4: What are the time costs of the entire algorithm? It seems that multiple evaluations are conducted during generation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1**: This is a timely pilot work so we organized it as a short paper. We will move RQ2 and RQ3 to the Experiments section. Meanwhile, we will include a discussion section on related works. But we may not be able to reformat the current submission into a regular long paper, we are committed to enhancing the overall presentation and readability to the best of our abilities. For example, we will highlight the key contents with colored fonts to assist readers in navigating and focusing on their interested sections.\n\n**R2**: Upon further consideration of your concern, another question to ponder is how to unveil the degree of dependency between the backbone model and performance. We are introducing a new specific section that will present experimental results of other backbone models (e.g., Llama, Vicuna) to answer this question.\n\n**R3**: This is going to be addressed in the revision. We will incorporate performance comparisons towards multiple state-of-the-art instruction-based models and present discussions about these experiments. We provide the experimental results of ChatGPT on two datasets in the following table:\n\n| InstOptima |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.2$_{\\pm 2.2}$ | 512.9$_{\\pm 51.5}$ | 1.08$_{\\pm 0.02}$ |\n| Restaurant14 | 96.3$_{\\pm 1.9}$ | 487.3$_{\\pm 55.9}$ | 1.09$_{\\pm 0.02}$ |\n\n\n\n**Q1**: Our experiments are conducted to prove it is effective for LLMs, including ChatGPT. We provide the performance of ChatGPT in the following tables:\n\n| RanInstruct |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.1$_{\\pm 0.8}$ | 877.6$_{\\pm 51.5}$ | 1.05$_{\\pm 0.03}$ |\n| Restaurant14 | 92.1$_{\\pm 1.3}$ | 421.6$_{\\pm 82.4}$ | 1.10$_{\\pm 0.02}$ |\n\n| NoInstruct |  |\n| --- | --- |\n| Dataset | Accuracy $\\nearrow$ |\n| Laptop14 | 67.8$_{\\pm 5.8}$ |\n| Restaurant14 | 75.2$_{\\pm 6.1}$ |\n\nwhere RanInstruct is ChatGPT based on the randomly generated instructions (i.e., definition and example), and NoInstruct means only examples (no definition) are provided. the experimental results show the effectiveness of InstOptima for ChatGPT. Due to the tight response period, the experiments are evaluated on two datasets, but we will share more experimental results on the rest of the three datasets in the appendix. \n\n**Q2**: When considering the generalizability of InstaOptima in other tasks, employing the combination of multiple metrics is a more scalable approach. To elaborate, when dealing with a composite of diverse metrics spanning various facets (which may occasionally exhibit conflicts), the evolutionary process can be effectively undertaken.\n\n**Q3**: It depends on the existence of a validation set, which means if there is no validation set, such as the Restaurant14 and Laptop14 datasets, the EvaluateObjectives evaluates the test set.\n\n**Q4**: We report the average time costs in a population and the time costs for FlanT5-small and FlanT5-base are shown in the following tables, respectively:\n\n| **Dataset** |**Performance (s)** | **Perplexity (s)**| **Length (s)** |\n|---|---|---|---|\n| Laptop14 | $138.72_{\\pm 27.3}$ | $0.48_{\\pm 0.01}$  |$6.24\\times 10^{-5}$|\n|Restaurant14 | $149.82_{\\pm 34.1}$  | $0.42_{\\pm 0.02}$  |$6.88\\times 10^{-5}$|\n| SST2| $117.76_{\\pm 25.8}$ |$0.38_{\\pm 0.01}$ |$7.55\\times 10^{-5}$|\n| AGNews| $117.16_{\\pm 23.5}$|$0.45_{\\pm 0.02}$|$8.27\\times 10^{-5}$|\n| SNLI | $400.46_{\\pm 55.8}$| $0.39_{\\pm 0.02}$   |$7.84\\times 10^{-5}$|\n| MNLI | $879.17_{\\pm 81.3}$|$0.44_{\\pm 0.02}$  |$8.05\\times 10^{-5}$|\n\n| **Dataset** |**Performance (s)** | **Perplexity (s)**| **Length (s)** |\n|---|---|---|---|\n| Laptop14 |$298.58_{\\pm 55.7}$ | $0.38_{\\pm 0.02}$  |$6.39\\times 10^{-5}$|\n|Restaurant14 | $342.72_{\\pm 49.7}$ |$0.35_{\\pm 0.02}$| $7.81\\times 10^{-5}$ |\n| SST2| $185.60_{\\pm 32.1}$| $0.38_{\\pm 0.02}$  |$6.53\\times 10^{-5}$|\n| AGNews| $202.46_{\\pm 38.1}$| $0.41_{\\pm 0.02}$  |$8.81\\times 10^{-5}$|\n| SNLI | $468.63_{\\pm 48.8}$| $0.32_{\\pm 0.02}$  |$7.56\\times 10^{-5}$|\n| MNLI | $950.98_{\\pm 77.8}$| $0.36_{\\pm 0.01}$  |$7.14\\times 10^{-5}$|\n\nOur experiments are based on an RTX3090 GPU. More details about the environment and implementation can be found in the appendix.\n\nThank you for your insightful review. We hope that your concerns have been addressed. All of the responses will be included in the revision. We are looking forward to answering other questions in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1: The proposed method is novel and makes sense.\n\nA2: The paper is easy to read and comprehend.\n\nA3: The experimental results demonstrate the effectiveness of the proposed method.", "reasons_to_reject": "R1: The organization of the paper needs improvement. For instance, crucial sections like experimental settings and RQ2 and RQ3 in section 3.2 are only presented in the Appendix. Additionally, the paper lacks related work. Therefore, I suggest that the paper should be reformatted to a long paper.\n\nR2: According to Table 1, the performance gap between InstOptima and RanInstruct is significantly influenced by the backbone model used. For instance, the gains are not substantial when FlanT5-base is used. Hence, it is better to employ more powerful backbones, such as llama.\n \nR3: The paper fails to compare InstOptima with other instruction generation methods, like self-instruct.", "questions_to_authors": "Q1: What is the performance of ChatGPT on the five datasets?\nQ2: Why not just use F1 as performance objective in Section 2.2.2?\nQ3: Is the EvaluateObjectives() function in algorithm 1 evaluated on the validation set? More details need to be provided.\nQ4: What are the time costs of the entire algorithm? It seems that multiple evaluations are conducted during generation.", "paper_topic_and_main_contributions": "The authors investigate the problem of generating instructions for in-context learning and define objectives such as performance, length, and perplexity that affect the quality of the instructions. They propose InstOptima, which treats instruction generation as an evolutionary multi-objective optimization problem. InstOptima randomly selects an operator (definition formulation/definition crossover/example mutation/example crossover) to evolve the instruction by feeding the objectives back. The authors report that InstOptima achieves better performance on various datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "535: NoInstructablates --> NoInstruct ablates"}, "extra_text": {}}, {"review_id": "ohiyjLnBEK", "reviewer": "Reviewer_RjTW", "summary": "", "questions": "1. What was the initial motivation for using a genetic algorithm? \n2. How is initialization implemented?\n3. The number of evolutionary iterations in A.2 does not seem to affect perplexity, and the fluctuation of perplexity is large. The paper claims that this can be solved by increasing the population size, number of generations, and scale of training data. Can you provide some quantified results to support this claim?\n4. Can we replace the random operator part with traditional text augmentation methods (such as synonym replacement, back translation, typos, etc.)?\n5. What is the extra search time introduced by InstOptima compared to random instruction?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1**: Due to resource and time limitations, organizing the specific experiments related to adversarial instruction attacks cannot be finished in the rebuttal period. However, We have decided to incorporate these experiments in the appendix to ensure comprehensive coverage. Please kindly allow us some time to sort out the robustness evaluation details.\n\n**R2**: We will incorporate comparisons towards state-of-the-art instruction generation methods. As a complementary experiment, we have reported new results of InstOptima based on the ChatGPT backbone:\n\n| InstOptima |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.2$_{\\pm 2.2}$ | 512.9$_{\\pm 51.5}$ | 1.08$_{\\pm 0.02}$ |\n| Restaurant14 | 96.3$_{\\pm 1.9}$ | 487.3$_{\\pm 55.9}$ | 1.09$_{\\pm 0.02}$ |\n\n| RanInstruct |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.1$_{\\pm 0.8}$ | 877.6$_{\\pm 51.5}$ | 1.05$_{\\pm 0.03}$ |\n| Restaurant14 | 92.1$_{\\pm 1.3}$ | 421.6$_{\\pm 82.4}$ | 1.10$_{\\pm 0.02}$ |\n\n| NoInstruct |  |\n| --- | --- |\n| Dataset | Accuracy $\\nearrow$ |\n| Laptop14 | 67.8$_{\\pm 5.8}$ |\n| Restaurant14 | 75.2$_{\\pm 6.1}$ |\n\n**Q1**: First, it is because the genetic algorithm is an easy-to-implement method for multi-objective instruction optimization. Therefore, the experiments are easy to reproduce. Everyone can change the evolution engine on their demands. Second, the text edition-based operators of genetic algorithms have been proven to be effective for text evolution [1][2]. This inspires our LLM-based operators.\n\n**Q2**: We first define the instruction GA operators based on ChatGPT. We hand-crafted ten seed instructions (We will clarify these seed instructions in the revision.) and used GA operators to manipulate the seed instructions. The new instructions operated by ChatGPT are combined into a population. We will update Figure 1 to indicate the start of initialization.\n\n**Q3**: Figure 2 visualized the trajectories of a single individual with the best accuracy, so it cannot properly visualize the evolution progress. We have fixed this problem by visualizing the average accuracy, length and perplexity within the population and prepared the fixed figure on the Laptop14 dataset. In the fixed figure, we do not need to improve the population size, number of generations, and scale of training data to show clear convergence trajectories. Please refer to https://anonymous.4open.science/status/InstOptima-7C28 to find the population’s convergence trajectories. In the new figure, please note that the instruction lengths are growing gradually because our seed instructions in initialization are too short compared to Pareto front instructions.\n\n**Q4**: No, this will lead to poor diversity as well as poor accuracy of instructions in our proof-of-concept experiments.\n\n**Q5**: We report the average time costs in a population. Please refer to Q4 in the rebuttal of Reviewer WrSn. The search time of InstOptima is approximately calculated by multiplying # of generations, # of individuals in a population, and the evaluation budget for each individual, while the random instruction requires no extra search time budget. \n\nFinally, thank you for your careful review. We hope that your concerns have been addressed. All of the responses will be included in the revision. We are looking forward to answering other questions in the discussion period.\n\n[1] Alzantot, Moustafa, et al. \"Generating natural language adversarial examples.\" *arXiv preprint arXiv:1804.07998* (2018).\n\n[2] Zang, Yuan, et al. \"Word-level textual adversarial attacking as combinatorial optimization.\" *arXiv preprint arXiv:1910.12196* (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Compared to the random instruction generation method of RanInstruct, the comprehensive multi-objective optimization considering instruction length and perplexity is more reasonable.\n2. Good performance, which provides significant improvement over the random RanInstruct.\n", "reasons_to_reject": "While the contribution claims that the proposed method enhances robustness against adversarial instruction attacks, the current experiments do not provide sufficient evidence to substantiate this claim. Although relevant findings from prior studies are cited, it is important to experimentally verify their applicability to the specific scenario addressed in this paper. Additional rigorous experiments are needed to demonstrate the efficacy of the proposed instructions in defending against such attacks.\n\nThe comparison of algorithms conducted by the paper is somewhat limited, considering the constraints of this short paper. It would be beneficial to include a broader range of algorithms for comparison to ensure a more comprehensive evaluation and a clearer understanding of the proposed method's performance. Some quantified results would facilitate a more robust analysis and reinforce the significance of the work. ", "questions_to_authors": "1. What was the initial motivation for using a genetic algorithm? \n2. How is initialization implemented?\n3. The number of evolutionary iterations in A.2 does not seem to affect perplexity, and the fluctuation of perplexity is large. The paper claims that this can be solved by increasing the population size, number of generations, and scale of training data. Can you provide some quantified results to support this claim?\n4. Can we replace the random operator part with traditional text augmentation methods (such as synonym replacement, back translation, typos, etc.)?\n5. What is the extra search time introduced by InstOptima compared to random instruction?\n", "paper_topic_and_main_contributions": "This paper presents a new method called InstOptima, which treats instruction generation as a multi-objective optimization problem. This method uses a large language model (LLM) to simulate the generation of instruction operations using a genetic algorithm, including mutation and crossover. Meanwhile, an objective-guided mechanism is introduced to help the LLM generate high-quality instructions. InstOptima divides the direction of instruction search into multiple objectives to achieve fine control over instruction quality. Using multi-objective optimization algorithms helps to automatically search for a set of high-quality instructions and improve the performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5UmDZoR7om", "reviewer": "Reviewer_YhoU", "summary": "", "questions": "1. What is \"Crowdness Evaluation\" in details?\n2. What is the detail of RanInstruct? What is the prompt used here?\n3. Why we want \"Lower perplexity\" of instructs? The author claimed \"indicates that instructions are more easily understood by language models.\" But I am not sure what does this imply, a better generalization?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1**: We are checking the soundness of this paper, and will actively work on refining the clarity and self-containment of the paper. More specifically, we will rearrange the contents in the main sections and appendix section, and we will add details and references to the method section to make it self-contained.\n\n**R2**: We provide the experimental results (on two datasets currently) of InstOptima using ChatGPT as the backbone model (no fine-tuning) in the following tables:\n\n| InstOptima (ChatGPT) |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.2$_{\\pm 2.2}$ | 512.9$_{\\pm 51.5}$ | 1.08$_{\\pm 0.02}$ |\n| Restaurant14 | 96.3$_{\\pm 1.9}$ | 487.3$_{\\pm 55.9}$ | 1.09$_{\\pm 0.02}$ |\n\n| RanInstruct |  |  |  |\n| --- | --- | --- | --- |\n| Dataset | Accuracy $\\nearrow$ | Length $\\searrow$ | Perplexity $\\searrow$ |\n| Laptop14 | 83.1$_{\\pm 0.8}$ | 877.6$_{\\pm 51.5}$ | 1.05$_{\\pm 0.03}$ |\n| Restaurant14 | 92.1$_{\\pm 1.3}$ | 421.6$_{\\pm 82.4}$ | 1.10$_{\\pm 0.02}$ |\n\n| NoInstruct |  |\n| --- | --- |\n| Dataset | Accuracy $\\nearrow$ |\n| Laptop14 | 67.8$_{\\pm 5.8}$ |\n| Restaurant14 | 75.2$_{\\pm 6.1}$ |\n\nwhere the experimental results show the effectiveness of InstOptima for ChatGPT. Due to the tight response period, the experiments are evaluated on two datasets, but we will share more experimental results on the rest of the three datasets in the appendix. \n\n**R3**:  We are proofreading the manuscript to improve its writing, and we will rework Figure 1 to show evolution details.\n\n**Q1**: Crowding distance is a concept used in multi-objective optimization to assess the diversity and distribution of solutions within a Pareto front. The calculation aims to provide a measure of how crowded the solutions are in the objective space, which helps in maintaining a diverse set of solutions. In other words, the crowding distance calculation measures how much an individual is \"crowded\" by comparing the distances between the neighbors along each objective dimension. Larger differences in objective values contribute to larger crowding distances, indicating that an individual is more spaced out from its neighbors. This calculation helps in selecting a diverse set of solutions for further stages of the optimization process. Please refer to the supplementary materials for the code implementation, i.e., lines 272 - 292 in nsga2.py.\n\n**Q2**: We employ ChatGPT to generate a population of instructions using hand-crafted seed instructions. This process is similar to InstOptima, yet without using an evolution process. The instructions and performance results obtained through InstOptima have already been made available in the supplementary materials. Additionally, we will include the random instructions and their corresponding results in the revised version.\n\n**Q3**: Yes, it is useful for generalization. The reason why we need low perplexity is because:\n    \n    (1) A higher perplexity indicates that the words in the text are not well-organized. For example, text edits based on word deletions and insertions can lead to grammatical or syntactical errors, resulting in higher perplexity compared to natural texts. \n    \n    (2) The perplexity issue is particularly significant in the language generation of small language models. Therefore, generating instructions that are easy to read and that reduce the risk of misinterpretation is important for both language models and instruction engineers.\n\nThank you for your questions. We hope that your concerns have been addressed. All of the responses will be included in the revision. We are looking forward to answering other questions in the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A clear method and framework for improving instructions with the help of LLMs.\n2. Optimizing multiple pre-defined dimensions is a promising and reasonable idea.", "reasons_to_reject": "1. Although this is a short paper, there are still many details that need to be clarified to make this paper self-contained.\n2. Experimented classification model is T5 instead of an LLM, does this mean this method only applied to medium size models? I think this is acceptable but the authors should explain this.\n3. The writing and figures can be improved", "questions_to_authors": "1. What is \"Crowdness Evaluation\" in details?\n2. What is the detail of RanInstruct? What is the prompt used here?\n3. Why we want \"Lower perplexity\" of instructs? The author claimed \"indicates that instructions are more easily understood by language models.\" But I am not sure what does this imply, a better generalization?", "paper_topic_and_main_contributions": "The paper proposed InstOptima, a multi-objective framework for achieving a varied set of improved instructions. It relies on LLMs like chatgpt to optimize the instructions (experimented with text classification tasks) on several pre-defined dimensions, including Performance, Length and Perplexity. It involves an iterative process. Experiments are done on six datasets for three classification tasks (by finetuning FlanT5-small and FlanT5-base with instructions as classification models).  In general, InstOptima achieves superior objectives in most cases\n\nOverall I think this paper can have some value in LLM era, but there are some concerns.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Figure 1 can add some elements to indicate the starting point of the iteration\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Caption of Table 1 is strange (the \"We\")"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method called InstOptima, which treats instruction generation as a multi-objective optimization problem. InstOptima randomly selects an operator (definition formulation/definition crossover/example mutation/example crossover) to evolve the instruction by feeding the objectives back. The reviewers did raise a few concerns and during the rebuttal period, the authors provided further explanations and experimental results, which solved the concerns to some extent. A critical though non-technical issue is that once accepted, the authors need to carefully reorganize the content into 5 pages so as to come up with a self-contained version.
Decision: Accept-Findings