# Bayesian Optimization for Instruction Generation
## 4 Computational Results
### Selected Tasks  
The tasks were selected from [14]; due to budgetary constraints, we selected a subset of 6 among the original 24 tasks to reduce the required testing and implementation time. The selected tasks were chosen to be representative of different levels of difficulty (based on the scores reported for APE, Uniform, and InstructZero).  
**Easy tasks:**  
_Letter list_: Provide the input string, separating each character with a space.  
_Word sorting_: With a list as input, the task is to produce an output that is the list in alphabetical order.  
**Challenging tasks:**  
_Taxonomy_: Create a program that generates a list of animals based on the provided input.  
_Synonyms_: Provide the synonyms of the input word.  
_Second word letter_: Taking a string as input, return the second character.  
_Common_: Extract the relationship between the input words.  
Table 1 summarizes the results obtained by BOInG on the selected tasks and compares them against three other methods. BOInG's test score is in line with other BO-based approaches (i.e., Insinct and InstructZero) and outperforms them on two tasks.  
An important consideration is the significantly different costs entailed by the three methods. BOInG requires the invocation of \(\mathcal{M}_{g}\) via an API, implying a minimal additional cost of approximately USD 0.01 per 20\({}^{\prime}\)270 tokens (using GPT-3.5-turbo-0125, ANI Technologies Pvt.Ltd., Bengaluru, India). This cost is significantly lower than that required to run white-box LLMs. For instance, the GPU time required for InstructZero (using an NVIDIA A6000, NVIDIA, Santa Clara, CA, USA) would cost approximately USD 0.01 for just 25 s, during which only a few local LLM calls could be executed.  
Finally, we provide some examples of generated instructions and associated hard prompts over the BO process. In Table 2, the \"hard prompt\" column displays the hard prompt prepended to the input given to the instruction generator LLM (\(\mathcal{M}_{g}\)), consisting of seemingly random character strings that represent the algorithm's exploration in the prompt space. The Instruction column shows the human-readable, task-relevant instructions generated by \(\mathcal{M}_{g}\) based on these hard prompts and task examples. The \"score (train)\" column indicates the performance of each instruction, ranging from 0 to 1, with higher scores denoting better performance. The table captures the optimization process across six rows, including initialization and subsequent iterations. Initially, two different hard prompts yield similar instructions with identical scores of 0.40. The third iteration shows a marginal improvement, achieving a score of 0.45. The fourth row demonstrates the best performance, reaching a score of 0.55 with the instruction, \"Provide synonymous terms for the given words\". This table showcases the algorithm's progression in identifying increasingly effective instructions for the synonym task.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Task** & **Difficulty** & **BOInG** & **APE \({}^{**}\)** & **Insinct \({}^{**}\)** & **InstructZero \({}^{*}\)** \\ \hline Synonyms & Challenging & 0.30 & 0.36 & 0.30 & 0.38 \\ Common & Challenging & 0.17 & 0.07 & 0.21 & 0.15 \\ Letter List & Easy & 1.00 & 1.00 & 1.00 & 1.00 \\ Taxonomy & Challenging & 0.82 & 0.35 & 0.85 & 0.82 \\ Word Sorting & Easy & 0.65 & 0.33 & 0.51 & 0.64 \\ Second Word Letter & Challenging & 0.93 & 0.75 & 0.10 & 0.62 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Test scores comparing BOInG against APE, Insinct, and InstructZero (\({}^{*}\) for APE and Insinct results from [15]; \({}^{**}\) for InstructZero results from [14]).