# Bayesian Optimization for Instruction Generation
## 2 Related Works
### InstructZero  
A more sophisticated approach is proposed in [14], named InstructZero. Instead of directly optimizing the prompt, which is an instruction to the LLM, InstructZero uses BO to search for the optimal prompt in a low-dimensional space associated with the embedding of an open-source LLM, which generates a human-readable and task-relevant instruction given a few examples of the target task. The instruction is then submitted to the black-box LLM. The entire pipeline is reported in Figure 3 and detailed in Figure 4.  
InstructZero uses an open-source LLM to convert a soft-prompt (**p**) into an instruction that is not only task-relevant but also human-readable. The black-box LLM uses this instruction to perform zero-shot prediction, while BO estimates the relation linking every soft prompt to its associated score.  
Random projection is largely used because its distance-preserving property it suitable for construct kernels. This has the important implication that the behavior of BO is consistent across the original space and the low-dimensional space. This property is particularly important for in-context learning, in that low-dimensional soft prompts can produce task-relevant instructions. The similarity between two prompts is measured in terms of correlation, usually by a Matern or a square exponential kernel. InstructZero proposes a kernel function that is task-specific:  
\[s\big{(}v_{i},v_{j}\big{)}=E_{X\sim D_{i}}\big{[}sim\big{(}f([v_{i};X]),f\big{(} \big{[}v_{j};X]\big{)}\big{)}\big{]} \tag{5}\]  
where \(sim(.,.)\) is the similarity of the predictions for the tasks (e.g., exact match, F1, or BLEU score). Then, the two kernels are combined in an instruction-coupled kernel, which, applied to the soft prompts, recovers the instruction matrix. Then, the instruction-coupled  
Figure 4: Detailed pipeline of the Bayesian optimization process in InstructZero (source: [14]). The flow chart depicts the progression from soft prompts and instructions through open-source and black-box LLMs.  
Figure 3: Schematic representation of the iterative Bayesian optimization loop for prompt optimization in InstructZero (source: [14]).  
kernel drives the combination of the two kernels and aligns BO in the latent space. The structure of the algorithm is captured in the following pseudo code (Algorithm 1).  
```
Input:  Examples \(\left(\overline{\mathbf{X}},\overline{\mathbf{Y}}\right)\) and a validation set \(\left(\mathbf{X},\mathbf{Y}\right)\), instruction generator LLM \(g(.)\), solver LLM \(f(.)\), maximal steps \(T\), random projection matrix \(\mathbf{A}\ \in\ \mathbb{R}^{d\times q}\). Initialize: \(\mathbf{P}\sim Uniform\Big{(}\mathbb{R}^{d}\Big{)}\); \(m\gets 1\); \(\mathbf{V}\leftarrow\varnothing\); \(\mathbf{H}\leftarrow\varnothing\). while \(m\leq T\) do:  generate instruction \(\mathbf{v}_{m}\gets g\big{(}\mathbf{A}\mathbf{p}_{m};\left(\overline{ \mathbf{X}},\overline{\mathbf{Y}}\right)\big{)}\) through the open-source LLM \(g(.)\)  evaluate zero-shot score \(h_{m}\gets h(f(\mathbf{v}_{m},\mathbf{X}),\mathbf{Y})\) on the black-box LLM \(f(.)\) \(\mathbf{P}\leftarrow\mathbf{P}\cup\{\mathbf{p}_{m}\}\) \(\mathbf{V}\leftarrow\mathbf{V}\cup\{\mathbf{v}_{m}\}\) \(\mathbf{H}\leftarrow\mathbf{H}\cup\{h_{m}\}\)  update the instruction-coupled kernel matrix for P  update the posterior mean and variance function of BO  find the next prompt \(\mathbf{p}_{m+1}\) maximizing the acquisition function (i.e., Expected Improvement in InstructZero) \(m\gets m+1\) End Output:  the best instruction \(\mathbf{v}_{i^{*}}\) so far, where \(i^{*}\in\underset{i\in[m]}{argmax}h_{i}\)
```  
**Algorithm 1**InstructZero