Automatic Task-aware Instruction Optimizer for Black-box LLMs

Authors: Yunzhe Qi, Jinjin Tian, Ruirui Li, Tianci Liu, Tianxin Wei, Hui Liu, Xianfeng Tang, Monica Xiao Cheng, Jingrui He
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models; Instruction Optimization

Abstract:
Large Language Models (LLMs) have demonstrated superior capabilities in terms of solving various real-world tasks. However, their performance and generated content quality heavily depend on task-relevant instructions, which makes instruction optimization a challenging but critical direction to explore. In particular, as practitioners generally cannot access black-box (or API) LLMs' internal parameters and gradient information, it consequently makes instruction optimization for black-box LLMs especially non-trivial. Existing methods for optimizing black-box LLM instructions mainly focus on in-context learning using manually designed or heuristic disciplines, which can be insufficient due to the extreme complexity of modern black-box LLMs that can contain hundreds of billions of parameters. 
To address these challenges, we propose a novel automatic instruction optimization framework named Automatic Instruction Optimizer (AIO). AIO is designed to perceive target task information and adaptively adjust its task-aware instructing strategy for a task-solver black-box LLM. 
By leveraging a white-box LLM with parameter fine-tuning for enhanced representation power, AIO can automatically update its instructing strategy based on the feedback from task-solver black-box LLM. 
To achieve this goal, AIO adopts a novel LLM parameter fine-tuning process powered by zeroth-order gradient approximation and Contextual Bandit techniques, which can effectively and efficiently help address the challenge of inaccessible black-box LLM internal parameters and gradients, as well as help alleviate expensive API cost concerns by flexibly reusing collected black-box LLM feedback.
Extensive empirical evaluations are presented to demonstrate properties of our proposed AIO, and its effectiveness in comparison with strong baselines.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729852162628

thread_type:
official_review

<thread_object>
creator: reviewer_f3s6

<content>
summary:
This paper mainly focuses on optimizing task-aware instructions in the absence of task-solver LLMs’ internal parameters and gradient information. Finding that the current instruction optimization methods based on in-context learning are insufficient to match the extreme complexity of modern black-box LLMs, this paper proposes a novel framework, called Automatic Instruction Optimizer (AIO), which fine-tunes a white-box LLM and enables it to adaptively adjust its instructing strategy for a particular task-solver black-box LLM. To address the optimization challenges, AIO adopts zeroth-order gradient approximation and Contextual Bandit techniques, eliminating the need for internal parameters and gradient information.

strengths:
- This paper demonstrates that fine-tuning a white-box LLM into the instruction optimizer is able to have better instruction generation performance due to a stronger representation power.
- This paper addresses the optimization challenges using zeroth-order gradient approximation and manages to reduce the API cost by reusing historical records.
- It conducts comprehensive experiments to verify the effectiveness of the proposed method, including different combinations of white-box and black-box LLMs, instruction optimization trajectory results, comparison to CoT.

weaknesses:
- A minor point: study the transferability of fine-tuned instruction optimizers across different black-box task-solver LLMs.
- The right-hand side of equation 1 misses a right parenthesis.
- Figure 3 lacks proper legends for different instruction optimization methods.

questions:
Kindly refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729930745963

thread_type:
official_review

<thread_object>
creator: reviewer_r8cj

<content>
summary:
This paper proposes a task-aware instruction optimization for black-box LLMs. This method leverages both white LLM and black LLM. AIO adopts the parameter fine-tuning process with gradient approximation and contextual bandit techniques.

strengths:
The paper proposes the AIO for the instruction optimization. The AIO provides some theoretical analysis and demonstrations.

weaknesses:
1.	The evaluation benchmarks in this paper are not very common. The proposed improvements should be demonstrated in the general benchmarks, including the instructions for the following benchmarks.
2.	The proposed method is unlike other task-aware instruction optimization methods. As there are many methods and literatures in this field, it would be beneficial to compare the proposed methods with other baselines.
3.	Although the white box LLM and black box LLM show improvements, it is obvious that this combination will perform better than the white box LLM and black box LLM individually. So the more analyses on these component designs are needed.

questions:
Please see the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602390445

thread_type:
official_review

<thread_object>
creator: reviewer_ovqk

<content>
summary:
The paper proposes an instruction optimization framework called Automatic Instruction Optimizer (AIO) for black-box Large Language Models (LLMs). LLMs often rely on precise, task-relevant instructions to achieve high-quality outputs, but manual optimization is challenging, especially for black-box models where internal parameters are inaccessible. AIO addresses this by employing a white-box LLM, which can be fine-tuned to generate optimized instructions based on task-specific information and feedback from the black-box LLM. The optimization process involves a novel zeroth-order gradient approximation method, aided by Thompson Sampling, which bypasses the need for direct access to gradients, enabling effective fine-tuning. Extensive experiments demonstrate that AIO not only improves task performance but also reduces API token costs. Compared to existing methods, AIO achieves better performance on various tasks and maintains efficiency through the adaptive reuse of feedback. This framework represents a significant advancement in automating instruction generation, enhancing the usability and adaptability of black-box LLMs across diverse tasks​

strengths:
+ The paper introduces a unique framework, Automatic Instruction Optimizer (AIO), that optimizes instructions for black-box LLMs without requiring access to internal parameters, which is a critical advantage in real-world applications where only API-based models are available.
+ AIO’s use of zeroth-order gradient approximation with Thompson Sampling is innovative, as it enables effective instruction fine-tuning without direct gradient access, thereby overcoming a major limitation in black-box model optimization.
+ The framework significantly reduces API token consumption by reusing black-box LLM feedback adaptively, making it a cost-effective solution compared to existing in-context learning or manual tuning methods.
+ The paper also explores lightweight PEFT variants (e.g., LoRA and Linear Probing) that maintain performance with fewer parameters, showcasing AIO’s flexibility for different fine-tuning needs.

weaknesses:
+ The experiments mainly use llama+API pairing, which may not fully represent the framework’s effectiveness across a broader range of LLMs. I suggest the authors do more combinations of white-box LLMs, e.g., Mistral, for making the study of the method more comprehensive.

questions:
The paper focuses on optimizing instructions based on task-specific feedback from a black-box LLM; how well does the optimized instruction generalize to related tasks or domains, especially in cases where task-specific data is sparse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731081982755

thread_type:
official_review

<thread_object>
creator: reviewer_sk6i

<content>
summary:
This work proposed to fine-tune a white-box LLM to generate optimal instruction for the black-box LLM for downstream tasks. Under the assumption of linear optimization landscape of $\phi$, a zero-order gradient approximation approach was leveraged to update the parameters, in conjunction with a decision-making process that employed contextual bandits to select the most informative directions of gradient perturbation. Experiments of $15$ instruction induction and reasoning tasks showed the superior performance of the proposed algorithm.

strengths:
1. The idea of automatic instruction optimizer is important to domain experts who do not have enough experience in prompting engineering. An added benefit would be increased transparency and trustworthiness between human and LLM agents.

2. The zero-order optimization framework with gradient approximation and decomposition is reasonable when involving many parameters.

weaknesses:
1. The structure can be significantly improved for better clarification and illustration. There are many re-defined notations and formulations, such as $\phi$, $Theta_t$ and $\mathbf{z} \sim \mathcal{N}(0, I)$. Showing the trivial calculation process makes it difficult to grasp the core structure of the algorithm. It is suggested to first give the overall framework of the algorithm and then briefly present the calculation details.

2. The motivation of introducing Thompson sampling, in consideration of the zero-order gradient approximation, is confused. In addition, additional hyperparameters $\beta$ and $B$ were introduced without enough sensitivity analysis in experiments.

3. There is missing legend in Figure 3.

questions:
1. Is the task context known by the white-box LLM? How can domain expert be involved in the design of instruction?

2. Does the Thompson sampling introduces bias of gradient estimation? What is the influence of hyperparameters $\beta$ and $B$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732552340058

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking Forward to Your Valuable Feedback on Rebuttal and Manuscript Updates

comment:
Dear Reviewers,

We sincerely thank you for your time and dedication to the review process. Your insightful feedback, comments, and questions have been invaluable in improving the quality of our work. 
As the discussion period will **conclude in two days**, we would greatly appreciate it if you could review our responses and let us know whether they sufficiently address your concerns. Any further comments or suggestions you may have would be deeply appreciated.


**Changes to the Updated Manuscript and Summary of Our Added Experiments:**


1. Improvements made to the manuscript based on reviewers' comments:

- (1) *[Figure 1]* **Enhanced pipeline illustration**: Enlarged fonts, added references to specific sections, etc.

- (2) *[Beginning of Section 4]* **Added introductory paragraph at the beginning of Section 4**: Briefly introduces the main framework pipeline and components to help readers understand the core structure of the proposed framework.

- (3) *[Subsection 4.1]* **Improved narrative of the calculation process**: Provides clearer presentation for a broader audience, especially those unfamiliar with zeroth-order gradient approximation techniques or Contextual Bandits. Detailed calculation steps are included later in the pseudo-code (Algorithm 1).

- (4) *[Equation 1 and Figure 3]* **Corrected missing legend and typographical errors.**

- (5) *[Appendix B.7 - B.11]* **Incorporated experimental results from the rebuttal phase.**

- (6) **Included discussions from the rebuttal phase**:

  1. *[Appendix E]* **Future Directions**:
     - Task information for instruction generation and optimization, including involving human experts. [Reviewer sk6i]
     - Generalizing optimized instructions to related tasks or domains. [Reviewer oVqK]

  2. *[Appendix D]* **Discussions and Clarifications**:
     - Reasoning behind our TS-aided ZO approach, and discussions on the bias in TS direction selection strategy. [Reviewer sk6i]
     - Clarifications on our choice of datasets and baselines, which are similar to closely related works. [Reviewer r8CJ]
     - Clarification on model usage: The black-box LLM is treated as part of the learning objective rather than the learning model, consistent with closely related works. [Reviewer r8CJ]


2. Additional experiments (already included in the updated manuscript):

- [Reviewer sk6i] Enhanced AIO performance by incorporating **task descriptions** as prior knowledge into the optimization process.

- [Reviewer sk6i] **Parameter study** on the threshold $\beta$ and the number of chosen arms $B$.

- [Reviewer oVqK] Additional empirical results on **other families of white-box LLMs** (Mistral and Qwen).

- [Reviewer r8CJ] Comparison with an **additional baseline "EvoPrompt"**, which leverages "initial instructions" as task prior information for instruction optimization.

- [Reviewer f3S6] Experimental results on **transferring** fine-tuned instruction optimizers across **different black-box LLMs** (Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4o).



**The current rebuttal discussions and additional empirical results have been incorporated into the manuscript. We kindly invite you to review our updated manuscript for your reference.**




Sincerely,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732555310620

thread_type:
official_comment

<thread_object>
creator: area_chair_fjex

<content>
title:
Please actively participate in the reviewer-author discussion

comment:
Dear Reviewers,

Thank you for your efforts and contribution to ICLR! The authors have posted their responses to your original comments. Only less than two days are left for the reviewer-author discussion. Given the current borderline ratings, your help and prompt responses are important. Please actively check the authors' responses and actively participate in the discussion.

Thanks!

Best regards, 

Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary: 
The paper introduces an Automatic Instruction Optimizer (AIO) framework designed for optimizing task-relevant instructions for black-box Large Language Models (LLMs), where internal parameters and gradients are inaccessible. AIO finetunes a white-box LLM (via parameter efficient fine-tuning) using Contextual Bandit (TS) based zeroth-order gradient approximation to adaptively adjust instructing strategies, based on feedback from the black-box LLM. Through experiments on instruction induction and reasoning tasks. AIO showcases effectiveness in automating instruction generation, enhancing the usability and adaptability of black-box LLMs across various tasks, leading to improved task performance and reduced API token costs. AIO follows the existing framework of instruction optimization but adopts a different zeroth order optimization, which offers a promising solution for optimizing task-aware instructions in the absence of detailed task-solver LLM information.

## Strengths:
1. AIO's use of zeroth-order gradient approximation with Thompson Sampling is innovative, enabling effective instruction fine-tuning without direct gradient access.
1. The paper explores lightweight PEFT variants, showcasing AIO's flexibility for different fine-tuning needs and maintaining performance with fewer parameters.
1. Comprehensive experiments validate the effectiveness of the proposed method, including diverse white-box and black-box LLM combinations, instruction optimization trajectory results, and comparisons to existing approaches like CoT.

## Weaknesses:
1. This paper's contribution is incremental to some extent: it follows the existing framework of white-box LLM + API (black-box) LLM framework that optimizes a few parameters of the white-box LLM  zeroth order optimization. The main difference is in the zeroth-order optimization approach, which is Thompson sampling (parametric prior and posterior) in this paper but Bayesian optimization (nonparametric prior and posterior) in previous works such as InstructZero. However, theoretical analysis and more motivations are needed to justify the TS approach is better than previous strategies.  
1. The proposed approach mainly focuses on optimizing instructions for a specified task, while a novel extension is to generalize the instruction optimization for other tasks. While the authors proposed valuable ideas in the discussion, extensive study and experiments of them can greatly strengthen the novelty and contribution of this paper.
1. Different combinations of white-box LLM + black-box LLM need to be examined. In addition, fixing the white-box LLM but changing the black-box LLM can reveal the generalizability/transferability of the instructions while fixing the black-box LLM but changing the white-box LLM shed insights on which white-box LLMs are better instruction generators. These results should be compared with the zero/few-shot performance on the individual white-box LLM or the black-box LLM only.  
1. The original paper lacks several ablation studies on the hyperparameters of TS, the number of exemplars, whether using the task information, etc. The authors' responses provide these results in some experiments. The authors are encouraged to provide complete results for these experiments in the next version.
1. Reviewers also raised concerns regarding the notations, equations, and figures, whose presentations can be improved. 

## Decision:
The authors provided detailed clarifications and additional experimental results in the rebuttal, as requested by the reviewers. Reviewers did not participate in the author-reviewer discussion but one reviewer participated in the AC-reviewer discussion. Given the borderline ratings and the mixed comments, the meta-reviewer carefully read the paper and all the authors' responses. The authors put a great amount of effort into addressing the reviewers' concerns, including the additional experiments and revising the narratives. They also provided a nice summary of the updates for review convenience. However, the incremental contribution of the problem setup and the instruction optimization framework is still one of the key factors considered in the decision-making. Reviewers expect to obtain more novel insights in addition to empirical improvements in the previously studied datasets. Moreover, the rebuttal and discussion indicate that a great amount of additional improvement needs to be made to the current draft. The meta-reviewer believes another round of revision is necessary for this work. The authors are encouraged to improve the draft as suggested by the reviewers and meta-reviewer.