Task Facet Learning: A Structured Approach to Prompt Optimization

Authors: Gurusha Juneja, Gautam Jajoo, Hua Li, Jian Jiao, Nagarajan Natarajan, Amit Sharma
Primary area: foundation or frontier models, including LLMs
Keywords: prompt optimization, large language model

Abstract:
Given a task in the form of a basic description and its training examples, prompt optimization is the problem of synthesizing the given information into a text prompt for a large language model. Humans solve this problem by also considering the different facets that define a task (e.g., counter-examples, explanations, analogies) and including them in the prompt. However, it is unclear whether existing algorithmic approaches, based on iteratively editing a given prompt or automatically selecting a few in-context examples, can cover the multiple facets required to solve a complex task.  In this work, we view prompt optimization as that of learning multiple facets of a task from a set of training examples. We exploit structure in the prompt optimization problem and break down a prompt into loosely coupled semantic sections. The proposed algorithm, UniPrompt, (1) clusters the input space and uses clustered batches so that each batch likely corresponds to a different facet of the task, and (2) utilizes a feedback mechanism to propose adding, editing or deleting a section, which in turn is aggregated over a batch to capture generalizable facets. Empirical evaluation on multiple datasets and a real-world task shows that prompts generated using UniPrompt obtain higher accuracy than human-tuned prompts and those from state-of-the-art methods. In particular, our algorithm can generate long, complex prompts that existing methods are unable to generate.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729461261900

thread_type:
official_review

<thread_object>
creator: reviewer_veyq

<content>
summary:
This paper presents an improvement to current prompt optimization approach. It does so by clustering the input space and enabling the optimization algorithm to learn multiple facets of a task from a set of batched clusters. The paper discusses the empirical observation that inspires the improvement and conducts experiments to demonstrate its efficacy. Results suggest that the framework achieve better results than existent approach.

strengths:
1. The paper is well-structured and the story is well-told. It starts with essential observations that motivate the improvements so that the approach bears heuristic justification. 

2. Experiments are comprehensive, including several baselines, and benchmarks of different types. Experiment results also provide strong support to the thesis of the paper.

weaknesses:
My concern on the improvement is its generalization ability. The experiments cover hate speech detection, math, QA but there are many other task types not covered in this dataset. For example, OPRO and [Li et al 2024](https://arxiv.org/abs/2409.15199) ran experiments on BBH which contains a much more diverse task portfolio than presented in the paper. Since the proposed improvement centers around clustering the training examples, whether the samples can be cluster into meaningful subspace has a direct impact on optimization performance. But we don't see sufficient evidence to demonstrate the approach works on a divers task portfolio. 

Besides, the prompt optimization is very sensitive to the initial prompt and iterations (also discussed in Sec 2.1), but the reported metrics don't take the performance variation into consideration. I have concern on the robustness of the results.

questions:
1. If I understand correctly, the wrongfully predicted training samples are clustered only once at onset, how does the algorithm handle the wrong samples after the optimization starts? Does it use them to generate feedback?

2. Did you try using other model families? How does the performance look like when the expert model is not as powerful as GPTs? 

3. The clustering is conducted through LLM, but this could be an issue when the number of examples increases. Did you try other clustering approach? which might make it feasible to dynamically cluster the examples along the iterations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729987196318

thread_type:
official_review

<thread_object>
creator: reviewer_ynwp

<content>
summary:
This paper introduced a method for prompt optimization called UNIPROMPT. The prompt optimizaiton is deemed as the task of learning multiple facets of a task. UNIPROMPT exploit the structure in the prompt optimization and use a clustering approach to group examples with similar task facets. Experimental results on multiple datasets demonstrate that prompts generated using UNIPROMPT achieve higher accuracy than human-tuned prompts and state-of-the-art methods.

strengths:
(1) The paper introduces a new method for prompt optimization and consider multiple facets of a task and leveraging the structure in the prompt.
(2) The proposed clustering approach and two-tier feedback mechanism provide a systematic way to generate prompt edits that capture generalizable concepts.
(3) The experimental results on several datasets shows that UNIPROMPT outperforms human-tuned prompts and SOTA methods.

weaknesses:
(1) I am skeptical about the current importance of prompt optimization. Prompt optimization is more of a need in an era when large models are not powerful (for example last year). Nowadays, the instruction model follows human instructions very well, and many of them are 0-shot, see llama-3.1 model card. I would like to hear the author's opinion on this.

(2) For the results in table 1, it is recommended to add an average column to make the presentation clearer.

(3) I found that on the ARC task, compared with the llama prompt, the optimized prompt led to a decrease in performance. Is there any explanation?

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730441352973

thread_type:
official_review

<thread_object>
creator: reviewer_sd4j

<content>
summary:
The paper introduces UNIPROMPT, a method that uses task facet learning to enhance prompt optimization by breaking down prompts into distinct semantic sections, targeting diverse facets such as counter-examples and explanations.

strengths:
- UNIPROMPT clusters examples to capture task-specific facets effectively, with a two-tier feedback mechanism ensuring that prompt modifications are generalizable and not overly specific to individual examples.
- The authors provides strong empirical results, with UNIPROMPT outperforming state-of-the-art methods on multiple datasets, notably achieving significant improvements on hate speech classification.
- By focusing on automatic prompt generation without relying heavily on human-engineered prompts, this approach may reduce the manual overhead of prompt tuning.
- Extensive ablations validate the contribution of each component (clustering, feedback structure) to performance.

weaknesses:
- All experiments are conducted on powerful proprietary LLMs, I wonder whether the proposed method remains effective for opensource LLMs such as LLaMA or Qwen.
- It seems that the proposed method underperforms DSPy on MedQA, while surpassing it on other benchmarks, what might be the reason?
- How to determine the number of clusters when constructing the minibatches? How does this affect the method's effectiveness?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448446716

thread_type:
official_review

<thread_object>
creator: reviewer_gcge

<content>
summary:
This paper proposes a prompt optimization method $U_{NI}P_{ROMPT}$ to generate complex prompts for covering multiple facets of a task and improving overall accuracy. Its effectiveness is demonstrated through relevant experiments.

strengths:
The strategy for creating mini-batches to generate edits and then aggregating them at the batch level to yield the final edit in a feedback mechanism is interesting. 

Experimental results achieves SOTA.

weaknesses:
1.The paper contains some detailed issues, e.g., $U_{ni}P_{rompt}$ mentioned in  line 100-101; an extra space before "So" mentioned in line 197-198. 

2. The experiments should be re-organized and polished carefully before submission, especially in Section 4.1,  4.2 and 4.5. 

3. Some qualitative experiments are supported be added to validate the differences in report accuracies under various mini-batch sizes.

questions:
The impact of mini-batch size. 

The impact of aggregation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311441660

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal discussion

comment:
Dear reviewers and AC,

We summarize the reviews, the questions during the discussion period, and our rebuttal below.

**Strengths pointed out by the reviewers**: All the reviewers point out the **two-tier feedback component** (mini-batching and batching) of UniPrompt as a strength. All the reviewers also appreciated the reasonably comprehensive empirical evaluation of the work, ablations validating the contributions, and asked for some additional evaluations/ablations during the rebuttal. All the reviewers also recognize that UniPrompt **outperforms the state-of-the-art methods**. In general, everyone has given a positive score (3) on the technical contributions and soundness of the ideas. 

**Concerns raised**: The primary concerns were regarding ablations (1-2) and generalization aspects (3-4): (1) mini batching, (2) aggregation/clustering, (3) use of open source LLMs like LLaMA, (4) experiments on BBH benchmark which is popular in the literature.

**Our rebuttal responses/results**: 

(1) With increase in mini-batch size, we observe an increase in accuracy (as we show on the ARC dataset). It is a hyperparameter that can be tuned for each dataset.

(2) When tested on the ARC dataset, using aggregation resulted in an accuracy of **90.36%**, whereas omitting aggregation led to a reduced accuracy of 84.07%. Aggregation proves beneficial by preserving the most valuable feedback and enabling the generation of broader task facets in the refined prompt.

(3) On the ARC-challenge dataset, UniPrompt with Llama3 8B Instruct as the solver model yields **91.67%** accuracy (using one-line task description as the Init prompt) compared to the baseline performance reported in their model card which is 82.4%. Similarly, for Llama3 70B Instruct as the solver model, UniPrompt achieves **95.58%** accuracy as against the baseline accuracy of 94.4% reported on the model card. These results, along with the results in the paper, show that UniPrompt generalizes well to open-source as well as closed-source models, and across model sizes.

(4) From the [table](https://openreview.net/forum?id=ViRDmDAfjg&noteId=E3NQWM3fjM) we provide in our rebuttal regarding the BBH benchmark, it is evident that UniPrompt shows a significant improvement over OPRO for all tasks except one. For instance, it achieves significantly higher accuracy in tasks such as "_Boolean Expression_" (**92.37%** vs. 78.74%), "_Date Understanding_" (**81.96%** vs. 52.59%), and "_Navigate_" (**77.16%** vs. 51.74%). UniPrompt's prompts are also much more informative compared to that of OPRO and other baselines.

These results, along with the results in the paper, show that UniPrompt generalizes well across task scenarios (fact checking, reasoning,), domains (math, code), and types (generative, discriminative).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a new prompt optimization algorithm, UniPrompt, which exploits the structure in the prompt optimization, uses clustering to capture different facet of the task feedback and feedback mechanism to capture generalizable facets. The experiment shows the effectiveness of the proposed method comparing with other prompt optimization algorithms. Reviewers all appreciate the contribution of the work; however, the concerns still persists on the ablation studies such as different base LLMs, prompt initialization, and clustering. Addressing these concerns through additional ablation studies would strengthen the paper and provide a more comprehensive understanding of UniPrompt's capabilities and limitations.