GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers

Authors: Sarkar Snigdha Sarathi Das, Ryo Kamoi, Bo Pang, Yusen Zhang, Caiming Xiong, Rui Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Prompt Optimization

Abstract:
The effectiveness of large language models (LLMs) is closely tied to the design of prompts, making prompt optimization essential for enhancing their performance across a wide range of tasks. Although recent advancements have focused on automating prompt engineering, many existing approaches rely exclusively on textual feedback, refining prompts based solely on inference errors identified by large, computationally expensive LLMs. Unfortunately, smaller models struggle to generate high-quality feedback, resulting in complete dependence on large LLM judgment. Moreover, these methods fail to leverage more direct and finer-grained information, such as gradients, due to operating purely in text space. To this end, we introduce, we introduce *GReaTer*, a novel prompt optimization technique that directly incorporates *gradient information over task-specific reasoning*. By utilizing task loss gradients, *GReaTer* enables self-optimization of prompts for smaller, lightweight language models (LM) without the need for costly closed-source LLMs, while maintaining reasonable prompt structures. This allows high-performance prompt optimization without dependence on massive LLMs, closing the gap between smaller models and the sophisticated reasoning often needed for prompt refinement. Extensive evaluations across diverse tasks demonstrate that \ours consistently outperforms previous methods, even those reliant on powerful LLMs. Additionally, *GReaTer*-optimized prompts frequently exhibit better transferability and, in some cases, boost task performance to levels comparable to or surpassing those achieved by larger language models, highlighting the effectiveness of *"gradient over reasoning"*-based prompt optimization. Code of *GReaTer* is available at: https://github.com/psunlpgroup/GreaTer

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730611456251

thread_type:
official_review

<thread_object>
creator: reviewer_7nfg

<content>
summary:
This paper proposed a novel prompt optimization method that leverages some small language models to generate reasoning/explanation for problem solutions and utilizes the gradients on probable token candidates from the loss of answer tokens over reasoning tokens to refine the prompt optimization process. Experiments over 2 models and several benchmarks demonstrate that this method is effective.

strengths:
- The idea of applying gradients over reasoning tokens to refine the selection of prompt tokens is novel.
- The proposed method has shown empirical success.

weaknesses:
- Presentation is unclear:
    - Slight notation abuse: the output space of $f_{\text{LLM}}$ is not well-defined. Does it output texts or token probabilities? They are different in Eq. 2&3.
    - It is not clear for the main results presented in Sec. 5.2 which model is used for prompt evaluation and which model is used for optimizing the prompt (e.g., in Table 2).
    - The complete algorithm 1 in Appx. B.1 should be moved to the main text for better understanding, as the presentation of the designed method is not very easy to follow.

- Sec. 5.6 does not show much insight which I think should be omitted from the main text.
- How to justify “simply considering fLLM(y|x ⊙ p) would give us the wrong objective to optimize, which in turn will give incorrect gradient information” in lin 237-238? The idea of taking gradients over reasoning paths is not well-explained. Intuitively, I expect the distribution of fLLM(y|x ⊙ p) to be more skewed than fLLM(y|x ⊙ p ⊙ r) as we can view $r$ as more computation to search for the right answer $y$.
- What does the gradient direction $\frac{\partial \mathcal{L}}{\partial \epsilon_i}$ physically mean? Why is there a negative sign and why is the token with the highest negative gradient value selected instead of the one with the largest gradient norm?
- Recent literature [1, 2, 3] related to applying the gradient concept in prompt optimization should be at least discussed.

[1] Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023). Automatic prompt optimization with" gradient descent" and beam search.

[2] Hu, W., Shu, Y., Yu, Z., Wu, Z., Lin, X., Dai, Z., ... & Low, B. K. H. (2024). Localized zeroth-order prompt optimization.

[3] Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2024). Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery

questions:
- what if there is no intersection in Eq.4? And how to guarantee semantic coherence among different token positions?
- Can you explain why the performance gain from gradient over reasoning is much smaller on Gemma model compared with LLaMA model?
- How about the performance on 1.5B or 3B model? 9B model is actually not very small in practice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688772453

thread_type:
official_review

<thread_object>
creator: reviewer_zouy

<content>
summary:
The paper introduces a prompt optimization technique that leverages gradient information over task-specific reasoning to enhance the performance of smaller language models (LLMs). GREATER enables self-optimization of prompts without relying on large, computationally expensive LLMs, which is a significant departure from existing methods that depend on textual feedback from large models.

strengths:
1. GREATER presents an approach to prompt optimization by incorporating gradient information directly into the process, which is a departure from traditional text-based feedback methods. This innovation could reduce the computational costs associated with prompt engineering.

2. The paper is well-structured, with a clear problem statement and a detailed explanation of the GREATER method.

weaknesses:
1. While GREATER shows promising results, it is not clear how well these findings generalize to other types of tasks beyond reasoning tasks. Future work could explore the applicability of GREATER to a broader range of NLP tasks.

2. Although GREATER aims to reduce reliance on large LLMs, the paper does not discuss the computational resources required for the gradient-based optimization process itself. More details on this aspect would be beneficial.

3. The paper could benefit from a more detailed comparison with other gradient-based methods, especially those that also aim to optimize prompts without relying on large LLMs.

questions:
1. Can the authors comment on the potential of GREATER to be applied to tasks outside of reasoning, such as generation or summarization tasks?

2. How do the computational costs of GREATER compare to traditional text-based prompt optimization methods, especially when considering the training and inference phases?

3. The paper mentions the impact of different initialization prompts. Could the authors elaborate on how sensitive GREATER is to the choice of initialization and whether there are best practices for initializing prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688790700

thread_type:
official_review

<thread_object>
creator: reviewer_24zw

<content>
summary:
The paper presents GREATER, a novel prompt optimization technique that enables smaller language models to self-optimize their prompts without relying on larger, costly language models.

strengths:
* This paper addresses a significant limitation in current prompt optimization methods
* This paper involves gradient information to prompt optimization
* This paper shows practical utility for improving smaller language model performance
* Figure 1 clearly demonstrates the difference between GREATER and previous methods

weaknesses:
* The case studies section could benefit from more detailed analysis of why certain prompts work better
* Could benefit from more theoretical analysis of why gradients over reasoning work better
* Line 462 explicitly mentions large language models (LLMs) which have already been defined in the introduction section. There seems like a typo in line 1186 as well, "Use these logical reasoning process steps and explain Step. step. Here is correct answer.","step" appears multiple times.
* Could include more diverse task types beyond reasoning tasks

questions:
* How does the method scale with increasing model size? Have you considered using GREATER on larger open-source models like Llama-3.1-70B?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a gradient-based approach to optimize a set of candidate prompts towards a target task, represented by a data sample that can be used for "learning" the prompts. The proposal is akin to recent methods that leverage an external model to iteratively improve the prompt, though as authors argue, requiring access to that external model is a major limitation of existing methods, and the proposal in the paper, dubbed GReaTer, offers a self-improving alternative. The approach works by performing gradient descent steps over one-hot vectors that continuously represent the vocabulary elements. In each time-step, continuous gradient updates are performed on one-hot codes of the inputs, and the tokens are replaced by the argmax of the one-hot representation after the update.

The reviewers are unanimously in favor of acceptance, and I side with them since the community could clearly benefit of more efficient prompt tuning frameworks. I would nonetheless note limitations of the approach in that it requires full access to the model one wants to optimize for, while tuning methods that rely on external auxiliary prompt optimizers also support black box language models. I would also add that, as highlighted by the authors, the proposal could yield prompts that are not grammatically correct, or even more so, non-natural (or unsafe/unethical) prompts that still perform well in the task at hand could be obtained. An investigation on to what extent that risk is relevant or at least an in-depth discussion is missing from the paper.