GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
Authors: Sarkar Snigdha Sarathi Das
Ryo Kamoi
Bo Pang
Yusen Zhang
Caiming Xiong
Rui Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Prompt Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a prompt optimization method using gradient over reasoning to boost performance on open-source, smaller language models.

Abstract: The effectiveness of large language models (LLMs) is closely tied to the design of prompts, making prompt optimization essential for enhancing their performance across a wide range of tasks. Although recent advancements have focused on automating prompt engineering, many existing approaches rely exclusively on textual feedback, refining prompts based solely on inference errors identified by large, computationally expensive LLMs. Unfortunately, smaller models struggle to generate high-quality feedback, resulting in complete dependence on large LLM judgment. Moreover, these methods fail to leverage more direct and finer-grained information, such as gradients, due to operating purely in text space. To this end, we introduce, we introduce *GReaTer*, a novel prompt optimization technique that directly incorporates *gradient information over task-specific reasoning*. By utilizing task loss gradients, *GReaTer* enables self-optimization of prompts for smaller, lightweight language models (LM) without the need for costly closed-source LLMs, while maintaining reasonable prompt structures. This allows high-performance prompt optimization without dependence on massive LLMs, closing the gap between smaller models and the sophisticated reasoning often needed for prompt refinement. Extensive evaluations across diverse tasks demonstrate that \ours consistently outperforms previous methods, even those reliant on powerful LLMs. Additionally, *GReaTer*-optimized prompts frequently exhibit better transferability and, in some cases, boost task performance to levels comparable to or surpassing those achieved by larger language models, highlighting the effectiveness of *"gradient over reasoning"*-based prompt optimization. Code of *GReaTer* is available at: https://github.com/psunlpgroup/GreaTer

Published as a conference paper at ICLR 2025
GREATER: GRADIENTS OVER REASONING MAKES SMALLER LANGUAGE MODELS STRONG PROMPT OP- TIMIZERS
Sarkar Snigdha Sarathi Das Caiming Xiong Rui Zhangi "The Pennsylvania State University fdd5555, rm25227 (es22227@ppu.eed
Ryo Kamoi!
Bo Pang
Yusen Zhang?
SSaesforce Research
ABSTRACT
The effectiveness of large language models (LLMs) is closely tied to the design of prompts, making prompt optimization essential for enhancing their performance across wide range of tasks. Many existing approaches automating prompt engineering rely exclusively on textual feedback, refining prompts based solely inference errors identified by large, computationally expensive LLMs. Unfor- tunately, smaller models struggle to generate high-quality feedback, resulting in complete dependence leverage more direct and finer-grained information, such as gradients, due to op- erating purely in text space. To this end, we introduce GREATER, a novel prompt optimization technique that directly incorporates gradient information over task- specific reasoning. By utilizing task loss gradients, GREATER enables self- optimization of prompts for open-source, lightweight language models without the need for costly closed-source LLMs. This allows high high-peeffrrmnce prompt opti- mization without dependence massive LLMs, closing the gap between smaller models and the sophisticated reasoning often needed for prompt refinement. Ex- tensive evaluations across diverse reasoning tasks including BBH, GSM8k, and FOLIO demonstrate that GREATER consistently outperforms previous state-of- the-art prompt optimization methods, even those reliant powerful L.Mss. Ad- ditionally, GREATER- opttimized prompts frequently exhibit better transferability and, in some cases, boost task performance to levels comparable to or surpass- ing those achieved by larger language models, highlighting the effectiveness of prompt optimization guided by gradients over reasoning. Code of GREATER is available at: https: htt//tttubb./////////////t httpuuu.cooooonnoooooo//rrrreee
large LLM judgment. Moreover, these methods fail
INTRODUCTION
Large Language Models (LLMs) have demonstrated impressive performance across various task do- mains (Brown 2020 Achiam et al.) 2023 Reid et al. 2024). However, these models are known to
exhibit prompt sensitivity, a phenomenon where slight variations in input prompts lead to sig- nificant differences in output quality (Lu et al. 2021
Madaan & Yazdanbakhshi 2022 Zhao et al.)
2021 Reynolds & McDonell 2021 Wei al. 2022 Kojima et al.) (022). Consequently, prompt design has emerged as a critical factor in achieving optimal LLM performance. As the popular- ity of LLMs has surged, "Prompt Engineering" has become focal point of attention in the field. Traditionally, this process has been carried out by domain experts who iteratively query expensive LLMSs until the desired response obtained. However, this manual approach is cone-consu and resource-intensive, prompting researchers to explore more efficient alternatives. Recent research has focused on Automated Prompt Engineering (Zhou et al. (022). which aims to systematically search for prompts that improve target task performance. Following this line of research, (Pryzant et al.) 2023 Ye et al. 2023) improved upon it by resorting to computationally expensive stronger LLMS to reason about failure causes in smaller, efficient LLMs deployed in practical tasks. Pryzant et al. 2023) termed this feedback as "textual gradient", since this feedback is leveraged to improve the prompts iteratively.

---

Published as a conference paper at ICLR 2025
Textual Feedback based Prompt Optimization
Our Gree Ter Prompt Optimization Based on Loss Gradient
LM suggested candidates
RC. Iox O prompt)
Prompt
Smaller
Larger
Prompt
Answer Extraction
Task Input
LM,
LM,
Task Input.
Smaller LM, F,
Reasoning,
F
Lyst prompt extractor)
X
Rely on Larger LMs judgement and refinement
Optimization directed by loss gradient, a more direct signal
x Requires Larger Proprietary LMs
Does not require larger LMs prompt optimization
Suboptimal Performance amaller models
GreaTer Performance smaller models
X Limited Transferabiltyy to smaller models
GreaTer Transferability to smaller models
Comparison of textual feedback-basee prompt optimization and GREATER. Left: textual feedback relies entirely on larger language model's judgments. Right: GREATER avoids external large. proprietary models, using token suggestions from small model and guiding prompt token selection with loss gradients. GREATER incorporates model reasoning by first generating reason- ing, then applying an extraction prompt to obtain answer logits for computing loss gradients. This "gradient over reasoning" approach optimizes using direct signals rather than relying on language model feedback.
Figure 1:
Despite showing promising performance, the primary limitation of this category of prompt opti- mization is the reliance on massive LLMs like GPT-4 for optimizing smaller model performance. Smaller L.LMs used practice rely on large models like GPT-4 for optimization, as these big models generate the "textual gradients" needed to refine and transfer knowledge. (Zhang et al. (024) found that smaller LMs are incapable of generating such optimization feedback, further emphasizing the dependence on large models. Thus, enhancing smaller models depends on the computational power of larger ones. Additionally, the optimization process increases computational costs due to the need for sizeable prompt length due to multiple task samples, and heavy dependence on the optimizer LLM's judgment may result in less reliable outcomes.
To mitigate these issues, propose GREATER that allows smaller LLMs to optimize prompts using true gradients (i.e., numerical loss gradients) without resorting to larger models. Figure (right) gives an overview of our approach. GREATER leverages "gradient over reasoning" for more accurate prompt improvement direction. GREATER first calculates the forward token probabilities to generate small number of probable token candidates at the selected position conditioned on the input. Then, it utilizes the LLM to generate the reasoning for problem solution, and extracts the final answer logits for calculating the loss. Finally, we leverage the gradient calculated for the probable token candidates to select the best tokens for optimization. Our technique innovates by tackling token discreteness while integrating reasoning chains, crucial for limited datasets like Big Bench Hard (uuzuun et al. 2022). which provide final labels without explicit reasoning paths.
GREATER shows strong prompt optimization performance, where optimized prompt often deliv- performance equivalent larger LLMs in solving the task. In our experiments, we selected -1ama-3-8B-Inst ruct (Meta) 2024) and Gemma-2-9B-it (Team et al.) 2024), two highly popular smaller language models that proven to be also very useful in solving different tasks. Across a wide variety of selected BBH (Suzgun et al.) 2022 tasks, mathematical reasoning task
GSM8k, and first-order logic task, FOLIÓ (Han al.) 2022). GREATER shows up 8.9% perfor- mance improvement over SOTA prompt optimization technique on average in BBH suite of tasks. Moreover, GREATER optimized prompts perform on par or better than GPT-4 optimized prompts Ye et al. 2023). demonstrating superior performance without resorting to larger proprietary LLMs.
2 RELATED WORK
LLMs as Prompt Optimizers. Recently, significant attention has been brought to the prospect of
LLMs as prompt optimizers for less powerful models. This line of work was first proposed by Zhou et al. (2022) by prompting LLMs with input-output pairs to infer the instruction. Pryzant et al. (2023) formalized the term "textual gradient" to refer to textual feedback based optimization. Here the authors introduced the use of mini-batches of data to create natural language feedback. These
2

---

Published as a conference paper at ICLR 2025
gradients critique the current prompt, mimicking the role of numerical gradient in optimization.
Later, a large body of works has improved upon it by using optimization logic in text space (Yang
et al. 2023 Yuksekgonul et al. 2024). meta-prompt engineering YYe et al. 2023). agent-based
learning and reasoning (Wang et al. 2023 Liu et al. 2024; [Shinn et al. 2024). external trained
model (Cheng al.] 2023), evolutionary algorithms (uuo al.] 2023 Liu et al. 2024), etc. (Hu
et al. (2024) introduced ZOPO, leveraging Gaussian processes inspired by the Neural Tangent Ker-
nel systematically explore local optima prompt optimization using zeroth-order methods. Other approaches like programming model (Khattab et al. 2023), editing (Prasad et al.] (022), and rein- forcement learning (Deng et al. 2022) are also notable. These techniques usually rely on reasoning and judgment from larger LMs to improve the performance of smaller LMs. In other words, the larger model can share its knowledge through the optimized prompt which helps smaller models achieve performance uplift. Therefore, to get strong results with smaller, more lightweight language models, prompts must be optimized using powerful, expensive, closed-source models, as smaller models are inadequate at this optimization on their own (Zhang et al. 2024).
Prompt Tuning. Frompt-tuning has been explored in prior works as task task-specific continuous vec- tors tuned by gradient- gradient-based methods to improve task performance (Li & Liang] [022 Lester al. 2021 Qin & Eisner 2021; Gao et al., 2020). Discrete prompts on the other hand involve search- ing for discrete vocabulary tokens through gradients (Shin et al. 2020 Shi et al. (022). These approaches can be further extended for visual prompt tuning (Wen et al. 2024). where the authors optimize hard text based prompts through efficient gradient-based optimization. A fundamental flaw in these methods stems from the fact that these methods are typically only suitable for classi- fication tasks or tasks with fixed input-output structures, as they rely on predefined templates and verbalizers. Reasoning tasks on the other hand require complex analytical reasoning chains, e.g., Big-Bench- (Suzgun et al. 2022), which leads to the final output, where using a fixed template verbalizer is incompatible and impractical.
Jailbreaking LLMs. Gradient-based search methods have also been applied to find trigger prompts that bypass LLM alignment-based filtering and generate harmful responses (Zou et al.
2023). These methods have been further refined to improve readability and effectiveness by includ- ing perplexity regularization and constraied--decodinn
(Guo et al. [021; Alon & Kamfonas 2023] Liu et al., 22023] Guo et al.] 2024). Similar to prompt tuning, these methods also adhere to a simple input-output structure. The target output typically is affirmative response "Sure here is". without
emphasis on the reasoning chain.
3 PROBLEM DEFINITION
We formally define the problem of prompt optimization to lay the foundation of the optimiza- tion target. Given language model fLIM. and small representative task dataset, Drask
(.l,,,),.... (Ina }n)] the goal of prompt optimization is to find a prompt pd such that:
pd arg max
Σ (
m (fuLM(z;p).)
(1)
P
where fLLM(X:)) p) is the output from task language model fLLM upon channeling the input x with the prompt p, and m(-) is the evaluation function for this task.
Textual Feedback Based Prompt Optimization. As shown in the left part of Figure | to search for p", previous prompt optimization methods based on textual feedback use an optimizer model foptimizer which is usually substantially larger and more expensive than fULM (Zhou et al. 2022 Yel
et al. 2023 Pryzant et al. )202). Conceptually, fopumizer mm fLLM (x;; p), (z) y) Drask) drives the optimızation process by assessing and providing feedback for refining the prompt. Therefore, finding p² primarily relies on the capabilities of Soptimizer and its hypothesis for prompt refinement.
4 OUR METHOD
While Section 3 provides the formal explanation of prompt optimization, it's important to understand the role of reasoning in this process. A well-crafted prompt gives clear problem-solving instructions
3

---

Published as a conference paper at ICLR 2025
Final Prompt
Updated Prompt
Use parerdhesss
pensent seas resscrting think by step.
Candidato Gradient Calculation
the stop wise onder.
de
10112+3'9) 9)...
Handle the operations...
Initaal Prompt
Use logigal reasoning ng
logi:
Evalualo the fi's part...
Therafore, the fhaa answer NUMBER)
00
06+9 34 80 ...
and thin dighity siep......
First handle additons...
-50
8:4+8-3r9)-
Embeddings.
Exaluste part..
702
Task Samples
Top-k Logits LLMs
One-Hot Vector,
Remsoning Stops
Answers
Figure 2: Overall workflow of GREATER. (i) The language model fLLM generates token candidates by conditioning on input samples. (ii) fLLM uses task input and current prompt to generate reasoning and extract final answer logits. (iii) The logits are used to calculate loss and compute gradient over generated reasoning with respect the candidate tokens. These gradients determine the selection of candidate token to update the current position of the current prompt.
that guide the language model to think through the problem in a specific way, helping it arrive at a valid answer. Our method, GREATER, is built on this principle. As shown in Figure 2 GREATER begins by analyzing task examples to propose potential token candidates, essentially exploring dif- ferent ways solve the task. The task input and prompts are then used to generate reasoning steps, from which the final answer and loss are extracted. By applying gradients over the generated rea- soning, GREATER refines the candidate selection, ensuring it follows the optimal path for improved performance.
4.1 MЕTHON OVERVIEW
Given an input and prompt p [pI. Ps....... consisting of several tokens Pi, fLLM generates the reasoning chain, r ~ fLLM (x O p) for the input (O for concatenation). Then, we can extract the final answer from r by prompting fLLM with a formatted extractor prompt Pextmct, e.g., "Therefore, the final answer (SNUMBERS) is $" Consequently, this produces the final answer logits, y' fULM(I 2 p ④ r C Pexunet) We define our loss function as:
pppprrr C L= (LLM ( Pextnett
(2)
In Eq. 2 for fixed Pextract and x: only p will affect the loss; therefore can calculate Op Pr.pnm This loss gradient takes the reasoning into account, making it more direct signal drive the prompt optimization process. Therefore, this "gradient over reasoning" can be highly potent alternative to current textual feedback based prompt optimization that entirely relies on massive LLM feedback. Note that, in Eq. 2 the entire chain is equivalent to one single forward pass through fLLM. since key, values from reasoning generation can be cached and utilized during logit extraction.
dc
We sequentially apply GREATER optimization in each token position ps of p. GREATER first em- ploys a Candidate Proposal (Section (42) stage, where uses language model-guided forward prob- abilitics to generate potential token candidates for optimized prompt. Subsequently, the current prompt is channeled through LLM, fULM for solution generation, and then answer logits extraction (Section 4.3) Using these logits, we can calculate the loss and gradient with respect to the small number of prompt token candidates which we use for token selection. (Section 4.4). This process can be applied over all the prompt token positions repeatedly.
In the following subsections, we discuss each stage of GREATER to optimize prompt token at position i. Algorithm [ shows all the steps in GREATER. This process is sequentially applied over all the positions repeatedly until convergence.
4.2 PROMPT TOKEN CANDIDATE PROPOSAL.
For optimizing prompt token pt, we first leverage the task language model (LM) fLLM probabilities to propose candidates for the position. For a sample input x, E Drask, we can calculate the top-k probabilities for candidate token proposals:
cand, top-k top-k [LIM @ P1,, P.. .........PP........ P(-1]
(3)
4

---

Published as a conference paper at ICLR 2025
Algorithm GREATER
Input: Initial Prompt Pinit
(P1, Ph, Ps....), Task Dataset Dunk = (X,Y), Language Model
fLLMM. Extractor Prompt Pextmat
2: Output: Optimized Prompt p
3: P. A.P", Lbest < Pinit Pinits
4: position, =-0
5: for tort- to T do
6: // Cannddatte Proposal Stage
7: candidates; + -Nayee, cand, where Dy C Pask
From Eqn. and Eqn.
8:
Compute one-hot token indicator e, for candidates; // Reasoning Generation and Extraction
From Section
9:
10:
Calculate logits, Y <- {fLL @ p fLLM(X ④ p) ④ Pextrac) ) E D)
Extract Solution
and Logits (From Eqn. 5
11:
// Gradient Over Reasoning driven Candidate Selection
12:
Compute total Loss L(Y, Y) for 'D and Oc
From Eqn.
öc. // select top-p tokens with gradients and find the candidate with the lowest loss aL) 26,)] where μ = 3
13:
14:
selections, -- token[top-p/
15:
Lmin &- arg min seslectioos; (x,y)E L (fra(( &pei @
> Find the best
candidate, assuming arg min also returns mos_ if Lmin Liest then
17:
Loea t Lymin
A Update the best loss
p¹ +p it 1) mod length(Pmm))
Update the best prompt
19:
return p*
Therefore, we take top-k tokens from fLIM conditioning 2j followed by the previous tokens from the optimized prompt. We calculate cand.,, for randomly sampled set inputs, D, C Dussk Since we want the token candidates to be relevant for all samples in the task, token candidates for position will incorporate candidates for all samples, so
candidates;
n cand..j
2yED,
Equation gives candidates,, of promising token candidates for position i. Since candidates; are suggested by LM and conditioned on the inputs, they are representative of the problem domain and more interpretable. Therefore, gradient mass will be calculated only for small number of promising token candidates, c. preventing it from being dispersed over the entire vocabulary.
Hot Token Indicators. As shown in Figure 2 (i), one-hot token indicator e; is created for only candidates, and the current token pt. with value of one only for and zeros for all other candidates. Concurrently, from the original LM embedding table, we extract only the rows that correspond to 6; to create a subset of the original embedding table, E. This table is then multiplied by the one-hot indicator 6,, allowing the input be passed through fLIM-
4.3 REASONING GENERATION AND EXTRACTION
A straightforward way of calculating loss and subsequently taking gradient over is to only consider the output of fLLM( & p) and taking cross-entropy loss of the output token that of the ground truth. However, this completely ignores the role of reasoning as discussed before, whereas modern language models require the generation of complex reasoning chain to generate correct output Wei et al. 2022). Therefore, simply considering fiLM (ylx p) would give us the wrong objective to optimize, which turn will give incorrect gradient information.
Consequently, in this stage, first generate reasoning generates reasoning to derive the final answer. To compute the loss accurately, we must extract the answer logits and compare with the ground truth label. A simple way do that is to get fuLM logits when conditioned on the input, reasoning, followed by a formatted extractor prompt, Pextract-
fLLM Oz) In r., the language model
fLIM( ④ p ① ? Pextract)
(5)
5

---

Published as a conference paper at ICLR 2025
Equation 5 results in LM logits that consider the reasoning generated from the current prompt p. Therefore, it better represents the reasoning chain making suitable for loss calculation.
4.4 GRADIENT OVER REASONING DRIVEN CANDIDATE SELECTION
EquationSggivee us the logits ý that incorporate reasoning chain information originating from current Therefore, optimize the cross-entropy loss as below. Additionally, add the perplexity regularization term Lyerpt to promote the interpretability of the optimized prompt similar to (Zou
al. 2023 Guo et al. 2024).
Cen cross entropy(O, v), Sperpt exp
Σ log fLLM p.p))) , LCe + Xepppp
(6)
However, the perplexity term is less important in our case given that we are optimizing only over the top-k candidates suggested by the LM. Consequently, the candidate proposal stage handles most of the interpretability duties.
Upon calculating the loss L. we can do a backward pass calculate the gradient over the generated reasoning with respect to one-hot token indicator eş. As de, gives us the gradient directions for each of the token candidates in Equation 4 the token replacement at position i is given by:
ac
ac OE: )
P: = token arg max(
6
In practice, select the top-three candidate tokens with the highest negative gradients and evaluate the token replacement with a forward pass on the training set. This ensures the robustness of the replacement selection process leading to higher performance from optimization.
EXPERIMENTS
In this section, we demonstrate that GREATER highly effective in prompt optimization delivering substantial performance improvement across different tasks. Section 5.1 describes the experiment setup. Section 5.2 presents the main results of the GREATER performance with smaller language models. In Section 5.3 we compare GREATER prompts optimized by smaller language models against the prompts optimized by larger proprietary language models using state-of-the-arr baseline methods. Section 5.4 performs an ablation study on the effectiveness of gradient over reasoning in
GREATER. Section 5.5 demonstrates the transfcrability of GREATER prompts. Section 5.6 shows some case studies.
5.1 EXPERIMENT SETUP
Datasets. To evaluate the efficacy of our approach,
GSMSK (Cobbe et al. 2021 Big- Bench-Hard (BBH) (Suzgun et al. 2022), and FOLIO (Han et al. 2022) benchmark datasets for diverse reasoning tasks in mathematics, commonsense, and logical reasoning. For GSM8K, we used the same test split as (Cobbe et al. 2021). For BBH, we used the similar split setting as (Yuksekgonul evaluate the natural language reasoning with the first-order logic task
al. 2024). For FOLIO, for the evaluation which also requires complex reasoning capabilities. We used the natural language reasoning task with premises text, conclusion text infer the labels. and we use the validation set of the updated version of FOLIC"" More details about these are given in Appendix A.
Models. To demonstrate that our approach can generalize to different backbone language models, choose strong and widely-used open-source language models: Llama-3-8B (Meta 2024 and Gemma-2-9B (Team et al. 2024). For both models, we use the instruction tuned versions
wttpss//huggingfacc.ooooo/ie wwt/eta.ccc/a/e/////...........t -lam///ataa-aaaaaaa-- Ins-tre
https:/hugginggfacccoo htw.:///n......oooooooo////////.

---

Published as a conference paper at ICLR 2025
Table 1: Overall results. GREATER brings substantial performance improvemetts across different
reasoning tasks, demonstrating its efficacy in prompi optimization with smaller models. It consider- ably outperforms state-of-the-ar prompt optimization methods. Detailed prompts and results with breakdown across all the tasks are shown in Appendix Hnd and Appendix
Gemma-22
Llama-3-8B
Method
GSM8K
BBH
FOLIO
GSM8K
BBH
FOLIO
ZS-CoT (Kojima et al.
2022
88.6
71.7
65.0
79.6
62.2
58.6
APB (Zhou ct al. 2022
88.6
71.7
67.5
79.9
63.1
57.6
APO
(Pryzani
2023
88.6
72.3
63.1
81.1
62.7
58.6
PE2
Ye et al. 20233)
88.6
68.9
62.1
80.1
61.5
62.6
TexIGrad Yukkskkonuul et al.
2024
87.8
72.9
67.5
78.5
58.5
56.2
GREATER
89.4
76.6
69.1
82.6
68.7
62.6
Baselines. We compare with state-of-the-art prompt optimization baselines: APE (Zhou al. 2022), APO (Pryzant et al. 2023). PE2 (Ye et al., 2023). TextGrad (Yuksekgonul et al. 2024] where we use them on Llama-3-8B and Gemma-2-9B for optimization. In addition, we also show the efficacy of our approach by comparing against the prompts optimized by massive proprietary LLMs (e.g.. GPT-4, PaLM-2-L) using APE, APO, PE2, TextGrad, OPRO (Yang et al. 2023). Evo-
Prompt (uuo et al. 2023
5.2 OVERALL RESULTS
We evaluate performance of GREATER in comparison to state-of-the-art baselines, including APE, APO, PE2, TextGrad, and the original zero-shot chain-of-thought approach (Kojima et al. 2022]. across the GSM8K, BBH, and FOLIO benchmarks. Table highlights the significant and consistent improvements achieved by GREATER over ofate-of-theeart prompt optimization meth- ods, especially when optimizing prompts using lightweight language models. Across both Gemma- 2-9B and Llama-3-8B, GREATER demonstrates remarkable stability and outperforms the baselines.
This is particularly noteworthy given the high variability in performance seen in these textual feed- back methods, which tend be unreliable with smaller models due increased prompt sensitivity. For instance, while TextGrad performs reasonably well on BBH with Gemma-2-9B. it falters sig- nificantly with Llama-3-8P
contrast, GREATER delivers outstanding results across the board, excelling across diverse tasks. Note that, in the FOLIO dataset, GREATER performs on with PE2 (Ye et al., 2023), suggesting that the model has likely reached its performance ceiling, leaving minimal room for further improvement.
5.3 COMPARISON WITH PROMPTS OPTIMIZED BY LARGER PROPRIETARY MODELS
Besides optimization performance in lightweight language models, we also evaluate the efficacy of GREATER optimized prompt compared with the prompts optimized by massive LLMs. These mas- sive models, like GPT-4 and Pa----2-L, inherently possess deeper knowledge of complex reasoning tasks and often provide richer guidance in their optimized prompts, making them strong baselines to compare with. In Table 2] we compare the performance of GREATER optimized prompt to that of APE (optimized by GPT-4), APO (optimized by GPT-4), PE2 (optimized by GPT-4), OPRO (optimized by PaLM-2-L), EvoPrompt(optimize by GPT-3.5). We use GSM8K, and five randomly selected tasks from BBH to compare the performance of these prompts. We see that in both Llama-3- 8B and Gemma-2-9B. GREATER optimized prompts perform substantially better and robustly com- pared to baseline prompts optimized with massive LLMs. The consistency performance uplift makes GREATER viable choice to boost task performance in reasoning tasks with lightweight language models.
5.4 ABLATION OF GRADIENT OVER REASONING
As outlined in Section and Section dient Over Reasoning. Conventional gradient- gradient-based optimization methods rely solely on the tar- get answer labels to compute the loss and gradients, overlooking the crucial role that reasoning
feature of GREATER is the concept of Gra-

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of GREATER with prompts optimized by larger proprietary LLMSs. GREATER performs on par with or notably better than prompts optimized by GPT 4 and PaLM-2-L across GSM8K and five randomly chosen BBH tasks using Llama-3-8B and Gemma-2-9B, EvoPrompt does not report its prompts on GSM8K. Here, Target Model: Llama-3-88 and Method (Optimized by): APE (GPT-4) indicates that Llama-3-8B was used for prompt evaluation while the prompt was optimized by GPT-4 with APE.
randomly chosen tasks)
Target Model
Method (Optimized by)
GSMBK
movie
object count.
tracking live.
hyperbatio
causal
Average
APE (GPT-4)
80.7
50
50
76
56
62.8
Llama-3-5 3-8-3-8B
EvoPrompt (GPT-3.5)
$6.0
APO (GPT:4)
81.1
56
68
49
75
51
59.8
PE2 (GPT-4)
81.5
OPRO (PaLM-2-L)
82.3
60
40
70
57
61.0
GREATER (Llama-3-8B)
82.6
71.6
Gemma-2-990 2-99
APE (GPT-4)
89.2
67.0
EvoPrompi (GPT-3.5)
51
70
82
83
61
69.4
(GPT.4)
PE2 (GPT-4)
89.6
50
65
71
84
64
66.8
M-2-1.)
GREATER (Gemma2-99B
89.4
56
85
88
75.4
Llama-3-8B: Impact of Gradient Over Reasoning Gemma-2-9B: Impact of Gradient Over Reasoning
Llama-3-88 Reasoring
70
Germma-2- 2-98 Reasoning
84
86
Ulsma-3-06 Reesoning
50
Germma-2-90 w/ Reasoring
50
Performance
42
Performance
52
40
40
20
20
10
movie rec.
tracking five.
movie rec.
tracking five.
Task
Task
Figure 3: Ablation study on "Gradient Over Reasoning" in GREATER. Gradient calculation without reasoning causes notable performance drops, showing the importance of reasoning for gradients.
plays in this process. To highlight the significance of incorporating reasoning into gradient cal- culations, we conducted a comparative analysis of GREATER with and without applying Gradi- Over Reasoning, while keeping all other steps identical. For this experiment, with both Llama-3-8B and Gemma-2-9B on two BBH tasks: movie.recommende. recommendat in and tracking-shur weffffeddool ed wwwpeetsssiieeoeeeeeeeeeeeoooeee ve-oojectss task. Figure 3 shows the comparison be- tween with and without gradient over reasoning. It clearly demonstrates a substantial performance drop when Gradient Over Reasoning omitted. Fundamentally, from Equation [S] removing the reasoning generation part r equates to calculating an incorrect objective function, which will drive gradient-based token selection diverge from the optimal solution.
optimized
5.5 PROMPT TRANSFERABILITY
We conduct further experiments to evaluate the transferability of GREATER optimized prompt. In Section 5.5.1 we first evaluate the transferability GREATER prompt between two smaller lan- guage models, Llama-3-8B and Gemma-2-92. by evaluating the performance of Gemma-2-95 opti- mized prompts on Llama-3-8B, and vice-versa. Next, in Section5.3.2] we evaluate the transferability of prompts from smaller model (I.lama-3-8B) much larger model (Gemma-2-27BB)
8

---

Published as a conference paper at ICLR 2025
Table 3: Transferability of Llama-3-8B optimized prompts Gemma-2-9B (Upper) and vice versa (Lower). The results demonstrate that prompts produced by GREATER exhibit strong transfcrability compared with those produced by other state-of-the-ar prompt optimization methods.
Turget Model
BBH (5 randomly chosen tasks)
Method (Optimized by)
movie.ree rec.
object count.
tracking five.
hyperbaton causal judgement
Average
Llama 1omaa-3-80 3-88 - Gemma-2-PP
Liama-3-88 Gemma-2-9B
TextGrad (Llama-3)
53
78
56
63
66.8
APO (Llama-3)
84
58
69.4
PE2 (Llama-3)
84
60
69.6
GREATER (Llama-3)
55
90
85
91
60
76.2
APO (GPT-F
'22
84
72"
82
39
66..
Gemma-2-9B
Llama-3-8B
TextGrad (Gemma-2)
35
29
49
65
36
42.8
APO (Gemma-2)
69
PE2 (Gemma-2)
56
69
53
55.4
GREATER (Gemmna-2)
58
87
56
70
52
64.6
APO (GPT-4)
55
68
49
75
51
55,,8
Table 4: Transferability of Llama-3-8B optimized prompts to Gemma-2-27B. The results demon- strate that GREATER optimized prompts exhibit strong transferability from smaller larger lan- guage models.
Target Model
BBH (5 randomly chosen tasks)
Method (Optimized by)
movie_rec.
object_count.
tracking_fiv
hyperhation
causal jundemjjddementt
Average
Llama-3-88 1-ma--3-BB- Gemma-2-278
Gemma--2277
PE2 (Llama-3)
92
83
54
72.2
APO (Llama-3)
53
92
83
72
58
71.6
Ours (Llama-3)
59
91
86
57
75.0
APO (GPT-I)
92
38
5.5.1 TRANSFER BETWEEN GEMMA-2 AND LLAMA-3
As shown in 3 GREATER exhibits exceptional transferability across smaller models compared to other methods. Notably, prompts optimized by GREATER significantly outperform even APO prompts, which were optimized using GPT-4, reported by Pryzant al. (2023). This distinction is particularly important, as it highlights that GREATER optimized prompts are not only more effective also better suited for broader, more generalized use across smaller models. These results clearly emphasize the efficacy and versatility of our approach.
5.5.2 TRANSFER TO LARGER MODELS
To assess the transferability GREATER-optimizze prompts to much larger models, evaluated their efficacy using the Gemma-2-27B model. As demonstrated in Table the GREATER-optiimzee prompts continue exhibit strong performance when compared baseline methods. However, notable observation is that the performance gap between GREATER and the baselines narrows as the model size increases. Additionally, we notice that APO GPT-4 optimized prompts yield similar performance with GREATER across various tasks, often resulting in closely matched outcomes.
Tables 3 and reveal that optimized prompts enable smaller models achieve performance com- parable to larger ones, highlighting an efficient approach to maximize model capabilities without relying on resource-intensive models.
5.6
CASE STUDY
One of the key characteristics of GREATER is that the generated prompts tend to more varied and innovative compared to the textual feedback-based SOTA baselines. As noted in Table T and Table
14 the optimized prompts from traditional approaches are often verbose variations of the original query standard Chain of Thought (CoT) prompts. In contrast, GREATER prompts frequently

---

Published as a conference paper at ICLR 2025
Table 5: Example prompts (abridged) generated by GREATER and APO. GREATER prompts guide structured ways to solve tasks, leading to improved task performance compared to traditional Chain of Thought (CoT) prompts and their variations often generated by textual feedback- baadbc optimiza- tion methods like APO. More examples can be found in the Appendix and
Task
Optimized Prompt by GREATER
Optimized Prompt by APO
Hormalformal fallacies
Use formal notation and and think step
Analyze the argument step by step consid- ering premises, logical
-aaaa- judgement
Use causal diagram...
Analyze the situation by identifying the di- rect and indirect causes...
1ama3-object counttin llaman-nvvigate
Use only addition. Add think step by Use your reasoning here. I would like numbers assigned.. to., To represent mov- ing
Let's think step by step.
Analyze ze the instructions step by step, con- sidering each action's...
Ilama3-
Use context or a sentence similar prior knowledge. Assume you a journalist, would have been covering NHL hockey Min- nesota before joining this assignment to report sports.
Assess the plausibility of sentence, considering both literal and figurative meanings, as well context and domain knowledge. Evaluate the sentence's coher- ence and relevance the given context... Let's think step by step.
uoorsssunderttnniing
gemma-
Use parentheses and and the step wise or- der...
multistep arithmetic two dnngrrstaddnnddddddddnnnnnggg
Use your format Excel formula for this an- swer to find it... Use your logic. Please answer. person.... Analyze the given text and answer
Let's think step by step.
semma_reasmningcolore
provide highly insightful guidelines that better aid in problem-solving. Table 5 demonstrates a selection of abridged prompts generated by GREATER and APO. For tasks like formal fallacies and causal .understanding, GREATER prompts encourage logical analysis rather than mere language- based reasoning, which enhances performance. Similarly, in the object.counting and navigate tasks, the prompts simplify the original problem by converting it into a more straightforward mathematical task, leading to significant improvements. More interestingly, in the sports_ understanding task, the prompt promotes agentic behavior, encouraging the model to take an active role in solving the task. Additionally, for multistep.arithe arithmetic two, the prompt focuses on parentheses to avoid errors in computation, in date understanding task it instructs to convert into programming problem (by waaggaaboo_oooooeeeeeee
using Excel formula), and in prompts are comparatively more interesting than prompts generated by textual feedback (c.g. APO) where most prompts only show verbose instruction regarding the original question, without any guidance of how to solve the problem. They also demonstrate that GREATER prompts do more than just provide basic Instructions-thed offer clear and practical strategies tailored to each task. By helping the model think about problems in more structured and systematic way, GREATER leads to better problem-solving and improved performance compared to traditional methods.
focuses on logical analysis. These
While GREATER generates highly effective and innovative prompts, they may occasionally exhibit minor grammatical issues or a more informal tone compared to standard prompts. For instance, the navigate task and multistep.arithmeet .two task in Table reflect such characteristics. However, these issues can be mitigated by adjusting the Top-k parameter in Equation 3 Incorporating dynamic Top-k selection could further enhance the naturalness and accuracy of the prompts.
6 CONCLUSION
We present GREATER, novel gradient-guided prompt optimization technique that enhances perfor- mance without relying on massive proprietary LLMs. GREATER proposes token candidates using sample inputs, generates reasoning, and extracts final answer logits to compute loss and gradients. This "Gradient over Reasoning" provides a strong signal for selecting optimal candidates, enabling significant performance gains on tasks like BBH, GSM8K, and FOLIO, outperforming state-of-thee Additionally, GREATER prompts demon- strate notable transferability across smaller models and often match the performance of larger mod- els. Future directions include integrating GREATER with textual feedback-bars based methods for more robust and effective prompt optimization.
art textual feedback-barc based methods for smaller models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported in part by NSF grant 2338418.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXi::2333.08774 2023.
Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, arXiv preprint arXiv:2308. 14132, 2023.
Tom B Brown. Language models are few-shot learners, arXiv preprint arXiv:2005.14165 2020.
Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Kc, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlic Huang. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv2231.004555 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint wrXiv:2110.14111 2021.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205. 0212205..254.. /2548, 2022.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. 2020.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kicla. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.1377 2021.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.0853. 2023.
Xingang Guo, Fangxu Yu. Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking 11ms with stealthiness and controllability. arXiv preprint arXiv:2402.0848 022.098777. 2024.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng. Yujie Qiao, Luke Benson, al. Folio: Natural language reasoning with first-order logic. arXiv preprint wrXiv:2209.00844 2022.
Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiangqiang Lin, Zhongxiang Dai, See-Kiong Ng. and Bryan Kian Hsiang Low. Localized zeroth-order prompt optimization. arXiv preprint arXiv:2403.02993, 2024.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714. 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2223 22213, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint SXXx:2104.08691 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00/90, 2021.
11

---

Published as a conference paper at ICLR 2025
Shengeai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew-Soon Ong. Large language models as evolutionary optimizers. In 2024 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint NXXv::204400876, 2021.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.0766 07666, 2022.
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2024.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-hased in- struction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495, 2023.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.06550 06599, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrae, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstructs of the 2021 CHI conference on human factors in computing systems, pp. 1-7, 2021.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's 's the shining is good movie, and good prompt too? arXiv preprint arXiv:2212. 2022.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1 / 15900, 2020.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoc V Le, Ed H Chi. Denny Zhou, et al. Challenging big-bench tasks and whether tf can solve them. arXiv preprint arXiv:2210.09261, 2022.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.000 001000011811 18, 2024.
Xinyuan Wang. Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert- level prompt optimization. arXiv preprint urXiv:2310.16422 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in 2022.
neural information processing systems, 35:24824-2438
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.034400 2023.
Qinyuan Ye, Maxamed Axmed, Reid Pryzant, and Fereshte Khani. Prompt engineering prompt engineer. arXiv preprint arXiv:231 1.05661, 2023.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07996, 2024.
Tuo Zhang, Jinyue Yuan, and Salman Avestimehr. Revisiting opro: The limitations of small-scale Ilms as optimizers. arXiv preprint arXiv:2405.102777 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp- 12697-12706. PMLR, 2021.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. 2022.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15045 2023.
A BENCHMARK DATASETS, MODELS, AND BASELINES
In this section, we discuss the details of Datasets, Models, and Baselines.
A.1 DATASETS
As discussed in Section 5.1 we use GSM8K (Cobbe et al., 2021), BBH (Suzgun et al. 2022), and FOLIO (Han al. 2022) datasets. For GSM8K, used 100/100 for train/dev set, and original test set of 1319 size. Then, for BBH datasets, we used 21 selected BBH tasks as in Table IT] and Table 14 This covers almost all types of tasks in BBH dataset. We skip word sorting and dyck Jangulass tasks from our evaluation since we found that the smaller LLM outputs are very difficult to reliably evaluate due to highly inconsistent output pattern. Finally, for logical deduccion and tracking we use 50/15000000 10/1000 train/dev/test splits similar to Yuksekgonul et al. 2024).
select five objects tasks as representative of the tasks. For all tasks
Finally for the FOLIO dataset, used the latest version of FOLIO (Han et al. 2022 for evaluation. We use the natural language reasoming task with premises text, conclusion text to infer the labels. The original validation split (203 rows) used for the evaluation of FOLIO, whereas 50/100 samples are taken for train and dev set respectively out of the original train split.
A.2 MODELS
As described in experimentation setup, we only used Jlama-3-8B-Instree and Gemma-2-9B-i in our experiments due to their smaller footprint and strong performance. However, in several tasks Gemma-2-9B-it shows markedly stronger performance than Llama-3-8B, although the memory re- quirements for Gemma-2-92 lesser benefit from prompt optimization, as good prompts only offset for lower capabilities of weaker models. Across our experiments, we also notice substantial improvement in Hama-3-8B-IIstruus whereas the margin gets narrower for the stronger Gemma-2-99-ii.
is substantially higher. As have shown, stronger models get
A.3 BASELINES
Our primary baselines are APO PPryznt et al. 2023). iterative APE Kojima et al. 2022). PE2 Ye et al. 2023), TextGrad Yuksekgonul al. 2024h. For APO, iterative APE, and PE2 the
13

---

Published as a conference paper at ICLR 2025
original implementation by et 2023) for benchmarking. And used TextGrad's own library for evaluating their results. Other than that, we also compared against original Zero-Shot CoT (Wei and the larger model optimized reported prompts from APE, APO, PE2, OPRO YYng
et al. 2022),
et al.
2023 EyoPrompt (Guo et al. 2023).
B ADDITIONAL EXPERIMENTATION DETAILS
As shown in Algorithm we run GREATER for T = 105 steps with k = 10 for top-k, q = 5 and 0.2 (in Eq. 6). While Algorithm I shows fixed length prompt optimization, prompt. To attain that, we look for the presence of ending token in the last position. no ending token found, continue the optimization process by adding placeholder token and optimizing over it. As ending token is encountered we move over to the first position for optimization. We run all our experiments on 2X NVIDIA A100 80GB GPUs.
allow dynamic
C PERFORMANCE COMPARISON WITH OTHER GRADIENT-BASED METHODS
GREATER leverages the Gradient Over Reasoning chain to identify the optimal prompt required for enhancing task performance. Previous works, such as Shin et al. (2020), utilized gradient-gun -uideed search discover trigger tokens for improving performance sentiment analysis and natural language inference. This concept has also been extended to text- to-image tasks by incorporating gradient-bast However, these approaches are limited to simpler tasks and fail to address tasks requiring complex reasoning. As demonstrated in Table GREATER EEEETe--ppiieeed prompts deliver a substantial perfor- mance boost compared to prior methods that do not incorporate the reasoning chain.
relatively straightforward tasks like
learning in embedding spaces (Wen et al. 20024)
Comparison of performance in moviccrccommennttion and track- ing shuffled.objeets five..obeeeee.eee.. oojecct for prompts optimized on Llama-3-8B and Gemma-2-90 The results demonstrate that prompts optimized by GREATER outperform other methods across both models.
Table 6:
Llama-3-8B
Gemma-2-9B
Method
movie rec.
tracking five
movie rec.
tracking five
AutoPrompt (Shin et al. 2020)
51
38
50
73
PEZ (Wen et al. 2024) GREATER
42
35
45
73
56
70
57
86
D COMPLEXITY COMPARISON: GREATER vs. TEXT-BASED FEEDBACK APPROACHES
Assume each sample contains L input-output tokens average.
FOR GREATER
Operations Per Sample:
Forward Pass: (
Backward Pass: (
Total Complexity:
For N samples, one forward pass, and one backward pass are performed per sample single iteration.
Total complexity:
N ((T3)
14

---

Published as a conference paper at ICLR 2025
FOR TEXT-BASED FEEDBACK APPROACHES
Overview:
No backpropagation stage.
Each iteration involves, evaluating all samples, identifying incorrect samples (n in- correct samples), and chaining incorrect samples into single sequence to generate feedback.
Complexity Breakdown:
First Forward Pass:
O(NI²)
Feedback Generation:
* Incorrect samples (n) are chained together.
* Chained length: n
* Feedback generation complexity:
=
Total Complexity:
Behavior Based on Task Difficulty:
Simple Tasks (n 0):
Total Complexity - O(NL2)
Difficult Tasks (n + Ny:
Total Complexity =LW))) -(N2)))
COMPARISON
GreaTer Complexity: O(NL²) (consistent).
Text-Based Feedback Complexity: Can scale up to O()2
This complexity difference is also translated into real-world performance. In movie_ rew.rieerrcooonenaaatioon task, GREATER is required to be optimized for ~5 hours, whereas TextGrad (Yuksekgonul et al. (024) required total of ~4 hours for prompt optimization in our own setup.
E PROMPT OPTIMIZATION VS. FEW-SHOT IN --CONTEXT LEARNING
In-context learning has proven be highly effective for reasoming tasks in large language models. This raises the question of whether prompt optimization provides any advantages over in-context learning for smaller models. In Figure 4 we compare the performance of five- shot in-context learning with zero- shot reasoning using an optimized prompt from GREATER in Llama- Inst-lns The results demon- strate that GREATER offers a sig- nificant performance improvement over five-shot reasoning. More- using optimized prompt eliminates the need for repeated input-output examples during inference, leading greater efficiency.
5-Shot Zero-Shot GreaTer: Performance Comparison
70
S-shot
Zere-shat Grea
56
Performance
50
48
39
10
recommendatioon
tracking shuffled five
Task
Figure 4: Efficacy of GREATER in zero-shot setting compared to five-shot inference with Llama-3.4 1-ama-3-8B... 8-8B-Inntruch
15

---

Published as a conference paper at ICLR 2025
Table 7: Impact of Initialization Prompt. We that different initialization has resulted in different optimized prompt, however they offer comparable performance.
Initialization Prompt
Optimized Prompt
Score
Default (Use proper logical
Use movie ratings data available here above movies for reference. This HOFF has an interesting analysis based solely on options to options based on movies rat- ings. Expect from the other movies you are asked, choose option from those men- tioned below.
56
Misleading (Use no thinking just feeling.)
Use one one-liner and explain stepwise for why. ONLY READING IS ALLOWABLE AND NO CHATTY CHAT OR EXCL.
55
Table 8: Performance comparison across selected BBH tasks for Llama-3.2-1P (target model for evaluation) with prompts optimized by different methods. The results demonstrate that prompts optimized by GREATER outperform other optimized prompts.
Method (Optimized By)
movie rec.
causal judgement
hyperbaton
tracking five
object count.
AVG
APO (GPT4)
23
45
61
24
51
40.8
PE2 (GPT4)
27
54
69
21
59
46.0
iAPE (GPT4)
32
57
53
15
41
39.6
GREATER (Llama-3.2-IBB)
46
69
62
24
67
53.6
F EFFECT INITIALIZATION
For GREATER, start with fixed prompt: "Use proper logical reasoning and think step by step. Finally, give the actual correct answer." This fixed initialization convenient for adapt- ing various tasks. However, fects performance. To test this, we conduct a small-scale experiment with Llama-3-8B on the BBH BBH-movie recommendation task. Table 7 highlights the impact of different prompt initial- izations on optimization. As shown, a completely misleading initialization leads to vastly different optimized prompt. However, both prompts result in very similar performance, despite solving the task in different ways. While the default initialization produces prompt that leverages movie rat- ings and genre information from databases, the misleading prompt emphasizes concise explanations. Despite these contrasting approaches, both deliver comparable outcomes.
also explore whether using different initialization af-
G PROMPT OPTIMIZATION PERFORMANCE IN VERY SMALL LANGUAGE MODEL LLAMA-3.22 3.-.-...-------T--
While we primarily focused on two popular small language models 1lama33-8B-Instmuu and Gemma-2-9B-ii it, is also interesting to see how GREATER perform for even smaller LLM namely Llama-3.2 1-221BB-nnne B-Intruuct. Given that is one of the smallest modern language models, getting good task performance out of it just by optimizing prompt can be very useful for real world task deployment. As see in Table g GREATER optimized prompts perform significantly better than prompts optimized by GPT-4 with other methods. This elearly shows the efficacy GREATER in prompt optimization with very small language models.
16

---

Published as a conference paper at ICLR 2025
Table Optimized Prompts for 1aaamma------nstruue GSM8K dataset.
Method
Optimized Prompt
Score
TextGrad
You will answer mathematical reasoning question. Think step by step.
78.5
The last line of your response should be of the following format: 'An-
swer: VALUE' where VALUE is numerical value.
APE
Work in sequence: Complete each task order, tackling one task time, and only moving on to the next once it's finished.
79.9
APO
Break down complex problems into smaller, logical steps, considering mathematical operations, variable relationships, and implicit rules. Pro- vide clear, sequential solution, accounting for nuanced language and context.
81.1
PE2
Break down complex problems into smaller, manageable steps, and solve them step by step. Use your knowledge reasoning and think step by step. Finally give the actual correct answer.
80.1
GREATER
82.6
H PROMPT OPTIMIZATION RESULTS: LLAMA-3 -----3------NNNUE
H.1 GSM8K AND FOLIO: OPTIMIZED PROMPTS
In Table and Table TO we first show the optimized prompts with and FOLIO dataset respectively. As we can see, GREATER performs on par or better than all the baselines in every scenario.
on GSM8K
Table 10: Optimized Prompts for ILLama---BB-Insrrus on FOLIO dataset.
Method
Optimized Prompt
Score
TextGrad
You will answer a reasoning question by identifying the essential in- formation, making specific conclusions, and providing nuanced and de- tailed reasoning. Think critically and systematically, focusing on the most relevant details, and avoid unnecessary complexity... Tackle incrementally!
56.2
APE APO
57.6
Analyze the premises step by step, identifying specific details, assump- tions, and ambiguities. Draw a logical conclusion based on the evidence provided, considering multiple perspectives and potential counterargu- ments, while accounting for scope, context, and edge cases.
58.6
PE2
Analyze the statement based the provided premise, determining whether it is true, false, or uncertain. Consider all relevant informa- tion reach a logical conclusion.
62.6
GREATER
Use of logical deductions to show if your conclusion matches an ap- propriate option you chose from multiple options above by explaining how to determine whether the given conclusion follows from the given information above by explaining each step taken during the process.
62.6
H.2 BIG BENCH HARD (BBH): OPTIMIED PROMPTS AND DETAILED RESULTS
Figure 5 shows that GreaTer outperforms state-of-ther art (SOTA) prompt optimization methods, in- cluding APO (Pryzant et al. 2023), TexiGrad (Yuksekgonul et al. 2024), APE (Zhou et al.) 2022). and PE2 Ye et al. 2023), across 11.4% to 85.% of tasks. A full breakdown of these results is pro- vided in Figure 6 Upon closer inspection, it becomes evident that GREATER consistently matches surpasses the performance of other SOTA methods, highlighting the robustness and reliability of our approach. In contrast, other methods show markedly less consistency due to their sole reliance on L.M judgment.
17

---

Published as a conference paper at ICLR 2025
GreaTer vs Other Methods Win/Draw/Loss Comparison
Greater vs APO
GreaTer TextGrad
35.2%
4.8%
9.5%
Gree Ter va APE
Grealler PEZ
85.7%
4.8%
9.5%
20
80
80
100
Figure 5: Win/Draw/Loss Comparison of GREATER and SOTA prompt optimization techniques APO, TextGrad, APE, and PE2 in optimization with significant winning margin over these methods, highlighting its effectiveness in optimization.
GREATER maintains
Finally, Table 11 presents all the optimized prompts generated by the baseline methods and our proposed approach. TextGrad prompts are truncated due to their excessive length, making them challenging fit page, which may also explain their inconsistent performance.
Table 11: List of optimized prompts for BBH tasks by different prompt optimization methods with Llama-3-8B-Instee
Method
Optimized Prompt
formal fallacies
PE2
Determine the validity of the given argument.
APE
Simplify and analyze.
TextGrad
You will answer reasoning question by explicitly identifying the key relation- ships between the premises and the conclusion, and explaining how they lead to the conclusion. Use clear and concise language to facilitate understanding, and...
APO
Analyze the argument step by step, considering premises, logical connections, and conditional statements. Identify the conclusion and evaluate its validity, considering sufficient and necessary conditions, counterexamples, and alterna- tive scenarios.
GREATER
Use formal notation and and think step by step. Finally give the actual correct answer.
salient_translattn wrw.slatinnnnnnnomooooo wwtpaaatiiiiineonoeeeeeeeeeeeeee eeteeeerrreeteee detection
PE2
Identify the type of eтror in the translation from German to English. nSource: [insert source text] nTranslation: [insert translation] nError type: [one of the following] n(A) Modifiers or Adjectives n(B) Numerical Values n(C) Negation...
APE
Clarify your thoughts, break it down step step.
TextGrad
You will answer a reasoning question by providing detailed analysis of the original text and the translation. Think step by step, considering multiple possible explanations for the error. Clearly explain how each step leads...
Continued on next page
18

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
APO
Analyze the translation error by carefully reading the original sentence and iden- tifying the specific mistake. Consider the exact words, phrases, and grammatical structures to determine the correct error type from the options.
GREATER
Use your answer reasoning as had step. I would be taking correct answer.
tracking -bbjects .bveccs
PE2
Let's think step by step.
APE
Take it one step at a time: Focus on one task, complete it, then move on to the next.
TextGrad
You will answer a reasoning question by providing a step-by-step breakdown of the process. Use vivid and descriptive language describe the events, and make sure to highlight the key connections and relationships between each...
APO
Let's think step by step.
GREATER
Use this process explanation stepwise for each step until you get given above Alice has got originaly the following as follows.
causal judgement
PE2
What action(s) led to the outcome? Let's break it down step by step.
APE
Break down your thinking into clear, consecutive steps.
TextGrad
You will answer reasoning question by explicitly connecting the events and outcomes, considering multiple perspectives and potential counterarguments and providing nuanced explanations that take into account the context in which the events occurred. Think...
APO
Analyze the situation by identifying the direct and indirect causes, considering multiple perspectives, and evaluating counterfactuals. Provide a clear and concise answer, taking into account the context and nuances of the situation. Focus the...
GREATER
Use causal diagram. The correct option ask about whether there the variable C of about whether a specific cause is sufficient. The answer a causal relationship between C to D if the probability that C occurs given E changes.
cxoleannexpressions
PE2
Evaluate logical expressions step by step, considering the order of operations and specific values. Break down expressions into parts, and evaluate each part using 'or', ''an', and 'not' rules.
APE
Analyze and simplify.
TextGrad
You will answer a reasoning question by breaking down the expression into smaller, manageable parts. Provide concise and clear explanation, using precise and concise language to describe the logical operations used to arrive at...
APO
Evaluate the boolean expression by following PEMDAS and applying boolean logic rules (AND, OR, NOT). Handle parentheses carefully. Consider edge and provide a step-by-step explanation of your reasoning, including any assump- tions made.
GREATER
Use this statement with a conditional if know what the value True of and what Not False means. Or not True and also boolean. In explain your.
object_counting
PE2
Let's think step by step. Break down, step by step.
APE
Continued on next page
19

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
TextGrad
You will answer reasoning question about counting objects. Think step by step, considering the context of the question and using it to inform your answer. Be explicit in your counting process, breaking it down...
APO
Let's think step by step.
GREATER
Use only addition. Add think step by step. Finally give the actual correct answer.
navigate
PE2
Check if the instructions return to the starting point by calculating the total num- ber of steps taken.
APE
Clarify your thoughts, analyze step by step.
TextGrad
You will answer reasoning question by breaking down the problem step-by- step and providing explicit explanations for each step. Think carefully about the instructions and consider alternative scenarios. Use clear and precise language to describe...
APO
Analyze the instructions step by step, considering each action's effect on your position. Use logical reasoning determine if you return to the starting point.
GREATER
Use your reasoning here. would like numbers assigned.. to.. To represent moving.
pports_understanding
PE2
Assess the plausibility of the sentence. Is it likely to be true or fictional?
APE
Break down the task into manageable parts, examining each element thoroughly.
TextGrad
You will answer a reasoning question by providing a clear and concise step- by-step breakdown of your thought process, focusing on the most relevant and concrete evidence to support your claims. Consider alternative explanations and counterarguments...
APO
Assess the plausibility of the sentence, considering both literal and figurative meanings, as well as context and domain knowledge. Evaluate the sentence's s coherence and relevance to the given context.
GREATER
Use the context or a sentence similar prior knowledge. Assume you a journal- ist, I would have been covering NHL hockey in Minnesota before joining this assignment to report sports.
reasoning about colored objects
PE2
Analyze the input and options step by step to identify the correct answer.
APE
Break down into simpler components.
TextGrad
You will answer reasoning question by carefully analyzing the problem state- ment, identifying the relevant information, and using logical deductions to arrive solution. Use precise and accurate language describe your thought pro- cess....
APO
Let's think step by step.
GREATER
Use this problem type as inspiration! which option best represents amu, the an- swer of all my are known.
multistep arithmetic.two
PE2
Evaluate step-by-step and provide the correct answer, following the order of op- erations (PEMDAS).
APE
Decompose and analyze each part carefully.
Continued on next page
20

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
TextGrad
You will answer a reasoning question by providing a clear, step-by-step break- down of your thought process, using simple language and avoiding ambiguity. Focus on the key steps and simplify the intermediate calculations. Use descrip- tive variable...
APO
Evaluate the expression by following PEMDAS, handling parentheses, and accu- rately calculating with negative numbers. Break down complex expressions into simpler steps and provide the final answer.
GREATER
Use PEMAS reasoning here and step by the STEP to the actual number result and explain what PEMAS means by each step of how I would evaluate this ex- pression correctly according follow these step wise...
date understanding
PE2
Let's think step by step.
APE
Analyze step-by-step.
TextGrad
You will answer reasoning question by breaking down into manageable steps, focusing on simplicity and clarity in your reasoning. Provide a concise and clear explanation of your thought process, avoiding unnecessary conversions and...
APO
Let's think step by step.
GREATER
Use the date today which will not would give us an error. solution is given as answer date is correct the option data and the current month and year to get to previous and current month of year to determine what the current data will look.
ruin names
PE2
Identify the humorous edit of this artist or movie name. Choose an option that cleverly replaces word or plays on words with the original name. Options may include the correct answer.
APE
Break it down, step by step.
TextGrad
You will answer a reasoning question by providing a step-by-step analysis of the options, highlighting the unique features and characteristics of each humorous edit. Consider the linguistic and cognitive factors that contribute to humor, such..
APO
Imagine a creative reinterpretation the original name. Think outside the box and come up with a clever edit that's unexpected yet amusing. Consider tone, context, and audience when selecting the most humorous and engaging...
GREATER
Use your logical reasoning to make this, not brute force checking.CONTEEX CONTEXT is provided below.
movie recommeaamonnnoonn
PE2
Find a movie that shares similar elements with the given films, considering nar- rative structure, memorable characters, genre blending, strong protagonists, and emotional impact.
APE
Calmly analyze, think critically.
TextGrad
You will answer a reasoning question by analyzing the given movies and identify- ing the most suitable match. Think step-by-step. focusing on the most distinctive features that connect the input movies, such as unique plot twists....
APO
Analyze the movies' tone, genre, and style, considering action, drama, and com- edy elements. Identify the most fitting movie from the options that shares these characteristics, focusing overall themes and elements rather than individual features.
Continued on next page
21

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
GREATER
Use movie ratings data available here above movies for reference. ThisHOFF has an interesting analysis based solely options to options based movies ratings expect from the other movies you are asked ones mentioned here you...
web of lies
PE2
Evaluate statements about the truthfulness of others in chain of lies or truth- telling. Determine speakers are telling the truth or lying, considering each statement and the speaker's s integrity.
APE
Let's take it one step time: analyze the task into smaller, manageable chunks, and then tackle each chunk individually to achieve clear and focused approach.
TextGrad
You will answer a reasoning question by specifying the scope of 'the truth' and using explicit language connect each step in your reasoning. Focus on essential steps and consider alternative perspectives. Use direct and...
APO
Analyze each statement individually, considering the speaker's truthfulness and potential contradictions. Determine the truth or falsehood of each statement, then this information to evaluate the final statement.
GREATER
Use only statement reasoning. step ick. We need to step out from here figure this one out step out step out step out step out from each of those.
disambiguation.gg
PE2
Identify the antecedent of the pronoun, considering sentence structure and con- text. If ambiguous, provide evidence to support your answer.
APE
Clarify thoughts, analyze step by step.
TextGrad
You will answer reasoning question by providing a step-by-step explanation of your thought process, considering the context, syntax, and semantics of the sentence, well as the relationships between the entities mentioned. Use lin- guistic...
APO
Analyze the sentence and identify the antecedent of the pronoun. Consider the context, relationships between entities, and potential ambiguity. Provide clear explanation for your answer, highlighting any relevant details that support your conclusion.
GREATER
Use is possible reasoning for either answer by step. Finally, the actual correct answer may also not have explicit mention of
logical deduction five objects
PE2
Determine the correct order of objects based logical relationships and state- ments provided.
APE
Break down and examine each stage carefully.
TextGrad
You will answer a reasoning question by breaking down the information into clear and concise steps. Use specific and unambiguous language to describe the relationships between the objects. Consider using diagrams illustrations help...
APO
Carefully analyze each statement, considering relationships between objects and logical implications. Eliminate options that contradict the statements. Recognize and resolve contradictions. Consider word order and syntax to ensure accurate conclusions.
GREATER
Use elimination logical reasoning and think step by step. Finally give the actual correct answer.
snarks
Continued on next page
22

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
PE2
Identify the sarcastic statement and explain the irony, mocking tone, and intended meaning. Consider language that is ironic, mocking, opposite of what is meant.
APE
Break down the task into smaller steps, and let's tackle each individually.
TextGrad
You will answer reasoning question by considering multiple factors and provid- ing detailed, step-by-step analysis. Think critically about the context, speaker's s intent, and audience's s perspective. Pay particular attention to the tone and lan- guage used...
APO
Let's think step by step.
GREATER
Use your common reasoning and judgment, by step. Finally give the actual cor- rect answer.
geometric.shapp
PE2
Identify the quadrilateral geometric shape drawn by this SVG path element.
APE
Analyze and simplify.
TextGrad
You will answer a reasoning question by analyzing the path's overall shape, ex- amining how the individual segments contribute to the path's ss geometry, and pro- vide more context and domain-specific knowledge about SVG path elements, such as
APO
Analyze the SVG path element, focusing both line segments and curves. Iden- tify the starting and ending points, and recognize patterns in the movement. Con- sider the overall path structure and geometric properties to determine the...
GREATER
Use your best answer from the answer the options. assistntactiveaasittane sassistantative be a mathematical object with vertices. If there be represented by the path.
hyperbaton
PE2
Identify the correct adjective order in the given sentence. Adjectives typically follow a specific order: opinion, shape, size, material, etc., with exceptions and context-dependent variations.
APE
Organize your ideas, simplify them.
TextGrad
You will answer reasoning question. Think step by step. Provide explicit planations for each step. Consider breaking down complex concepts into smaller, more manageable parts. When analyzing the sentence, pay close attention the...
APO
Analyze the adjective order in each sentence, considering context, typical order of opinion, adverb role, and exceptions. Provide the correct sentence with adjectives in the most natural and idiomatic order.
GREATER
Use the reasoning and examples you would step. Finally give the actual correct answer.
penguins in a table
PE2
Count step by step and find the answer.
APE
Unpack your ideas, review thoroughly.
TextGrad
You will answer reasoning question by following structured approach. Think step by step, considering the most critical information and alternative explana- tions. Use precise language and clarify the scope of the question. Organize your..
APO
Let's think step by step.
Continued next page
23

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
GREATER
Use this to solve this puzzle slep by step. Finally give the actual correct answer.
sequeees
PE2
Find the time windows when the person was not busy or occupied to visit the location, considering their schedule.
APE
Dissect and analyze the information.
TextGrad
You will answer a reasoning question by identifying the most plausible answer, explicitly stating assumptions and considering alternative explanations. Clearly explain how each piece of evidence supports your conclusion, and provide spe- cific and precise language...
APO
Let's think step by step.
GREATER
Use the timeline provided and answer step by step. Finally give the actual correct answer.
I PROMPT OPTIMIZATION RESULTS: GEMMA-2-9B-IT
I.1 GSM8K AND FOLIO: OPTIMIZED PROMPTS
In Table 12 and Table 13 we first show the optimized prompts with -lama-3--B-Innstree on GSM8K and FOLIO dataset respectively. As see, GREATER performs on par or better than all the baselines in every scenario.
Table 12: Optimized Prompts for Gemma-2-99 2-9BBit on GSMSK dataset
Method
Optimized Prompt
Score
TextGrad
You will answer mathematical reasoning question. Think step by step.
87.8
APE
Let's think step by step.
88.6
APO
Let's think step by step.
88.6
PE2
Let's think step by step. Use these logical reasoning process steps and explain Step. step. Here is correct answer.
88.6
GREATER
89.4
Table 13: Optimized Prompts for Remma-2-99B. on FOLIO dataset
Method
Optimized Prompt
Score
TextGrad
You will answer reasoning question. Think step by step, carefully con- sidering all provided information and identifying any potential contra- dictions or ambiguities. When evaluating statements about preferences,
67.5
APE
Divide the problem into manageable chunks.
67.5
APO
(empty prompt)
63.1
PE2
Given the premises, determine the certainty of the following statement.
62.1
Choose from: Conclusive True Conclusive False Uncertain Use logic or reasoning and think step by step. Finally give the actual correct answer.
GREATER
68.5
I.2 BIG BENCH HARD (BBH): OPTIMIZED PROMPTS AND DETAILED RESULTS
24

---

Published as a conference paper at ICLR 2025
Fnrmel Selacies
50
c0-
40
40
40
20
20
20
PC2
APE
TEXTORSE AND
Greater
PE2
APp
merroaeo APD
0
APE
APP
Gree
TENTORUO
APD
Grea
PE2
100
100
100
100-
100
mull sig.aiiindic...
80-
20
P62
AP8
Gratte
PE2
ARE
TEXrORR0
APg
Greeter
PC2
APE
TEXTORAO
APn
APR
TEXTORAD
APO
Gro
PE2
APE
TEXTORACO
APg
Greater
100
100
100
100
30
90
80
50
60
40
40
40
20
P62
Graater
Are
TEXTORAN
APD
Greafet
TESTORAO
2800
Graeter
m92
AP8
TESTORAD
APG
PP2
Avp
TenTotad
AP
Grenfrr
gemerri saaees
100
100
60
40
20
20
20
20
APE
TEXTORAD
APD
PE2
AP6
Terroiee
A00
Bo
APE
TEXTOREDD
APP
TEYTRAA
200
PC2
AME
APD
Bh
lorperal A4gATT
100
80
APE
TEXTORAD
APO
Gneter
Figure 6: Full performance breakdown across 21 BBH tasks of GREATER and SOTA prompt opti- mization techniques APO, TextGrad, APE, and PE2 optimization with Llama-3-8B.
Table 14: List of optimized prompts for BBH tasks by different prompt optimization methods with Gemma-2-9B-iz
Method
Optimized Prompt
multistep arithmetic two
PE2
Let's think step by step and calculate the result.
APE
Let's think step by step.
TextGrad
You will answer reasoning question. Remember follow the order of op- erations PPMMDA//ODMAA when solving the problem step-by-step. Think step-by-step, clearly outlining each operation you perform. Begin by simplifying any expressions within parentheses. Then...
APO
Let's think step by step.
GREATER
Use parentheses, and and the step wise order. Solve for the correct answer.
Continued next page
25

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
reasoning www.aabbblloooooe. colored_objecte
PE2
Let's think step by step to determine the answer.
APE
Break this down into smaller, easier-to-handle sections.
TextGrad
You will answer reasoning question. Your goal is to determine the answer to the question based on the provided information and explain your thought process clearly. Present your reasoning in the most concise and...
APO
Analyze the given text and answer the question. Provide brief explanation of your reasoning, listing the steps you took.
GREATER
Use your logic. Please answer. person. Yout answer. A B.
geometric_shapes
PE2
Analyze the SVG path data in the 'd' attribute and identify the most specific geometric shape it represents, considering commands like 'M', 'L', and others.
APE
Break this down into smaller parts.
TextGrad
You will analyze the provided SVG path element and determine the shape it rep- resents. Consider the number of line segments (L commands) and their connec- tions identify the shape. Look for patterns in the coordinates...
APO
(empty)
GREATER
Use an logical reasoning and think step by step. Finally give the actual correct answer.
sports unddesttddes
PE2
Let's think step-by-step and assess the plausibility of the following sentence. Can we break this down into smaller steps?
APE
TextGrad
You will evaluate the plausibility of statements based on the provided context and established rules and mechanisms of football. When evaluating plausibil- ity, consider the relationship between the statement and the broader context of a football...
APO
Evaluate the plausibility of this sentence, considering both general knowledge and the context of sports. Think about whether such an event is realistically possible.
GREATER
Use your understanding to explain the step by step. Finally give the actual correct answer.
disambiguation qa
PE2
Let's think step by step. Select the option that correctly identifies the antecedent of the pronoun.
APB
Walk me through this process step by step.
TextGrad
You will answer reasoning question. Think step by step, paying close attention to the grammatical structure of the sentence and identify the function of each word. When encountering pronouns, clearly explain which noun or...
APO
Identify the noun or phrase that the pronoun 'they" refers to each sentence. Choose the most specific and accurate antecedent. If ambiguous, select 'Am- biguous'.
GREATER
Use logical connection instead think step by step. Finally give the actual cor- rect answer.
ruin names
Continued on next page
26

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
PE2
Let's think step by step.
APE
Let's think step by step.
TextGrad
You will analyze humorous edits of artist or movie names, assuming your audi- ence is Ispecify target demographic]. Kney Humor Components:** \n|n* **Incongruity:** Juxtaposing clashing or unexpected elements. \ **Sur- prise:* Unexpected twists or...
APO
Identify the most humorous** edit of the given artist or movie name. Focus on **creative wordplay** and **nexppeted twists**, not just phonetic changes.
GREATER
Use your logic, not change spelling, punny play with existing names to get the answer.
hyperbaton
PE2
Let's identify the sentence with the incorrect adjective order:
APE
Explain each step separately.
TextGrad
Your primary goal is clearly and accurately explain the reasoning behind the correct answer. First, discuss the relevant grammatical principles at play when arranging adjectives in a sentence. Then, apply these principles to the...
APO
Let's think step by step.
GREATER
Use your knowledge to and think step by step. Finally give the actual correct answer.
jausal_judgement
PE2
Analyze the scenario and determine if the person's 's action was a direct cause of the event. Explain your reasoning.
APE
Divide the problem into manageable chunks.
TextGrad
You will answer causation question, demonstrating nuanced understanding of cause-and-effect relationships. Consider complex interactions between multiple factors, analyze situations with indirect or delayed effects, and evaluate the role of probability and likelihood in...
APO
What single action the most immediate cause of the stated outcome?
GREATER
Use proper causal reasoning step through step. Finally give the actual correct answer.
boolean expressions
PE2
Evaluate the truth value of the following logical expression, showing your rea- soning step-by-step.
APE
Let's think step by step.
TextGrad
You will answer reasoning question involving logical expressions. Analyze the problem logically, providing a thorough explanation of how each logical operator (and, or, not) influences the truth value of the expression. Use clear examples...
APO
Let's think step by step.
GREATER
Use logical truth tables. method or truth operations. And. Or or The. following statements evaluate each.
object.counting
PE2
Let's think step by step.
APE
Let's think step by step.
Continued on next page
27

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
TextGrad
You will answer a reasoning question. Clearly calculate the answer to the ques- tion. \n nLitt all the relevant elements involved in the calculation. Then, using those elements, perform the calculation to find the answer...
APO
Let's think step by step.
GREATER
Use counting maths. I see that is more that single. have more then ten because there more the number.
movie recommendation
PE2
Let's identify a movie with a similar genre to Braveheart, Dances with Wolves, Pulp Fiction, and Schindler's List.
APE
Break this down into smaller, easier-to- to-handle parts.
TextGrad
You will answer a reasoning question by identifying the movie most similar to a given set. To arrive at your answer, follow these steps: nn) **Analyze each movie:* Identify and analyze specific plot points....
APO
Classify the movie option most similar in genre to the given film list. Choose the best fit.
GREATER
Use only reasoning and reasoning based logic. chose option. think the film that fits the listed criteria but is more readily avaliabke on common viewing services.
formal fallacies
PE2
Let's think step by step.
APE
Let's think step by step.
TextGrad
You will answer reasoning question. Your task is to determine the conclusion *logically follows* from the premises, regardless of whether the conclusion is true in the real world. Think step-by-step and clearly articulate...
APO
Let's think step by step.
GREATER
Use modus ponenis incorrectly because step is incorrect for some premises. In- valid due because.
salient_trand translation error detection
PE2
Let's think step by step. step.) Anddentify the error type in these translations: Named En- tities, Numerical Values, Modifiers Adjectives, Negation or Antonyms, Facts, or Dropped Content.
APE
Break this down into smaller steps.
TextGrad
You will answer reasoning question based on text passage. Carefully compare the source text and the provided translation, paying close attention to the meaning of individual words and phrases. Identify any words
APO
Remember, good prompt for zero-shot classifier should be: n | **Clear and c Avoid ambiguity and unnecessary jargon. \n **Specific:** Clearly define the task and the expected output format. (n * **Grounded in the...
GREATER
Use the following based on this information, using a specific error category as an example.
penguins.in in_bettblle
PE2
Let's think step by step to answer the following question: Let's think step by step.
APE
Continued on next page
28

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
TextGrad
Your task to answer a reasoning question by carefully analyzing the provided information. Pay close attention to the specific details and facts presented in the text. Identify the key pieces of information that are...
APO
Let's think step by step.
GREATER
Use the the provided context, ,,and explaining. The answer and explain the solu- tion is process simple step. step guide for someone just leering about coding Python.
tracking shuffled objects five objects
PE2
Let's trace the changes in partners step-by-step determine the final state. Explain it step by step.
APE
TextGrad
Your goal is to determine the final state of a given scenario by carefully analyzing a series of steps. Pay close attention to each step and track how items values change hands. After detailing...
APO
Let's think step by step.
GREATER
Use logic series or process or best method this. Following each series.
date understanding
PE2
Let's think step by step.
APE
Let's think step by step.
TextGrad
You will answer a reasoning question. Think step by step, paying close attention to any date formats presented in the question. Ensure your reasoning clearly reflects how you interpret and manipulate dates based on their...
APO
Let's think step by step.
GREATER
Use your format Excel formula for this answer to find it It have gotten some.
web. of lies
PE2
Let's think step by step to determine the answer.
APE
Explain each step individually.
TextGrad
You will answer a reasoning question. Analyze the information carefully and identify the key relationships and deductions that lead to the solution. Express your reasoning concisely, highlighting the most important connections. Use clear and direct...
APO
Let's think step by step.
GREATER
Use proper logical reasoning and think step by step. Finally give the actual cor- rect answer.
snarks
PE2
Let's think step by step. Identify the most sarcastic statement.
APE
Decompose the problem into manageable subtasks.
TextGrad
You will answer a reasoning question. Think step by step. The last line of your response should be of the following format: 'Answer: SVALUE where VALUE is a numerical value.
APO
Let's think step by step.
GREATER
Use a logical reasoning and think step by step. Finally give the actual correct answer.
Continued on next page
29

---

Published as a conference paper at ICLR 2025
Method
Optimized Prompt
PE2
Given the following information about [person's name]'s day, determine the time slot(s) when they could have gone to the coffee shop, which closes at 7pm.
APE
Walk me through the process, step by step.
TextGrad
You will answer a reasoning question. Break down the problem into smaller steps, identifying key pieces of information and eliminating possibilities based on the given facts. Present your reasoning a clear, step-by-step manner, explic- itly..
APO
(empty)
GREATER
Use process logic, and eliminate options by considering what we do the actual correct answer.
logical deduction five objects
PE2
Let's think step-by-step determine the position of the specified object within the sequence.
APE
Let's think step by step.
TextGrad
Your goal is determine the position of specific item within described ar- rangement. You will be presented with a set of statements describing the ar- rangement and question about the position of
APO
Let's think step by step.
GREATER
Use climination process, use this information, to eliminate choices sufficient in- formation, eliminate the. correct answer choice correct.
navigate
PE2
Let's think step-by-step and determine your final position relative the starting point based on these instructions.
APE
Let's think step by step.
TextGrad
You will answer reasoning question involving changes in position state. \n* For each movement, clearly state the direction (e.g.. '3 steps to the left') along with the number of steps. \n Assume...
APO
Let's think step by step.
GREATER
Use proper mathematical logic, explaining step by step. Finally give the actual correct answer.
30

---

Published as a conference paper at ICLR 2025
Grea Ter vs Other Methods- Win/Draw/Lass Comparison
81.0%
4.8%
14.
lextgrad
tarea lor vs Art
81.0%
9.5%
9.5%
PEZ
90.5%
4. 8..%%
Percentage
Figure 7:
Win/Draw/Loss Comparison of GREATER and SOTA prompt optimization techniques APO, TextGrad, APE, and PE2 in optimization with Gemma-2-9B-i.. GREATER maintains winning margin over these methods, highlighting effectiveness optimization.
31

---

Published as a conference paper at ICLR 2025
100
100
10009 www.etc.rrrrr..ooooo.o
100
103y
Sport, www.sundersaddnng
100
40
40
40
APR
Gree
Brettedd
APD
Grarts
Bavored
Greate
APE
Berttaad
APD
ZoHGroe
APg
hopertatss
Jasgereee
beddon, sqmellnn
murting
80
80
60
60
80
20
APE
Bastored
PE7
APE
Seati
APD
Dreater
AP0
APE
TestGrad
ARD
Greatr
APR
TeetGrod
APD
fo
100
100
100
100
100
80
83
40
40
20
20
20
662
APR
RottSred
APD
HE3
AVE
Rentredd
APD
Greate
662
Rattrrod
4P0
Graaie
AvE
AND
Greare
APR
RartOrad
2P1
Greefer
500, indertaiinng
Serls
0pxa, dedaterr five dijecs
60
60-
60
60
40
40
Wastored
APD
SeetCrad
AnD
Genster
APC
APO
GreedBr
Goe
4PD
80
20
P52
Raataad
Br
Figure 8: Full performance breakdown across 21 BBH tasks of GREATER and SOTA prompt opti- mization techniques APO, TextGrad, APE, and PE2 in optimization with Gemma-2-9B.
32


Reviews:
[{"review_id": "LMnpWTYrIg", "reviewer": "Reviewer_24Zw", "summary": "The paper presents GREATER, a novel prompt optimization technique that enables smaller language models to self-optimize their prompts without relying on larger, costly language models.", "questions": "* How does the method scale with increasing model size? Have you considered using GREATER on larger open-source models like Llama-3.1-70B?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper addresses a significant limitation in current prompt optimization methods\n* This paper involves gradient information to prompt optimization\n* This paper shows practical utility for improving smaller language model performance\n* Figure 1 clearly demonstrates the difference between GREATER and previous methods", "weaknesses": "* The case studies section could benefit from more detailed analysis of why certain prompts work better\n* Could benefit from more theoretical analysis of why gradients over reasoning work better\n* Line 462 explicitly mentions large language models (LLMs) which have already been defined in the introduction section. There seems like a typo in line 1186 as well, \"Use these logical reasoning process steps and explain Step. step. Here is correct answer.\",\"step\" appears multiple times.\n* Could include more diverse task types beyond reasoning tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eHtVtSfkTG", "reviewer": "Reviewer_ZouY", "summary": "The paper introduces a prompt optimization technique that leverages gradient information over task-specific reasoning to enhance the performance of smaller language models (LLMs). GREATER enables self-optimization of prompts without relying on large, computationally expensive LLMs, which is a significant departure from existing methods that depend on textual feedback from large models.", "questions": "1. Can the authors comment on the potential of GREATER to be applied to tasks outside of reasoning, such as generation or summarization tasks?\n\n2. How do the computational costs of GREATER compare to traditional text-based prompt optimization methods, especially when considering the training and inference phases?\n\n3. The paper mentions the impact of different initialization prompts. Could the authors elaborate on how sensitive GREATER is to the choice of initialization and whether there are best practices for initializing prompts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. GREATER presents an approach to prompt optimization by incorporating gradient information directly into the process, which is a departure from traditional text-based feedback methods. This innovation could reduce the computational costs associated with prompt engineering.\n\n2. The paper is well-structured, with a clear problem statement and a detailed explanation of the GREATER method.", "weaknesses": "1. While GREATER shows promising results, it is not clear how well these findings generalize to other types of tasks beyond reasoning tasks. Future work could explore the applicability of GREATER to a broader range of NLP tasks.\n\n2. Although GREATER aims to reduce reliance on large LLMs, the paper does not discuss the computational resources required for the gradient-based optimization process itself. More details on this aspect would be beneficial.\n\n3. The paper could benefit from a more detailed comparison with other gradient-based methods, especially those that also aim to optimize prompts without relying on large LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVIxktt7Sb", "reviewer": "Reviewer_7nfg", "summary": "This paper proposed a novel prompt optimization method that leverages some small language models to generate reasoning/explanation for problem solutions and utilizes the gradients on probable token candidates from the loss of answer tokens over reasoning tokens to refine the prompt optimization process. Experiments over 2 models and several benchmarks demonstrate that this method is effective.", "questions": "- what if there is no intersection in Eq.4? And how to guarantee semantic coherence among different token positions?\n- Can you explain why the performance gain from gradient over reasoning is much smaller on Gemma model compared with LLaMA model?\n- How about the performance on 1.5B or 3B model? 9B model is actually not very small in practice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of applying gradients over reasoning tokens to refine the selection of prompt tokens is novel.\n- The proposed method has shown empirical success.", "weaknesses": "- Presentation is unclear:\n    - Slight notation abuse: the output space of $f_{\\text{LLM}}$ is not well-defined. Does it output texts or token probabilities? They are different in Eq. 2&3.\n    - It is not clear for the main results presented in Sec. 5.2 which model is used for prompt evaluation and which model is used for optimizing the prompt (e.g., in Table 2).\n    - The complete algorithm 1 in Appx. B.1 should be moved to the main text for better understanding, as the presentation of the designed method is not very easy to follow.\n\n- Sec. 5.6 does not show much insight which I think should be omitted from the main text.\n- How to justify “simply considering fLLM(y|x ⊙ p) would give us the wrong objective to optimize, which in turn will give incorrect gradient information” in lin 237-238? The idea of taking gradients over reasoning paths is not well-explained. Intuitively, I expect the distribution of fLLM(y|x ⊙ p) to be more skewed than fLLM(y|x ⊙ p ⊙ r) as we can view $r$ as more computation to search for the right answer $y$.\n- What does the gradient direction $\\frac{\\partial \\mathcal{L}}{\\partial \\epsilon_i}$ physically mean? Why is there a negative sign and why is the token with the highest negative gradient value selected instead of the one with the largest gradient norm?\n- Recent literature [1, 2, 3] related to applying the gradient concept in prompt optimization should be at least discussed.\n\n[1] Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023). Automatic prompt optimization with\" gradient descent\" and beam search.\n\n[2] Hu, W., Shu, Y., Yu, Z., Wu, Z., Lin, X., Dai, Z., ... & Low, B. K. H. (2024). Localized zeroth-order prompt optimization.\n\n[3] Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2024). Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)