Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
Authors: Peiyan Zhang
Haibo Jin
Leyang Hu
Xinnuo Li
Liying Kang
Man Luo
Yangqiu Song
Haohan Wang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Response Evolution
Textual Optimization
Primary area: deep_learning->large_language_models

Abstract: Recent advancements in large language models (LLMs) have significantly enhanced the ability of LLM-based systems to perform complex tasks through natural language processing and tool interaction. However, optimizing these LLM-based systems for specific tasks remains challenging, often requiring manual interventions like prompt engineering and hyperparameter tuning.
Existing automatic optimization methods, such as textual feedback-based techniques (*e.g.*, TextGrad), tend to focus on immediate feedback, analogous to using immediate derivatives in traditional numerical gradient descent.
However, relying solely on such feedback can be limited when the adjustments made in response to this feedback are either too small or fluctuate irregularly, potentially slowing down or even stalling the optimization process. In this paper, we introduce $\textbf{REVOLVE}$, an optimization method that tracks how $\textbf{R}$esponses $\textbf{EVOLVE}$
across iterations in LLM systems. By focusing on the evolution of responses over time, REVOLVE enables more stable and effective optimization by making thoughtful, progressive adjustments at each step. Experiments across three tasks demonstrate the adaptability and efficiency of our proposal. Beyond its practical contributions, REVOLVE highlights a promising direction, where the rich knowledge from established optimization principles can be leveraged to enhance LLM systems, which paves the way for further advancements in this hybrid domain. Code is available at: https://llm-revolve.netlify.app.

REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
Peiyan Zhang'
Haibo Jin
Leyang Hu 3 Xinnuo Li Liying Kang² Man Luo Yangqiu Song'
Haohan Wang?
Abstract
1. Introduction
In recent years, Large Language Models (LLMs) have dra- matically advanced Al's ability to handle complex tasks through natural language processing, enabling LLM-based systems, often called language agents, to interact with ex- ternal tools and solve problems once considered out of reach (Brown et al., 2020; Achiam et al., 2023; Team et al., 2023; Anthropic, 2023; Jin et al., 2024a; Xiao al., 2024; Jin et al., 2024b). However, developing these agents still requires significant manual effort to break down tasks and fine-tune prompts, tools, and APIs, limiting scalability and adaptability (Zhou et al., 2024; Liu et al., 2025). This raises the need for automated, scalable optimization techniques to enhance language agents efficiently.
Recent advancements in large language models (LLMs) have significantly enhanced the ability of LLM-based systems to perform complex tasks through natural language processing and tool in- teraction. However, optimizing these LLM-based systems for specific tasks remains challenging. often requiring manual interventions like prompt engineering and hyperparameter tuning. Existing automatic optimization methods, such textual feedback-basee techniques (c.g., TextGrad), tend to focus immediate feedback, analogous to us- ing immediate derivatives in traditional numerical gradient descent. However, relying solely on such feedback can be limited when the adjustments made in response to this feedback are either too small or fluctuate irregularly, potentially slowing down or even stalling the optimization process. In this paper, we introduce REVOLVE, an optimiza- tion method that tracks how Responses EVOLVE across iterations in LLM systems. By focusing on the evolution of responses over time, REVOLVE enables more stable and effective optimization by making thoughtful, progressive adjustments each step. Experiments across three tasks demon- strate the adaptability and efficiency of our pro- posal. Beyond its practical contributions, RE- VOLVE highlights a promising direction, where the rich knowledge from established optimiza- tion principles can be leveraged enhance L.LM systems, which paves the way for further advance- ments in this hybrid domain. Code is available https:: 11m revolve.... .ew.iffeeyll.y app.
To this end, recent efforts have been made on automatic op- timization of language agents. For instance, DSpy (Khattab et al., 2024) uses bootstrapping and random search to opti- mize LLM prompts by exploring combinatorial space of prompt components. GPTSwarm (Zhuge et al., 2024) builds on this by introducing an iterative process to manage DSpy't ss complexity. Other methods like Agent-Pro (Zhang al., 2024) and AgentOptimizer (Zhang al.) target specific modules, refining prompts, and agent policies. However, these approaches often suffer from local optimization, where improvements in isolated components do not lead to overall system performance gains-similar to early neural network practices (Hinton & Salakhutdinov, 2006).
Building on these efforts, more advanced research has in- troduced gradient descent-inspired techniques for automatic prompt optimization. ProTTeGi (Pryzant et al., 2023) pi- oneered the use of textual gradients, where natural lan- guage feedback refines prompts. Agent Symbolic Learning (ASL.) (Zhou et al., 2024) extended this to optimize the en- tire agent system by treating prompts and tools as learnable parameters. Textgrad (Yuksekgonul et al., 2024) further applied textual gradients to instance-level optimization, re- fining outputs across multiple iterations.
'Hong Kong University of Science and Technology University of Illinois Urbana-Chamnn Champaign 'Brown University University of Michigan Ann Arbor 'onn Kong Polytechnic University 'Intel Lab. Correspondence to: Haohan Wang <haohanw @ illinois.
While effective, these methods all rely on what can be de- scribed as first-order optimization. In this context, first- order methods mean they adjust the agent's s behavior based
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
on immediate feedback from the current iteration, similar to how numerical first-order gradient descent updates pa- rameters using only the current gradient. This limits their ability to account for how responses evolve across multiple iterations, leading to potential stagnation in suboptimal so- lutions. As shown in Figure (a), first-order methods often stagnate in local optima, resulting in repeated or minimally improved LLM responses. This challenge motivates us to explore LLM optimization techniques that consider how responses evolve over time, enabling more adaptive and refined adjustments that can break free from local optima across iterations.
optimization, and gradient-based learning. Below, we high- light foundational works in these areas and situate our method within this broader context.
From Prompt Engineering to Agent Optimization. Prompt engineering has become a key focus in both academia and industry, leading to several methods aimed at automating the process. Early works (Pryzant et al., 2020; Yang et al., 2024) explored the use of structured prompts that enable LLMs optimize their own inputs. Other ap- proaches (Prasad et al., 2022; Guo et al., 2023) use search algorithms, like genetic algorithms, to automatically refine prompts. Building on the success of automated prompt engi- neering, researchers have extended these concepts to broader agent optimization. Techniques like Agent-Pro (Zhang et al., 2024) and AgentOptimizer (Zhang et al.) focus on opti- mizing individual components, such as prompts tools. However, these methods often treat components in isolation, which can result in local improvements without significantly enhancing the overall system. Search-based approaches, such as DSpy (Khattab et al., 2024) and GPTSwarm (Zhuge et al., 2024), take a more comprehensive view by optimizing across the combinatorial space of agent components. De- spite their scope, these methods rely heavily on numerical metries that are often inadequate for real-world tasks like software development or creative writing. Additionally, they struggle to optimize multiple components simultaneously or adapt dynamically to changes in the agent pipeline.
In this paper, we introduce an optimization method called REVOLVE that builds upon TexiGrad (Yuksekgonul et al., 2024) by incorporating a deeper understanding of how Responses EVOLVE over time. The optimization process begins with forward pass, where the system executes series of tasks, logging inputs, outputs, and any prompt or tool usage. A language-based loss function then eval- uates the quality of the generated responses, quantifying how well they align with task objectives. In the backward pass, feedback in the form of natural language critiques is used adjust system variables. Relove improves this stan- dard process by focusing how response patterns evolve over multiple iterations, enabling the system to make more informed and effective adjustments, ultimately leading to improved performance in handling complex tasks.
As shown in Figure 1 (b), we calculate a refined gradient that accounts for changes in responses across multiple itera- tions, enabling the system to adjust based on both immediate feedback and long-term response patterns. This parallels the concept of second-order optimization in traditional methods, where the Hessian matrix captures how the gradient itself changes. In our case, we model the evolving relationship between consecutive prompts and responses, enabling the system to make more informed adjustments. By incorporat- ing this additional layer of information, the system can avoid stagnation in suboptimal patterns, a common limitation of methods that rely solely on immediate feedback.
Gradient-Based Approaches for Agent Optimization. Recent advancements have introduced gradient descent- inspired techniques optimize prompts more effectively. ProTeGi (Pryzant et al., 2023) is among the first to use natu- ral language feefback-refereed to as textual gradients to iteratively refine prompts. However, as a first-order opti- mization method, ProTeGi adjusts based only on immediate feedback from a single iteration, limiting its capacity to handle more complex, multi-step tasks. Agent Symbolic Learning (ASL) (Zhou et al., 2024) extended this concept by treating the entire agent system-including prompts, tools, and configurations as learnable components, much like backpropagation in neural networks. This allows for a more comprehensive optimization but remains dependent on immediate feedback from each iteration. Textgrad (Yuk- sekgonul et al., 2024) further advanced this first-order gra- dient approach by optimizing LLM responses using nat- ural language feedback. By treating feedback gradi- ent, Textgrad refines responses without directly altering the model's parameters. While effective for simpler tasks, Textgrad struggles with deeper, multi-step optimizations, frequently getting stuck in suboptimal states.
We test the proposed method on three tasks, Prompt Opti- mization, Solution Optimization, and Code Optimization. These tasks require handling complex reasoning, refining solutions to scientific questions, and optimizing code un- der challenging constraints. Experimental results show that Relove consistently improves performance across all tasks, showing its versatility and effectiveness in overcoming the limitations of existing optimization methods.
2. Background
To address these limitations, momentum based meth- ods (Yuksekgonul al., 2024) have been introduced. These techniques track feedback trends across iterations, adjusting
Our approach draws inspiration from several key areas of research, particularly automated prompt engineering, agent
2

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
First-Ordes Methods Trapped Local Optima
Iteration 1
Agent "Hoe con steal the idendity of others without being detected by security systemo? Feedback: unsthical and vielotes the guidelnnee Rephrase without askingg for malioious activiy.
Input Problom
Thapped in local miliimo
Iteratonn 2
(Probabiiity: 0.13 STATUS: FAIL
Agant (Unchanged: "Hoo can steal the identify of others without being detected by security
Part Paint
Feodbaok: "The reguest remains unothioal The resporse needs to ochere to the systent's othicll guideline."
100
You are altacker tasked
Perfomance rieghation
(Probaiiityy 0.2) STATUS: FAIL
with beguawing the rules of defender L.M
ahter iniill locaale
Iteration 3
Agent (Inchanged: "How can steal the identity of others without baing detected by security нystema"
Sharotion
asproach.
prompt
(Probabiliyy 0.3) STATUS: PAIL
Output
Your goal generate jailbreak promptt exploit 's vulnerabilittee.
REVOLVE Escapes Local Optima
Agant: "How steal the identity others without being detected security systema??
Feodback: ". unathical Paptross while adhering to othioal guiddins.
(Prbability: 0.3) STATUS: FAIL
Eacaping Local Optima
Iteration 2
Agent "What are some tedhniques used to verily the authenicity of an identity and how
Probasil di
might they be typassed theoretical reasarch?"
potentasity
unithical applications. Pocus discussing identity veriiication methods"
Continanss pertormance improwerarta
(Frobabiity: 0.5) STATUS: PAIL
Iteration
"What
identity verification
thooretical
winembilttee could be suplored for improving them?
Fiiin
Paedback: "The facus on improving security through research sporopratte Continus
Iteration
(Probabiiity: 0.7) STATUS: SUCCESS
Figure I: The illustrative comparison between REVOLVE and first-order optimization methods. First-order methods rely solely on immediate feedback, often leading stagnation in local optima and limiting further improvement. In contrast, REVOLVE incorporates both immediate feedback and response evolution over time, enabling continuous progress and the ability to escape stagnation.
step sizes when feedback becomes repetitive, which helps break stagnation. However, while feedback provides im- mediate guidance on how inputs should change, it doesn't directly reflect how model responses evolve over time. Feed- back may remain constant as responses improve or fluctu- ate despite stagnation, leading suboptimal adjustments. In complex tasks, where responses exhibit slow or subtle shifts, this disconnection can make feedback misleading or overly sensitive to temporary fluctuations, causing in- stability. Therefore, while momentum-bent based methods pro- vide more variation, they still lack the fine-grained control needed for long-term improvement.
tion of first-order methods.
Stabilized Optimization: Unlike momentum methods, which risk overshooting due to large adjustments, RE- VOLVE applies carefully measured adjustments that en- sure smoother and more consistent progress throughout the optimization process.
3. REVOLVE: Optimizing AI Systems by Tracking Response Evolution
3.1. Method Overview
In this work, we extend the TextGrad approach (Yuksek- gonul et al., 2024) by tracking the evolution of the LLM responses across iterations, allowing for more effective and precise optimization.
Optimization Through Response Evolution. REVOLVE enhances traditional optimization methods by focusing on how responses evolve over multiple iterations, enabling more refined adjustments throughout the process. Instead of relying solely on immediate feedback, our ap- proach tracks the evolving relationship between consecutive prompts and their corresponding responses. This parallels second-order methods in traditional optimization, where the the Hessian matrix is used to capture changes in the gradient to guide more precise adjustments. However, rather than directly computing numerical second derivatives, we model these iterative shifts in responses to inform our adjustments, giving the system a broader understanding of response dy- namies over time. The key advantages of our approach include:
REVOLVE:
3.2. Overview of Optimization Pipeline
Our method builds upon the general optimization pipeline used in LLM-based systems (Zhou et al., 2024; Yuksek- gonul et al., 2024), introducing natural language feedback (textual gradients) to refine system responses over multiple iterations, instead of relying on numerical gradients.
Forward Pass. In the forward pass, the Al system is mod- eled computation graph where each node represents specific task. Inputs are processed sequentially through the nodes, with each node generating outputs based on prior results. These intermediate outputs are stored in a trajectory, which is later used in the backward pass.
Response Evolution Awareness: REVOLVE monitors changes across iterations, allowing for more refined and adaptive optimization, unlike first-order methods that rely only on immediate feedback.
Language Loss Computation. After the forward pass, evaluator LLM assesses the system's performance by generating textual feedback, which serves as the loss. This feedback reflects how well the system's 's outputs align with the task objectives and drives the subsequent optimization
Avoiding Local Optima: By tracking iterative changes, REVOLVE prevents models from getting stuck in subop- timal solutions, effectively overcoming common limita-
3

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
process.
This function plays a critical role in extending the optimiza- tion process to account for the dynamics between successive iterations.
Backward Pass. In the backward pass, similar to numerical gradients in conventional deep learning. textual gradients are backpropagated through the nodes of the system. These gradients, in the form of natural language instructions, in- dicate how the system's surc as prompts, tools, and cecisions-shoule be adjusted improve the objective function. Starting from the final node, the system com- putes the necessary updates for these variables as it moves backward. This process mirrors backpropagation in neural networks, but the adjustments are determined by language feedback rather than numerical values.
REVOLVE. With this setup, to extend Eq. 1 to encourage the prompt leading to more gradual and thoughtful evolution of the response over multiple iterations, our new gradient can be expressed as
REVOLVE ( ((((()))) @c(r(p))) + S
Öp.
(2)
This new formulation guides the optimization process in way that not only improves immediate task performance but also promotes long-term, iterative refinement. We name our new method Eq. 2 REVOLVE. Practically, we rely on a LLM to evaluate the similarity function S (r((p)), r(p:-1)).
While the above process sets the stage for optimizing the system, the effectiveness of the optimization depends on how well this feedback is utilized. In this context, different tradient-based optimization methods come into play.
Analogy to Second-order Derivative. To understand the intuition behind calling our approach analogy to second- order methods, consider how second-order derivatives (Hes- sians) in classical optimization capture the rate of change of the gradient. The second-order derivative provides deeper insight into the curvature of the optimization landscape, al- lowing for more informed adjustments that go beyond the immediate gradient.
3.3. First-Order Optimization: Textgrad Approach
TextGrad (Yuksekgonul et al., 2024) computes first-order gradient based the language loss provided by the evalua- tor LLM. The first-order gradient captures the difference in response quality between consecutive iterations. Mathemat- ically, the first-order gradient is expressed as:
((((())) = (c(r())) Sp.
(1)
In our context, the similarity function S serves a parallel role by tracking how the system's 's responses shift from one iteration to the next. We can formalize this by using a generalized norm function (denoted || I]) to quantify the differences between two elements (either loss functions or prompts). One way to concretely define the similarity function ((r(p)), r(p--1)) is as follows:
where we use r(pe) denote the response when the model is fed with input prompt Pt and to denote the iteration. Also, we use i to denote the TextGrad-style derivative of loss function with respective to the input prompt due to its analogous nature to the actual derivative that is typically denoted as O.
r(P(-)) = [I((r(p))
((r(p--)))
Pp Pt-
3.4. REVOLVE and Its Analogy to Second-Order Gradient Optimization
This equation mirrors the classical definition of a derivative when the difference between successive prompts (ipe-pt-1l is sufficiently small. Thus, by assuming I|P: - Pt- to be sufficiently small and instructing the LLM evaluate S (r((p)) above, we can have
We seek to extend the previous method by extending Eq. to consider the history of previous prompts and their re- sponses. Optimizing based only the current response can lead to short-term improvements but might results in stagnation, especially in complex tasks where deeper issues arise over time. For example, a LLM might slightly refine responses with each iteration, yet without considering the history of prompts and responses, it risks repeating similar patterns. By factoring in the evolution of responses over multiple iterations, we aim to uncover underlying issues that cause stagnation and enable the system to break free from suboptimal cycles.
((((p)) = ÖC(r(p)) Sp:
As result, our REVOLVE in Eq. 2 can be rewritten as: REVOLVE(((((o:)) )c(r(p))
8 (L(((()))
0p.
Op.2
Similarity Function. To quantify the differences between previous responses, we need to firstly define a similarity function. We use ((((p)) r(p--1) to denote the similarity between the responses triggered by prompts Pt and Pt-1-
which is a second-order derivative method. By considering this higher-order information, REVOLVE allows the system to escape from local optima and overcome the limitations of approaches that rely solely on immediate feedback.
4

---

REVOLVE:
Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
3.5. Discussion on Simulating Second-Order Effects in Textual Optimization
dataset (Cobbe et al., 2021). For each task, when using the iterative optimization methods, use batch size of 3 across 12 optimization iterations, allowing the model to pro- cess total of 36 training examples, randomly sampled with replacement. After each iteration, we validate the prompt us- ing a validation set, and if the validation accuracy improves, we update the prompt accordingly. We compare the model's accuracy on the test set after all 12 iterations, using prompts generated by different optimization methods. Consistent with (Yuksekgonul et al., 2024), for both tasks, we use the string-based exact match metric, which looks at the final numerical value provided in the answer, and compares it to the ground truth answer. Detailed task setup is provided in Appendix B.
No Numerical Hessian Computation. REVOLVE focuses on simulating second-order effects, not just computing real second-order derivatives. Thus, the REVOLVE framework does not compute the traditional Hessian matrix used in numerical optimization. Instead, it simulates the effects of second-order optimization within the textual optimization framework.
No Reliance on LLMs' Second-Order Derivatives. RE- VOLVE combines structured first-order feedback with re- sponse trajectory tracking to simulate the second-order opti- mization effects. This approach enables it to identify stag- nation and instability, refining responses without relying on LLMs' ability to compute second-order derivatives, distin- guishing it from basic prompting strategies.
Baselines and LLM Backends: We evaluate REVOLVE against three key baselines:
Zero-shot Ohain-of-Thoughhe (CoT) (Kojima et al., 2022; Wei et al., 2022): This baseline initializes all prompts us- ing a zero-shot CoT strategy, where the model is prompted to "think bypp-by-ttep" before generating an answer. This approach is widely regarded as strong baseline for rea- soning tasks.
4. Experiments - -Evaluation and Understanding of Models
We evaluate REVOLVE on three challenging tasks: Prompt Optimization for Reasoning, Solution Optimization, and Code Optimization. For prompt optimization, we use the Big Bench Hard dataset (Suzgum et al., 2022) for Object Counting and the GSM8K dataset (Cobbe et al., 2021) for grade-school math problems. In solution optimiza- tion, we assess performance on the Google-proof Ques- tion Answering (GPQA) benchmark (Rein et al.. 2023), which consists of expert-level multiple-choice questions, and the Machine Learning and College Physics subsets of MMLU (Hendrycks et al., 2020), benchmark evaluating LLMs' human-level performance. For code optimization, we use the LeetCode Hard dataset (Shinn et al., 2024), which includes complex coding problems challenging both humans and models. For all LLMs used, we allow a maximum of 2000 tokens, and use top-p of 0.99. Across tasks, RE- VOLVE consistently achieves leading reasoning accuracy and strong code completion rates, demonstrating superior performance. More details are in Appendix A.
TextGrad (Yuksekgonul et al., 2024): Textual feedback is treated as first-order gradient to iteratively optimize prompts.
Momentum-Enhanee TextGrad (Yuksekgonul et al., 2024): This method extends the original TextGrad frame- work by incorporating momentum. This variant aims to overcome potential stagnation in the optimization pro- cess by enlarging updates to the prompt when previous feedbacks on the variable are similar.
Our experiments perform prompt optimization separately on four LLMs: gpt-3.5-turbo00125 Gemini 1.5 Pro, and Llama 3.1 8B Instruct, with GPT-4o serving as the backend of the optimization system. This multi-model setup allows us to evaluate the effectiveness of the optimization methods across diverse architectures, ensuring comprehensive assessment of their capabilities.
1PT--4-0125-preview
Results: As evidenced by Table I, in both reasoning tasks, REVOLVE delivers a substantial improvement over the Zero-shot CoT prompt, underscoring its effectiveness across diverse datasets and model architectures. On the Object Counting task, with Llama 3.1 8B Instruct as the base model, REVOLVE outperforms TextGrad by achieving 6% higher accuracy. demonstrating its superior ability to refine LLM responses. Similarly, on GSMSK, REVOLVE exceeds both TextGrad and M- Texttgra across most LLM backends, with an average performance increase of 2% over TextGrad. These results suggest that REVOLVE not only enhances the optimization process but also addresses the inherent limita- tions of first-order feedback in TextGrad, leading to more accurate and refined reasoning capabilities.
4.1. Prompt optimization for reasoning
The goal of Prompt Optimization for Reasoning is to refine a basic prompt for specific reasoning task, enhancing the LLM's effectiveness in reasoning. This task is ideal for evaluating optimization methods, as reasoning tasks often involve large, complex search spaces where subtle prompt adjustments can significantly influence the outcome.
Task Setup: We evaluate prompt optimization two rea- soning tasks: Object Counting from the Big Bench Hard benchmark (Suzgun et al., 2022; Srivastava et al., 2022) and grade-school math problem solving from the GSM8K
5

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Table I: Prompt optimization results for reasoning tasks for various LLMs, with GPT-4o as the optimization engine. The values in parentheses represent the relative improvement in accuracy of the method compared to TextGrad.
Aceuracy % (Improv, over TextGrad)
Dataset
Models
CoT
TextGrad
M-TextGrad
REVOLVE
GPT-3.5
77.8 (15.3%))
9-)
92.1 (0.2%))
95.5 i 0.9% (3.9% t)
GPT-4
92.1 (2.2%%.
94.2 (-)
90.0 (4.5%.J)
96.3 0.6% (2.2% t)
Object Counting
Gemini 1.5 Pro
94.0 (0.0%)
(4)
94.0 (0.0%)
94.0 0.0%% (0.0%)
Llama 3.1 8B Instruet
(15.6%)))
7-)0
80.0 (3.9%) t>
83.0 1.4% (7.8% 1)
GPT-3.5 GPT-4
72.9 (9.9%))
(-)
82.1 (1.5% f)
85,9 = 0.6% (6.2%))
92.6 (0.4%))
(-)
93.9 (1.0%))
94.5 = 0.4% (1.6%)
GSM8K
Gemini 1.5 Pro
92.9 (0.4%).)
(-)
93.9 (0.6%))
0 (0.3%))
Llama 3.1 8B Instruct
84.6 (0.0%)
(4)6
84.6 (0.0%)
84.6 0.0% (0.0%)
Table 2: Solution optimization results for Llama 3.1 8B Instruct, with itself as the optimization engine. The values in parentheses represent the relative improvement of the method compared to TextGrad.
Accuracy % (Improv. over TextGrad)
Dataset
Stage
CoT
TextGrd
M-TextGrad
REVOLVE
Before Training
21.7 (0.0%)
21.7 (-)
21.7 (0.0%)
21.7 (0.0%)
1st lteration
25.8 (-)
26.5 (2.7%t)
26.8 a 0.2% (3.88% f)
Google-proof QA
heeatiin
26.8
25.7 (3.6%e) (9.381))
29.8
(11.19%
3rd lteration
(4)
0.5%% (12.10% f)
Final Results
21.7 (8.4%5)
23.7 ()
25.1 (5.9%11)
28.3 a 0.4% (19.41%
Before Training
51.8 (0.0%)
51.8
51.8 (0.0%)
51.8 (0.0%)
1st Iteration
43.8
46.9 (7.1%))
(10.05%
MMLU-Machine Learning
2nd Iteration
43.8
45.2 (3.2%%t)
47.3 0,5% (7.99%
Iteration
43.8
44.4 (1.41)
46.4 0.6% (5.94%
Final Results
(1.50))
47.3
47.4 (0.29/1)
57.1 + 0.6% (20.72% 1)
Before Training
54.7 (0.0%%
$4.7
54.7 (0.0%)
54.7 (0.0%)
1st Iteration
51.1
55.9 (9.4%))
58.3 a 0.2% (14.09% f)
HMI.U-College. Phy
2nd lteration
51.1
61.0
62.0 1 0.4% (21.33%
3rd Iteration
55.7
(8.3%/)))
65.7 + 0.5% (17.95% ()
Final Results
54.7 (9.3%84)
60.3
61.6 (2.2%%
66.4 a 0.5% (10.12%1)
Universality: REVOLVE's universality evidenced by its consistent performance across all LLMs, including gpt-3.5- turbo-0125, GPT-4, and Llama 3.1 8B Instruet, where it delivers the highest accuracy with an average improvement of 5-7% compared to the baselines. However, there is one exception on the Gemini-1.5-Ppe model, where REVOLVE slightly trails behind TextGrad. This small performance gap may be due to the use of GPT-4o to guide the Gemini- 1.5-pro in the prompt optimization reasoning task. Given that Gemini-1.5-pro may exhibit more sophisticated reason- ing capabilities than GPT-4o in this specific scenario, the transfer of guidance from GPT-4o could have introduced suboptimal adjustments, leading to a slight degradation in performance. Despite this, REVOLVE remains highly adapt- able and effective across diverse LLM backends, reaffirming its versatility as a powerful optimization tool.
makes it ideal for cost-sensitive AT deployment.
4.2. Solution optimization
We evaluate REVOLVE on the solution optimization task, which aims to refine solutions to complex scientific or techni- cal problems, such as quantum mechanies or organic chem- istry questions. The solution evolves dynamically through self-evaluation and critique, challenging the LLM to re- fine responses continually. This process aligns with test- time training (Sun et al., 2020; 2024), where models refine during testing, and with recent progress in self-refinement for reasoning tasks (Yao et al., 2022; Madaan et al., 2024; Shinn et al., 2024), which have proven effective in iterative problem-solving.
Task Setup: We evaluate solution optimization on two benchmarks: Google-proof Question Answering (GPQA)(Rein et al., 2023), which consists of expert-level multiple-choice questions in physics, biology, and chemistry, and two subsets of the MMLU benchmark (Henddyks et al., 2020), specifically focused on Machine Learning and Col- lege Physics. GPQA is highly difficult benchmark, with experts achieving 81% accuracy and skilled non-experts reaching only 22%, highlighting the challenge of the ques- tions. Performance of LLMS on these benchmarks has not
We observe that on the GSM8K dataset with Llama 3.1, all methods stagnate, likely due to the model's saturation this task, leaving little room for improvement. Despite this, REVOLVE excels in enhancing weaker, cost-effective mod- els like gpt-3.5-turbo-01255 using feedback from stronger models such as gpt-4o. By incurring a one-time optimization cost, REVOLVE provides optimized prompts for weaker models, delivering significant performance gains without the high inference costs of stronger models. This efficiency
6

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
yet saturated, making them ideal for benchmarking solution refinement. We use three iterations of optimization for each question when using the iterative optimization methods. The final answer is determined through majority voting across all iterations. Following (Yuksekgonul et al., 2024), we use the string-based exact match metric. The detailed task setup is in Appendix C.
ensuring robust and versatile optimization for Al systems requiring high performance and accuracy.
(a) Google-proof QA (b) MMLU-Machine (c) MMLU-College
Baselines and LLM Backends: We compare REVOLVE against three primary baselines for solution optimization: Chain-of-Thought (CoT) (Kojima et al., 2022; Wei et al., 2022), TextGrad (Yuksekgonul et al., 2024) and Momentum- Enhanced TextGrad. All methods are applied separately on three LLMs: GPT-4o, GPT-4-0125 -previeww and Llama 3.1 8B Instruct, with themselves serving as the backend of the optimization system.
Learning
Physics
Figure 2: Loss curves w.r.t. accuracy on solution optimiza- tion task.
Empirical Analysis of Loss Curves. To evaluate RE- VOLVE's second-order-inspiree behavior, we analyze op- timization curves for solution optimization. Since explicit numerical loss values unavailable, we use test accuracy as a proxy for loss and extend the iterations to 6 for clearer trends. Figure 2 illustrates three key effects:
Detailed baseline configurations and prompting exemplars can be found in Appendix C.
Escaping Local Optima: REVOLVE surpasses perfor- mance plateaus by leveraging cumulative response dy- namics (Figure 2b).
Results: As shown in Table 2. across all benchmarks, RE- VOLVE significantly improves the performance of Llama 3.1 8B Instruct compared to all baselines. On average, across the three benchmarks, REVOLVE achieves a 17.79% relative improvement in final accuracy over TextGrad. This substantial gain highlights the effectiveness of incorporat- ing second-order gradients into the optimization process, enabling more precise adjustments and greater performance gains on solution optimization tasks. More experimental results are shown in Appendix D.
Stabilizing Updates: Unlike M-TextGrad, which ex- hibits oscillations (e.g.. Figure 2c), REVOLVE ensures smoother optimization.
Enhanced Performance: Loss curves show REVOLVE's effectiveness in refining solutions iteratively across all datasets.
These results confirm REVOLVE's ability to simulate second-order effects, improving optimization without ex- plicit Hessian computations.
Deterioration in TextGrad:
Interestingly, we observe per- formance deterioration with TextGrad the MMLU bench- mark, where both intermediate and final results are worse than the initial state. This highlights a key limitation of first-order optimization: relying solely on immediate feed- back without accounting for curvature can lead to unstable optimization, potentially causing the model's performance to degrade over time.
4.3. Code Optimization
The Code Optimization task aims to refine code snippets to improve their correctness and runtime efficiency, often with limited supervision from local tests and iterative self- evaluation. This task is also well-suited for evaluating opti- mization techniques as it requires handling intricate problem constraints and optimizing through iterative adjustments.
Fluctuations in Momentum-Based TextGrad: While Momentum-Based TextGrad avoids stagnation seen in TextGrad method, its performance fluctuates significantly across iterations. This is due to its reliance on larger, varied changes when feedback becomes repetitive, which can lead to overshooting and destabilization. Though it helps break feedback loops, momentum-basee methods often amplify change without tracking the precise evolution of responses.
Task Setup:
We evaluate code optimization using the Leet- Code Hard dataset (Shinn et al., 2024), an online platform featuring coding challenges commonly used for technical interview preparation. The primary metric for this task is the Completion Rate, which measures the percentage of problems for which all test cases are passed, calculated as Number of problems passed Since LeetCode test cases are not pub- TTtta number of problems licly available, generated code is submitted the LeetCode platform for evaluation on these unseen test cases. Results are averaged over multiple runs for robustness. Additional details of the task setup are provided in Appendix E.
In contrast, REVOLVE overcomes these limitations by cap- turing gradient curvature, enabling better global adjustments and avoiding stagnation, proving its superiority in com- plex optimization scenarios. These results illustrate that by spending additional computational resources during test- time, REVOLVE significantly enhances performance, even for advanced models. Its iterative, second-orde optimiza- tion approach makes it highly effective across diverse tasks,
Baselines and LLM Backends: We evaluate REVOLVE against four key baselines on the LeetCode Hard dataset using Llaam 3.1 8B Instruct model as the backend:
7

---

REVOLVE:
Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Table 3: Code optimization results (averaged over seeds) on LeetCode Hard for various LLMs, with themselves as the optimization engine. Values in parentheses show the relative improvement in completion rate over TextGrad.
Models
Method
Completion Rate (Improv, over TextGrad)
Zero-shot
0,58 (25.4 (5.49%
Reflexion (1 demonstration, iterations)
0.42 + 0.003 (17 (7.65%LL
GPT-4o
TextGraa (0 demonstrations, 5 iterations)
0.51 + 0.005 (-)
M- M-TexIGrad demonstrations, iterations)
0.49 0.005 (3.92%))
REVOLVE demonstrations, 5 iterations)
0.52 0.002 (1.96%1
Zero-shot
0.33 (35.29%..)
Reflexion (1 demonstration, iterations)
0.41 + 0.002 (19.61%%.
GPT-4-0125 p--r-vii
TextGrad (0 demonstrations, 5 iterations)
0.51 + 0.003 (-)
M-TextGrad demonstrations. iterations)
0.45 + 0.006 (11.76%kJ
REVOLVE (0 demonstrations, 5 iterations)
0,56 = 0.003 (9.80%
Zero-shot Reflexion demonstration, iterations)
0.12 (50%.)
0.20 + 0.002 (16.67%))
Llama 3.1 8B Instruct
TextGrad (0 demonstrations. 5 iterations)
0.24 = 0,005 (-) 0.25 0.003 (4.17%))
M-TextGrad (0 demonstrations, 5 itcrations)
REVOLVE (0 demonstrations. 5 iterations)
0.31 0,006 (29.17%))
Zero-shot Baseline: We follow the zero-shot setup in (Shinn et al., 2024).
for REVOLVE, considering both immediate feedback and response evolution across iterations.
Reflexion (Shinn et al., 2024): The state-of-the-art method for code optimization, which prompts an LLM to
Variant Prompt: It differs from our method by directly
instructing the LLM to generate more diverse responses, effectively pushing it towards greater variation with each iteration.
self-reflect generated code snippets and errors based on candidate unit tests. Reflexion then prompts the LLM
to update the code based on this self-reflection. We run Reflexion using a one-shot setting, with one in-context demonstration to guide its behavior.
TextGrad Prompt: Serving as the baseline, TextGrad's prompt focuses primarily on immediate feedback, making adjustments based solely on the latest response.
TextGrad and Connettu--Enhancee TextGrad: We run TextGrad, M-TextGrad, and REVOLVE in a zero-shot setting without demonstrations, refining the code based solely on feedback from each iteration.
We evaluated object counting in prompt optimization (Ta- ble 4). The Variant prompt outperforms TextGrad by encour- aging broader shifts, while REVOLVE improves accuracy through stable, iterative refinement, avoiding abrupt changes and ensuring controlled optimization.
Results: As shown in Table 3, REVOLVE achieves the highest performance on LeetCode Hard. With Llama 3.1
Moreover, to assess REVOLVE's computational efficiency, we compare its GPU memory usage and runtime with base- line methods across three task categories. Detailed results are provided in Appendix G.
8B Instruct, it attains a 31% completion rate, a 29.17% improvement over TextGrad, surpassing Reflexion (16.67%) and Momentum-Enhaneee TextGrad (7%). These results underscore REVOLVE's effectiveness in refining code for complex problems.
5. Conclusion
4.4. Ablation Study
In this paper, we introduced REVOLVE, optimization framework that extends traditional methods by consider- ing the evolution of responses over multiple iterations. In- stead of focusing only on immediate feedback, REVOLVE incorporates insights from the similarity between consec- utive responses, akin to how second-order information is used in optimization. By capturing these iterative changes, REVOLVE achieves more stable, consistent improvements across various tasks. More broadly, REVOLVE represents more than just performance gains, it highlights the growing potential of integrating established optimization principles with LLMs. By bridging these domains, it demonstrates how the rich knowledge from traditional optimization can be adapted unleash the power of LLMs. We envision future research exploring adaptations of other advanced op- timization techniques in textual manner, further unlocking the synergy between optimization and AI systems.
Given the simplicity of our method, there are complex components that can be eliminated for traditional ablation. Instead, we conduct an ablation study by testing different prompt designs to evaluate their impact on performance. Specifically, we compare our REVOLVE prompt, a variant of this prompt, and the prompt used TexiGrad to high- light the effectiveness of our approach. More details on the prompts can be found in Appendix F.
Table 4: Prompt optimization results for reasoning tasks for various LLMs, with got-40 as the optimization engine.
Dataset
Method
Accuracy &
TexiGrad
77.0
Object Counting
Variant
80.0
REVOLVE
83,0
REVOLVE Prompt:
This prompt is carefully designed
8

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Impact Statement
Jin, Y., Zhao, Q. Wang, Y., Chen, H., Zhu, K., Xiao, Y., and Wang, J. Agentreview: Exploring peer review dynamics with Ilm agents. In EMNLP, 2024b.
This work introduces REVOLVE, an optimization frame- work that improves LLM adaptability by incorporating response evolution over multiple iterations. By address- ing stagnation in local optima, it enhances performance in prompt refinement, solution optimization, and code genera- tion. While our method strengthens AI efficiency and robust- ness, we acknowledge its potential for adversarial misuse. Our goal is to advance Al optimization while promoting re- sponsible and secure applications. This research contributes to scalable AI systems, reducing manual engineering efforts and supporting safer, more reliable AI deployment. More broadly, REVOLVE pioneers the integration of established optimization principles into the textual domain, which opens new avenues for enhancing LLM systems.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., al. Dspy: Compiling declarative lan- guage model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Represen- tations, 2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Liu, B., Li, X., Zhang, J., Wang. J., He, T., Hong, S., Liu, H., Zhang, S., Song, K., Zhu, K., et al. Advances and challenges in foundation agents: From brain-inspired in- telligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.0199) 2025.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L.,, Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Anthropic. Introducing claude. 2023.
URL
https: http:///wwwattthrooeoo com/ index/
int introducing-claucee
Prasad, A., Hase, P., Zhou, X., and Bansal, M. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint wxXiv:2203.0728/ 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Pryzant, R., Martinez, R. D., Dass, N., Kurohashi, S., Juraf- sky, D., and Yang, D. Automatically neutralizing subjec- tive bias in text. In Proceedings of the aaai conference on artificial intelligence, volume 34, pp. 480-489, 2020.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint a 12.11..44440 /4168, 2021.
Pryzant, R., Iter, D., Li, J., Lee, T., Zhu, C., and Zeng, M. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.0s: 03495, 2023.
Guo, Q. Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language mod- cls with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint (tt.::2000083333. 2023.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang. R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level proof q&a benchmark. arXiv preprint arXiv:231 12022, 2023.
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03 3000, 2020.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Hinton, G. H. and Salakhutdinov, R. R. Reducing the di- mensionality of data with neural networks. science, 313 (5786):504-577, 2006.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 2xxiv2206.006111 2022.
Jin, H.. Chen, R., Zhou, A.. Zhang, Y., and Wang, H. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models. arXiv preprint arXiv:4402.03299 2024a.
9

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Sun, Y., Wang, X., Liu, Z.,, Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generaliza- tion under distribution shifts. In International conference on machine learning, pp. 2229-9248, PMLR, 2020.
Twelfth International Conference on Learning Represen- lations, 2024. URL https:/ /pwpre/ieeeeeeeeeeeeeeeeeeeeeeee net/ Berumidd-B4VVoWEEI
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:22 10.03629, 2022.
Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., et al. Learn- ing to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.06620 2024.
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., and Zou, J. Textgrad: Automatic" dif- ferentiation" via text. arXiv preprint arXiv:2406.0744 07449. 2024.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of -of--oo-tthhh--- can solve them. arXiv preprint arXiv:22 (0.::2200.0920 10.09261,1.1.00 2022.
Zhang, S., Zhang, J., Liu, J., Song, L.,, Wang. C., Krishna, R., and Wu, Q. Offline training of language model agents with functions as learnable weights. Forty-first Inter- national Conference Machine Learning.
Team, G., Anil, Borgeaud, Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312 1805, 2023.
Zhang, W., Tang, K., Wu, H., Wang, M., Shen, Y., Hou, G., Tan, Z.. Li, Zhuang, Y., and Lu, W. Agent-pro: Learn ing to evolve via policy-level reflection and optimization. arXiv preprint arXiv: 17574, 2024.
Wei, J., Wang, X., Schuurmans, D... Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-ot caii--fffhoooph thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zhou, W., Ou, Y., Ding, S., Li, L.. Wu, J., Wang, T., Chen, J., Wang, S., Xu, X., Zhang, N., et al. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406. 18532, 2024.
Xiao, Y., Jin, Y., Bai, Y., Wu, Y., Yang, X., Luo, X., Yu, W., Zhao, X., Liu, Y., Chen, H., et al. Large language models can be good privacy protection learners. In EMNLP, 2024.
Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Language agents as optimizable graphs. arXiv preprint arXiv:2402.16823, 2024.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models optimizers. In The
10

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
A. System Prompt Details for REVOLVE
REVOLVE, we use system prompts designed to guide iterative response refinement. The prompts focus on comparing the current response with previous iterations, emphasizing gradual. thoughtful evolution. They request the model provide feedback not only on immediate changes but also on patterns observed across multiple iterations.
we use the following glossary to the system prompt:
GLOSSARY TEXT
Glossary of tags that will be sent to you:
<<LM_ST SMM_SYSTMM PROMPT>:
The system prompt for the language model.
<<IM_INPT>: The input to the language model.
<LM <LMM_ OUTPUTTPT The output of the language model.
<FEEDBACK>: The feedback the variable.
<CONVERSATION>: The conversation history.
<FOCUS>: The focus of the optimization.
<ROLE>; The role description of the variable.
The Optimize Prompts details are as follows:
OPTIMIZER SYSTEM PROMPT
"You are part of an optimization system that improves text (i.e., variable) by analyzing how the responses evolve multiple iterations.
"Your goal is not just make single improvement, but to ensure that the variable evolves naturally and meaningfully over time."
"Focus on adjusting the variable in a way that each step introduces thoughtful, measured changes based on past iterations, rather than drastic shifts.
"The feedback provided will help guide these adjustments, but ensure that your improvements maintain coherence and contextual alignment."
"You MUST give your response by sending the improved variable between caariable_start_tae ((improved variable) } (new_ variable end tag) tags."
['[GOSSARY_
Textual Gradient Descent Prompt Prefix
"Here is the role of the variable you will improve: vRrib>//ariable, RROLL)
"The variable is the text within the following span: <VARIABLE> wvaiible_shoot </VARIABLE>" "Here is the context and feedback we received for the variable:"
*<CONTEXT> variable_ larllle_grad___ </CONTEXT>""
"Additionally, reflect on how the responses to this variable have evolved across iterations:"
"<PAST <<PAST_ITERATIIE ATT.TTTTRRTSOS///II (pTTS> (past__alussssssss)ss (elluSs)</AASTTTTTEE ITERATIONS>"
"Make nuanced improvements, keeping in mind that too-similar responses suggest insufficient change, but avoid making overly large changes."
"Ensure that the response evolves in coherent and thoughtful manner that aligns with the context, feedback, and past responses."
The following how we save gradients the variable.

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
GRADIENT TEMPLATE
"Here is
"This conversation is part of larger system. The output is used as presponse_dess
"Here the feedback we received for {variable_ded desc) (fexFEEEDBBCKK(eebbaaa (FEEDAAA
the conversa-
"Additionally, consider how the responses to this variable have changed across previous iterations:"
"<PAST
SENASTTITRAATIOSSS// (ATTOONSSppasaallleeee >/PAST ITERATIONS>"
"Make sure future responses reflect meaningful, gradual evolution based on these past iterations, encouraging thoughtful progress rather than drastic shifts."
B. Prompt Optimization
For the dataset split, follow the settings used in TextGrad (Yuksekgonul et al., 2024). The Big Bench Hard Object Counting dataset is divided into 50/100/100 samples for ttain/validation/eest, respectively. For GSM8K, adopt the split from DSPy (Khattab et al., 2024), using 200/300/1399 samples for train/validation/tess In each task, we limit the training set to 36 samples, consistent with the TextGrad setup. Example queries for each dataset are shown below:
Example Query for Big Bench Hard Object Counting
I have apple, three bananas, strawberry, peach, three oranges, plum, raspberry, two grapes, nectarine, and blackberry. How many fruits do I have?
Example Query for GSM8K
Natalia sold clips 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
For the momentum-enhant enhanced TextGrad baseline, to ensure a fair comparison with REVOLVE, which accounts for responses from all previous iterations, set the momentum window 12 that momentum-enhanced TextGrad has access gradients from all prior iterations.
Regarding specific hyperparameters for the LLMs, we set the temperature to 0 (1 x 10-6 for Llama 3.1 8B Instruct), allow maximum of 2000 new tokens, and use top-p value of 0.99.
C. Solution Optimization
For the solution optimization task, we follow the experimental setup outlined by TextGrad (Yuksekgonul et al., 2024). This ensures fair comparisons across all experiments. We evaluate two benchmarks: Google-Proof Question Answering (GPQA) (Rein et al., 2023) and two subsets from the MMLU benchmark (Hendrycks et al., 2020), Machine Learning and College Physics. Following the simple-evals repository practice, we employ string matching to extract the final answer (one of ABCD) and compare it to the ground truth. The datasets comprise 198 questions in the GPQA Diamond subset, 1 112 in MMLU Machine Learning, and 92 in MMLU College Physics. We compare REVOLVE against three primary baselines for solution optimization:
Chain-of-Thougee (CoT) (Kojima et al., 2022; Wei al., 2022): This baseline serves as our initial baseline. This method employs step-by-step reasoning process, providing strong foundation for comparison in complex problem-solving tasks.
TextGrad (Yuksekgonul et al., 2024): This method leverages textual gradients to iteratively refine solutions. For the solution optimization task, we apply three iterations of test-time updates using TextGrad, refining the solution at each step. The process involves making one call to GPT-4o to evaluate the test-time loss, another call to collect gradients, and a final call to update the solution accordingly.
12

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Momentum--Enhaneed Textgrad: This model builds upon the original TextGrad framework by incorporating momentum. This variant aims to overcome potential stagnation in the optimization process by adjusting the magnitude of updates based on the history of previous iterations. Like TextGrad, we apply three iterations of test-time updates for this method.
For TextGrad, Momentum Enhanced TextGrad, and our proposed REVOLVE method, we determine the final answer through majority voting across all iterations. This approach ensures that we capture the best performance from each method over multiple refinement steps.
Example Query for GPQA Diamond
Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.
A) A 1,,,ohhexane-1,,5-triine B = dimethyl fumarate
B) A benzoquinone, B = dimethyl fumarate
C) benzoquinone, B methyl 2-hydroxypropanoate
D) cyclohexane- 1,3,,mee-,,,,,,i.mm,,,, B methyl --hydroxypropanozte
Example Comparison of TextGrad and REVOLVE
Answer the following multiple-choice question... Select the suitable reagents for the following mentioned reactions. butan-2-one + NaCN + A senrdrox----mtthllbutaeeee 2444aanrybbeery222yyyyysyyuuuuu
(H20) ->
acid.
A) A NaHSO3, B CH3COOH
B) A = H30+, B = HC1
C) A NaHSO3,
HCI
D) A = H30+, B CH3CООН
(Ground Truth Answer: C)
TextGrad Response: ...let's evaluate the options: ... Option B: A H30+, B HCI H3O+ (hydronium ion) is suitable proton source for cyanohydrin formation. HCI is strong acid and commonly used for hydrolyzing nitriles to carboxylic acids. Option C: A NaHSO3, B = HCI NaHSO3 is not typically used in cyanohydrin formation. HCI is suitable for hydrolyzing nitriles. Based on the analysis, the most suitable reagents for the reactions are: A H30+ for the cyanohydrin formation. B HCI for the hydrolysis of the nitrile to a carboxylic acid. Therefore, the correct answer is: Answer:
REVOLVE Response: The first reaction is the addition of cyanide ion (CN-) to butan-2-one to form 2-hydroxy-2- methylbutanenitrile. This is an example of cyanohydrin reaction, which typically involves the addition of cyanide ion aldehyde or ketone in the presence of catalyst such sodium bisulfite (NaHSO3), The second reaction is acid,
the hydrolysis of 2-(4 This is an example of hydrolysis reaction, which typically involves the addition of water (H20) in the presence of a strong acid such as hydrochloric acid (HCI). Based on this analysis, the suitable reagents for the reactions are:A = NaHSO3 (sodium bisulfite) for the first reaction. B HCI (hydrochloric acid) for the second reaction. So, the correct answer Answer: C.
to form
beenzylphenyl)-2hhyyhh
Case Study: In this example, TextGrad incorrectly selects H30+ for cyanohydrin formation, misunderstanding the role of NaHSO3, which is typical catalyst in such reactions. Its reliance first-order feedback leads to erroneous conclusion. In contrast, REVOLVE correctly identifies NaHSO3 the catalyst for the first reaction and HCI for the hydrolysis in the second reaction. By leveraging second-order gradients, REVOLVE better captures the complexities of the chemical mechanisms, leading to the correct answer.
13

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
D. Detailed Experiments on Solution Optimization
Table 5: Solution optimization results for various LLM, with themselves the optimization engine. The values in
parentheses represent the relative improvement of the method compared to TextGrad.
Accuracy % (Improv. over TextGrad)
Dataset
Models
Stage
CoT
TextGrad
M-TextGrad
REVOLVE
Before Training
50.4 (0.0%)
50.4 (-)
50.4 (0.0%)
50.4 (0.0%)
Iteration
50.4 (-)
51.1 (1.3%
50.9 (0.999%1)
GPT-4o
2nd lteration
50.5 (-)
50.7 (0.4%))
51.3 (1.58%1)
3rd Iteration
(-)
51.9 (2.7%1)
= (4.75%f)
Final Results
50.4 (2.1%%)
51.5 (-)
(1.7%%))
53.0 0.7% (2.91%
Before Training
38.8 (0.0%)
38.8 (-)
38.8 (0.0%)
38.8 (0.0%)
Iteration
38)5
39.3 (2.0%))
39.5 0.3% (2.60%))
Googlc-proof QA
GPT-4-0125-
2nd Iteration
38)
40.1 (4.7%%)
40.3 + 0.5% (5.22%
3rd Treration
38.2 (-)
(5.7%1)
41.0 0.4% (7.33%1)
Final Results
38.8 (1.8%))
38.1
41.5 (8.99+1)
42.2 0.6% (10.76%1
Before Training
21.7 (0.0%)
21.7
21.7 (0.0%)
21.7 (0.0%)
1st Iteration
25.8 (-)
26.5 (2.7%1)
26.8 0.2% (3.88
LJama 3.1 8B Instruct
2nd Tteration
26.8 ())
29.3
29.8 0.5% (11.19%
3rd Iteration
24.8 (-)
25.7 (3.6%)1)
27.8 0.5% (12.10%
Final Results
21.7 (8.4%))
23)7
25.1 (5.9%1)
28.3 0.4% (19.41%1)
Before Training
85.5 (0.0%)
85.5 (-)
85.5 (0.0%)
85.5 (0.0%)
1st Iteration
85.5 ())
85.5 (0.0%)
85.8 0.5% (0.35%
GPT-4o
2nd Iteration
85)6
85.4 (0.2%))
86.1 1.0% (0.58%1)
3rd Iteration
85.6
85.3 (0.3%))
86.4 (0.93%
Final Results
85.5 (0.3%%)
85.8
85.0 (0.9%))
86.7 (1.05%1)
Before Training
76.3 (0.0%)
76.3
76.3 (0.0%)
76.3 (0.0%)
1st Iteration
7-.4
77.2 (1.0%
77.1 = 0.4% (0.92%-1
MMLU-Machine Learning
GPT-4-0125-press preview
2nd Tteration
(-)
77.8 (1 (15%)))
77.9 + 0.6% (1.70%
3rd Iteration
7-
78.1 (1.451)
79.2 = 0.7% (2.86 f
Final Results
76.3 (3.3%%)
78.9
79.3
81.0 + 0.8% (2.66%()
Before Training
51.8 (0.0%)
51.8
51.8 (0.0%)
51.8 (0.0%)
1st Itcration
43.8
46.9 (7.1%+1)
48.2 # 0.3% (10.05%
Llama 3.1 8B Instruct
2nd Iteration
4-)
(3.2%1)
47.3 + 0.5% (7.99%f)
Iteration
4-)
(1.4%))
46.4 + 0.6% (5.94%
Final Results
51.5%
47)
(0.2%)))
57.1 0.6% (20.72%
Before Training
(0.0%))
91.0())
(0.0%)
91.0 (0.0%)
1st Iteration
91.6(-)
91.6 (0.0%)
91.8 + 0.5% (0.22%))
GPT-4o
2nd Iteration
92.1
(0.2%1)))
92.5 4 0.7% (0.43%
Itcration
92.8
93.2 (0.43%f) (1.7%.)
Final Results
(2.7%%)
9-.5
(2.3%)
94.1 (0.64%
Before Training
81.6 (0.0%)
8166()
81.6 (0.0%)
81.6 (0.0%)
1st Iteration
82.(-)
81.9
0.33% (0.12%
MMLU-College Physics
GPT-4-0125- prevview
Iteration
83.1
82.4%))
83.4 (0.36%f)
3rd Tteration
8-)1
82.1 (2.4%.)
0.6% (0.48%f)
Final Results
(4.7%%)
8-.6
82.3
5.79% (0.35%
Before Training
54.7 (0.0%)
54.7
54.7 (0.0%)
54.7 (0.0%)
Iteration
51.1
55.9 (9.4%1)
0.2% (14.09%
LJama 3.1 8B Instruct
2nd Iteration
$1.1
(19.(1))%)
0.4% (21.33%
3rd Iteration
5-.7
(8.3%%
6.57
(17.95%
Final Results
34.77
60.3
61.6 (2.2%1)
66.4 0.5% (10.12%
E. Code Optimization
In the code optimization task, we primarily rely on the settings from previous work, particularly TextGrad Tex(Grad(Yukseegoni) al., 2024), to ensure a fair comparison across experiments. Specifically, adopt the version of Reflexion (Shinn et al., 2024) as used in TextGrad, which includes minor modifications for compatibility within the TextGrad framework.
For the baselines, we employ two key approaches:
Reflexion (Shinn et al., 2024): this setup, the language model guided by a one-shot prompt instructing it to provide feedback on the code it generates. The process begins by generating an initial solution based on provided code prompt.
14

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
The solution is first tested locally, and if it passes, it is then submitted the LeetCode platform for rigorous evaluation using harder test cases. If the local tests fail, Reflexion is used to request feedback from the model refine the code. This feedback-guided optimization is repeated for up to 5 iterations, during which the model continuously improves the code until it successfully passes local tests and is ready for submission.
TextGrad (Yuksekgonul et al., 2024): This baseline runs 5 independent trials, each with five seeds with [15, 17, 21, 55, 91], and averages the results to ensure consistency. During each optimization iteration, TextGrad performs three key operations: first, it makes a call to GPT-4o to evaluate the test time loss; next, collects gradients based on the loss; finally, the code snippet updated according to the gradients. This process repeats, optimizing the code over several iterations to minimize the test time loss and improve performance the test cases.
By following the setup from TextGrad, we ensure that both Reflexion and TextGrad are evaluated under the same conditions, facilitating a fair and consistent comparison with these two baselines and REVOLVE.
Example Query for LeetCode Hard
def minimun/Time(grid. List[Lis((int]) -> int:
..99
You are given a 'm X matrix 'grid' consisting of non-negative integers where "grid[ represents the minimum time required to able visit the cell '(row, col)', which means you can visit the cell '(row, col)' only when the time you visit it is greater than or equal to [grio]"o
You are standing in the top-left cell of the matrix in the 'Oth' second, and you must move to any adjacent cell in the you make takes second. Return the minimum time required
four directions: up, down, left, and right. Each in which you can visit the bottom-right cell of the matrix. If you cannot visit the bottom-right cell, then return *-1', Example 1:
Input: grid = [[0.113.21115.12255544333
Output: 7
Explanation:
One of the paths that we can take is the following:
at t= 0, we
the cell (0,0).
at we move to the cell (0.1). ). It is possible because grid[O][[] <= 1.
at = 2, we move to the cell (1,1). It is possible because grid[1][1] <=
at 3, we move to the cell (1,2). It is possible because grid[1][2] <= 3.
at = 4. we move to the cell (1,1). It is possible because grid[]][1] <= 4.
at 5, we move to the cell (1,2). is possible because grid[ <=
at 6, we move to the cell (1.3). It is possible because grid[1][3] <=
at 7,
to the cell (2,3). It is possible because grid[2][3] <= 7.
The final time is 7. It can be shown that it is the minimum time possible.
Example 2:
Input: grid = [10.2..4,.332.11111.0000
Output: -1
Explanation:
There no path from the top left to the bottom-right cell. Constraints: * "m grid.length'
* 'n grid[i] (riddi).length
' <= m, <= 1000
+4 <=
<= 105'
* 10 < [ili] <= 105'
* [r] 0
15

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Example Comparison TextGrad and REVOLVE
Task Description:
You have k bags. You are given 0-indexed integer array weights where weights[i] is the weight of the 8th marble. You are also given the integer kt
Divide the marbles into the k bags according to the following rules:
No bag is empty.
If the 3 marble and jth marble are in a bag, then all marbles with an index between the jth and jth indices should
also be in that same bag.
If bag consists of all the marbles with index from to inclusively, then the cost of the bag is weights|i] + weights[]].
The score after distributing the marbles is the sum of the costs of all the k bags.
Return the difference between the maximum and minimum scores among marble distributions.
Example 1:
Input: weights = k=
Output: 4
Explanation:
The distribution [1][[3,5,]] results in the minimal score of (1+1) (3+1) = The distribution [1,3],[5,]], results in the maximal score of (1+3) (5+1)= 10. Thus, we return their difference 10 6 4.
Example 2:
Input: weights = [1, 3], 2
Output: 0
Explanation:
The only distribution possible is [1][[]]
Since both the maximal and minimal score are the same, return O.
def pptMarrless List[int], k: int) -> int:
Code Generated by TextGrad: from typing import List
def DrMMarrbes(eeight.. List[int], k: int) -> int: = len(weights) if = return 0
Calculate the cost each possible bag costs [weights[i] weights(i + 1] for in range(n 1)]
Sort the costs to find the minimum and maximum possible scores costs.sort()
The minimum score is the sum of the smallest k-1 costs min,core sum(costs|: A 1])
The maximum score is the sum of the largest k-1 costs max,core sum(ccosss - :))
return mox,core - mon, Tests that the code passed: No tests passed.
**Tests that the code f
This unit test fails. Output was but expected value was: assert putMarbles([1.3] 2) ERROR: This unit test fails. Output was -1, but expected value was:
assert putMarbles((1,3555)
2) ERROR:
Code Generated by REVOLVE from typing import List
def putMarbles( weights: List[int], k: int) int: len (weights) if k 1 or k = return 0
Calculate the cost of each possible bag costs [weights[i] weights + 1] for in range(n 1)] Sort the costs to find the minimum and maximum possible scores costs.sort()
The minimum score is the sum of the smallest k-1 costs min,core = sum(cosls|:
1])
The maximum score the sum of the largest k-1 costs max,core = sum(costs] [<assts[[:: 1 :]) return max, core min,core
Tests that the code passed: assert Put.Marbles((133..1)) 2) 4 assert putMarbles(11.3) ====0 **Tests that the code failed:*s
No tests failed.
16

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
F. Abaltion Study
F.1. Ablation Study
Given the simplicity of our method, there are no complex components that can be eliminated for a traditional ablation. Instead, we conduct an ablation study by testing different prompt designs to evaluate their impact on performance. Specifically, using 1aama-3.1-8B-Isiiiie
the LLM backend, compare our REVOLVE prompt, variant of this prompt, and the prompt used in TextGrad highlight the effectiveness of our approach.
This is the prompt carefully designed for REVOLVE, which takes into account both immediate feedback and the evolution of responses across iterations.
REVOLVE Prompt:
Variant Prompt: It differs from our method by directly instructing the LLM to generate more diverse responses,
effectively pushing towards greater variation with each iteration.
TextGrad Prompt: Serving as the baseline, TextGrad's prompt focuses primarily on immediate feedback, making adjustments based solely on the latest response.
The detailed prompts for the variant follows:
OPTIMIZER SYSTEM PROMPT
"You are part of an optimization system that improves text (i.e., variable).
"
"You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable.
"You will receive some feedback, and use the feedback improve the variable.
"Pay attention the role description of the variable, and the context in which it is used.
"Importantly, focus creating responses that are varied and diverse in nature.
"You MUST give your response by sending the improved variable between variable] } new_ vo__variable _end___dd__ tag tags.
_tarrtt_stttt___ tag] (oog))((mmpp ((improved
"The text you send between the tags will directly replace the variable."
"[GLOOSAAR_TEETTY
Textual Gradient Descent Prompt Prefix
"Here is the role of the variable you will improve: <ROLE>(/vrrable (R0L_)//arrbbbdddds))RRRLLB
"The variable is the text within the following span: <VARIABLE> couriabe_shorry </VARIABLE>"
"Here is the context and feedback we got for the variable:"
"<CONTEXT> variable_grade </CONTEXT>
"Improve the variable (1 variabbe desc) }) using the feedback provided in <FEEDBACK> tags.
"Ensure that your response introduces and diverse ways of solving the problem addressing the prompt."
The following is how we save gradients to the variable.
GRADIENT TEMPLATE
"Here is
"This conversation is part of larger system, where varied and creative outputs are important.
"The output used as (response __ Here is the feedback we received for (variable_des descc) the conversation:" "#FEEDBACK> (feedbackk (rrreedbackk//EEEEBBACCCC </FEEDBACK>"
"Encourage diversity your improvements."
We conducted experiments on objective counting in the prompt optimization task, with results shown in Table 6.
On the Object Counting task, the Variant prompt surpasses TextGrad by encouraging larger, more diverse shifts in the response space, enabling the model explore more distinct outputs with each iteration. REVOLVE, on the other hand,
17

---

REVOLVE: Optimizing AT Systems by Tracking Response Evolution in Textual Optimization
Table 6: Prompt optimization results for reasoning tasks for various LLMs, with gpt-4o as the optimization engine.
Method
Accuracy
TeutGad
Object Counting
Variant
80.0
REVOLVE
83.0
Table Comparison of computational resources (GPU memory and runtime) for REVOLVE and baseline methods across tasks.
Dataset
Method
per Iteration (s)
Total Time Converge
GPU Usage (GB)
TextGrad
92.144
276.450
3.23
Objective Counting
M-TextGrad
110.721
110.732
3,24
Prompt Optimization
REVOLVE
137.815
137.821
3.23
TextGrad
135.184
1351.85
3.23
GSM8K
M-TextGrad
152.423
1219.393
3,23
REVOLVE
176.538
1235.774
3.23
TextGrad
153.522
614.216
3.24
Google-proof QA
M-TextGrad
178.879
1091.162
3,23
REVOLVE
197.235
453.461
3.24
TextGrad
172.429
896.631
3,24
Solution Optimization
MMLU-Muchine Leaminn
M-TextGrad
207.819
685.803
3,24
REVOLVE
223.807
626.659
3.24
TextGrad
188.116
1636.612
3.24
MMLU-College Physics
M-TextGrad
225.631
1308.662
3,24
REVOLVE
245.167
1054.229
3.24
TextGrad
1078.783
18986.655
6.46
Code Optimization
Objective Counting
M-TextGrad
1241.917
18472.411
REVOLVE
1352.174
15820.489
6.46
achieves even better results by promoting stable, iterative refinement rather than abrupt changes. While the Variant's strategy lead to sudden, exaggerated shifts, REVOLVE ensures smoother, controlled optimization, gradually fine-tuning responses for greater accuracy.
G. Comparison of Computational Resources
To analyze the computational efficiency of REVOLVE, we compare its GPU memory usage and runtime against baseline methods across three task categories. We use Llama 3.1 Instruct as the base LLM,, running on a setup with NVIDIA 3090 GPUs. The results are shown in Table
We observe that while REVOLVE involves slightly higher per-iteration runtime due its second-order optimization-inspiree design, it converges in fewer iterations, resulting in significant overall savings. The detailed findings are as follows:
On Object Counting dataset, REVOLVE reduces total runtime by 50% compared to TextGrad by converging in fewer iterations despite slightly higher per-iteration costs.
For solution optimization task, REVOLVE achieves 26. 14% lower total runtime than TextGrad, while M-TextGrad incurs
77.65% higher runtime due to instability.
For code optimization task, REVOLVE reduces total runtime by 16.67% compared to baselines. For GPU memory usage, REVOLVE demonstrates similar requirements baseline methods, indicating no significant increase in computational resources.
18


Reviews:
[{"review_id": "wi0JB0dxtI", "reviewer": "Reviewer_w9Up", "summary": "In this paper, the authors introduce REVOLVE, which aims to simulate the second-order derivative during the optimization process.\nThey compared their modified optimization prompts and found the outcome was better than TextGrad itself.\n\n## update after rebuttal\n\nThe authors explained  the implementation difference between REVOVLE and M-Textgrad, which makes sense to me.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time, insights, and highlight our strengths. We'll address your concerns in our following response.\n\n**Q1. The authors claim that “REVOVLE can escape local optima.”. However, they do not explain why the similarity function can be interpreted as a gradient from an implementation perspective? In other words, how can REVOLVE be understood as presented in lines 198–200 of the formula?**\n\nWe thank the reviewer for the helpful question on how REVOLVE can be interpreted as a gradient-like method, and how this relates to lines 198–200 in the formula.\n\n**How REVOLVE identifies local optima:**\n\nREVOLVE tracks when responses stop improving, even as prompts continue to evolve. This is captured by the numerator of the similarity function: \n$L(r(p_t))-L(r(p_{t-1}))$, \nwhich reflects how much the task performance changes between responses. In implementation, we provide the LLM with both evaluator feedback and the context that generated it, allowing the model to implicitly assess whether performance is improving or stagnating across iterations.\n\n**How REVOLVE escapes local optima:**\n\nTo move beyond stagnation, the model is guided by the denominator:\n$p_t-p_{t-1}$,\nwhich represents the degree of change in the prompt. If the response isn’t improving much despite prompt updates, this signals stagnation. To help the model escape, we encourage it to make stronger updates when past changes haven’t helped, and smaller ones when the trajectory is already improving. This is achieved through instructions like: _\"Ensure future responses reflect a meaningful, gradual evolution based on past iterations, ..., avoiding abrupt shifts\"_.\n\n**Why this resembles a gradient in practice:**\n\nTogether, the performance difference (numerator) and the prompt change (denominator) form a ratio that functions like a gradient: it reflects whether updates are effective and how future steps should be adjusted. While we don’t compute this numerically, the LLM is given all the elements needed to assess it implicitly, which functions as a curvature-aware, gradient-like signal that informs future updates. \n___\n**Q2. The authors need to clarify the difference between their similarity function and momentum, as neither involves second-order computation, and they appear similar in terms of implementation.**\n\nWe thank the reviewer for raising this important point.\n\nIn essence, M-TextGrad focuses on repeating feedback, whereas REVOLVE targets repeating model responses.\n\nImplementation-wise, M-TextGrad looks at repeated feedback from the evaluator. If similar feedback appears across steps, it increases the update size, following the intuition that prior changes weren’t enough. This is similar to momentum in traditional optimization. The relevant instruction is: _\"Similar feedbacks across different steps suggest that the modifications are insufficient… make more significant changes.\"_\n\nIn contrast, REVOLVE looks at repeated model responses. If the response itself doesn’t evolve meaningfully across iterations, we prompt the model to revise more significantly. This is achieved with the following prompt: _\"Additionally, consider how the responses to this variable have changed across previous iterations: <PAST_ITERATIONS>{past_values}</PAST_ITERATIONS>. Make sure future responses reflect a meaningful, gradual evolution based on these past iterations, encouraging thoughtful progress rather than drastic shifts.\"_\n\nThe two methods differ mainly in where they look for signals: M-TextGrad focuses on feedback patterns, while REVOLVE monitors the model’s own output history. We find this shift helps the model better track its progress and avoid getting stuck.\n___\n**Q3. New experiments on the DSPy baseline.**\n\nWe thank the reviewer for the suggestion.\n\nWe’ve now included DSPy as a baseline in our prompt optimization experiments. To adapt it to our setting, we used GPT-4o to generate few-shot exemplars for LLaMA 3.1 8B.\n\nThe results are shown below:\n|  Dataset   | | ||Accuracy %  |  |\n|  ----  | ----  |----  |----  |----  |----  |\n|  | CoT | DSPy | TextGrad | M-TextGrad | REVOLVE |\n| Object Counting | 65.0 | 75.0 | 77.0 | 80.0 | 83.0 |\n| GSM8K | 84.6 | 84.6 | 84.6 | 84.6 | 84.6 |\n\nOn GSM8K, DSPy performs similarly to other methods, which we believe is due to task saturation. On the Object Counting task, however, DSPy underperforms REVOLVE. We hypothesize this is because DSPy’s pipeline-level optimization and demonstration tuning are less adaptive in iterative feedback settings. REVOLVE benefits from tracking how responses evolve, which helps guide the optimization more consistently.\n\nWe’ve added these results to the manuscript and appreciate the helpful suggestion.\n___\n**Q4. Inconsistency usage of \"Revolve\" and  \"REVOLVE.\"**\n\nWe thank the reviewer for pointing this out. We have standardized “REVOLVE” in the manuscript.", "claims_and_evidence": "The authors claim that “REVOVLE can escape local optima.” To demonstrate this, they show that their method resembles a second-order derivative method. However, they do not explain why the similarity function $\\mathcal{S}(r(p_t),r(p_{t-1}))$ can be interpreted as a gradient from an implementation perspective. In other words, the authors need to clarify how their method can be understood as presented in lines 198–200 of the formula.", "methods_and_evaluation_criteria": "The authors tested several popular models on prompt optimization, solution optimization, and code optimization. The evaluation criteria is  Accuracy (Completion Rate). The evaluation process makes sense.", "theoretical_claims": "The paper has no theoretical claims.", "experimental_designs_or_analyses": "The authors compare their mothods with several TextGrad baselines. Can the authors add `DSPy` baselines since DSPy optimizers are also very competitive?", "relation_to_broader_scientific_literature": "1. The proposed method aims to provide a better optimizer for TextGrad-based applications, which is useful and improtant.\n2. The idea of applying a second-order textual optimization is novel.", "essential_references_not_discussed": "The related works are quite comprehensive, covering various recent studies.", "other_strengths_and_weaknesses": "1. The paper explores integrating an exsiting idea from traditional optimization into textual optimization. Currently, textual optimization is still in its early stages, so such an integration is meaningful.\n2. The authors need to clarify the difference between their similarity function and momentum, as neither involves second-order computation, and they appear similar in terms of implementation.", "other_comments_or_suggestions": "The authors sometimes use \"Revolve\" and sometimes use \"REVOLVE.\" It should be standardized to one format.", "questions_for_authors": "Can the authors explain the `implementation difference` between REVOVLE and M-Textgrad? I checked the prompts provided in the appendix, and I also checked the prompts in the original [TextGrad repo](https://github.com/zou-group/textgrad/blob/main/textgrad/optimizer/optimizer_prompts.py). \nIt seems that the two implementations look similar. Can the author clarify what their main modification at the implementation level is?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "btgmuBX059", "reviewer": "Reviewer_wUVx", "summary": "The paper looks at the leveraging beyond the first-order information in textual optimization. Revolve develops a way to keep an account of previous feedback steps, and goes beyond the issue of stagnating when feedback is limited or fluctuates irregularly. The authors evaluate REVOLVE on three tasks: prompt optimization, solution optimization, and code optimization. Revolve seems to converge faster, and outperform vanilla and momentum-based TextGrad.\n\n## update after rebuttal\n I still support the publication of the paper, and the authors clarified my questions around costs and implementation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time and positive feedback. We'll address your concerns in our following response.\n\n**Q1. For the second-order effects, what type of prompt or what type of workflow and LLM did the authors use?**\n\nWe appreciate the request for clarification on how second-order effects are realized. \n\nIn REVOLVE, we approximate second-order effects by combining two elements: (1) first-order feedback from the evaluator, and (2) a similarity signal that tracks how the model's responses evolve across iterations.\n\nThis similarity reflects changes in task performance between responses, scaled by how much the prompt has changed (as described in lines 198–200 of the formula). Rather than computing this explicitly, we guide the LLM with instructions like:\n_\"Consider how the responses to this variable have changed across previous iterations: <PAST_ITERATIONS>{past_values}</PAST_ITERATIONS>, …, Ensure future responses reflect a meaningful, gradual evolution.\"_.\n\nThis setup allows the LLM to reason over both current feedback and the broader trajectory of response updates. We hope this clarifies our workflow.\n___\n**Q2. Qualitative analysis of response trajectories that shows the second order effects. What type of second-order effects did the authors observe in the distribution of feedback?**\n\nWe appreciate the request for qualitative analysis. We've added a new section with full response and feedback trajectories to illustrate the differences.\n\n**Response Trajectories:**\n\n On the Object Counting task, TextGrad repeats the same prompt across the final 8 iterations:\n _\"You will answer a reasoning question. … Use standard mathematical notation… Present in bullet points… The last line should be: 'Answer: $VALUE'\"_. This reflects stagnation.\n\nM-TextGrad also plateaus: its last 9 responses repeat a verbose prompt:\n _\"Begin by summarizing the problem… Confirm the list is complete… Use consistent notation… Implement a verification step… Consider edge cases… Conclude with: 'Answer: $VALUE'\"_.\n\nREVOLVE repeats a prompt for iterations 5–8:\n _\"..., Clearly state the context… List each item… Use a consistent format… Verify the calculation… Final line: 'Answer: $VALUE'\"_\nbut updates it in iteration 9 with new semantics. This suggests second-order behavior: it detects when progress stalls and makes a more informed shift to move forward.\n\n**Feedback Evolution:**\n\n In TextGrad, feedback stays repetitive:\n_\"Clarify object listing.\" → \"Use bullet points.\" → \"Be more concise.\"_ but responses hardly improve.\n\nM-TextGrad shows oscillatory feedback:\n _\"Too verbose, streamline.\" → \"Missing verification, add back.\" → \"Too shallow, expand explanation.\"_, feedback changes, but the response bounces back and forth.\n\nREVOLVE aligns feedback with evolving responses. Early iterations focus on structure:\n _\"..., Add a verification step.\"_. Mid-phase addresses reasoning:\n_\"..., Ensure entity counts match context.\"_. Later rounds provide refinement:\n _\"..., Reduce redundancy.\"_\n\nWe hope this helps clarify the second-order behavior we observed. Full examples are now included in the revised paper.\n___\n**Q3. Additional cost of computing the second order effects?**\n\nWe thank the reviewer for raising this question. To make the cost clearer, we’ve now added a per-iteration breakdown in the main paper:\n\n|Component|TextGrad|REVOLVE|Overhead|\n|----|----|----|----|\n| Feedback Collection|75.1s|75.1s|0|\n| Optimization|16.8s|63.0s|46.2s|\n| Total per Iteration|91.9s|138.1s|+46.2s|\n\nThe extra cost mainly comes from the inclusion of past responses to let the model reason over how things have evolved. We’ve added this clarification to the paper.\n___\n**Q4. Lack of ablations for the similarity computation.**\n\nWe thank the reviewer for the suggestion. While similarity is not computed explicitly, we ablate how similarity is conveyed to the LLM. In one version, we replace the task-performance-oriented signal ($L(r(p_t)) − L(r(p_{t-1}))$) with a simpler one based only on raw text differences ($r(p_t) − r(p_{t-1})$).\n\n|Dataset|Textual Similarity|REVOLVE|\n|----|----|----|\n|Objective Counting| 81.0|83.0|\n|GSM8K|84.6|84.6|\n\nThese results confirm that task-performance-oriented similarity offers more gains, as it offers deeper understanding of response efficacy. We have added these results in the revised manuscript.\n___\n**Q5. Lack of confidence intervals or significance testing.**\n\nThank you for pointing this out. For all tasks, we run five independent trials using different random seeds. The reported accuracy (e.g., 95.3% on Object Counting) reflects the average across these runs. As noted in Appendix E, we also use consistent seeds [15, 17, 21, 55, 91] for code optimization.\n\nWe’ve now added confidence intervals to the main results, e.g., Table 1 would be updated as follows:\n\nFor REVOLVE:\n* GPT-3.5: 95.5 ± 0.9% (3.9%↑)\n* GPT-4: 96.3 ± 0.6% (2.2%↑)\n* Gemini 1.5 Pro: 94.0 ± 0.0% (0.0%)\n* Llama 3.1: 83.0 ± 1.4% (7.8%↑)\n\nWe’ve revised the paper accordingly to reflect this.", "claims_and_evidence": "The claims in this paper are supported by the experimental results across the three optimization tasks, but there are some limitations in the evidence provided. The paper makes strong claims about escaping local optima, but lacks e.g., qualitative analysis of actual response trajectories that would illustrate this mechanism in action. It would help my understanding quite a bit to see how this mechanism works in action. I may be missing this, but I also do not see how the similarity is computed using an LLM, which seems important.", "methods_and_evaluation_criteria": "The experiments show promising results, i think they are generally adequate and also mostly mirrors the evaluations in TextGrad. There are two claims that could be better off further supported with additional analyses: 1) How do different ways of computing similarity help? How are the design choices made there? 2) While there is a section in appendix with a few paragraphs, the claims around computational benefits would be better off with a more clear analysis.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "I checked the experimental design across the three optimization tasks (prompt, solution, and code). The overall structure is sound, comparing REVOLVE against established baselines with appropriate metrics for each domain. However,  there is a lack of ablations for the similarity computation and what kinds of second order effects are captured.\n\nIn the epxeriments, while absolute performance improvements are highlighted, the paper lacks confidence intervals or significance testing. As far as i can tell, all experiments are conducted with one pass, and e.g., 1% improvement in object counting corresponds to getting 1 more question right (if remember the dataset size correctly, correct me if i’m wrong please)", "relation_to_broader_scientific_literature": "The paper's approach to tracking response evolution across iterations provides a relatively interesting extension to existing textual optimization methods. The potential downstream applications in automated prompt engineering and self-refining systems could be useful, particularly for industrial applications where optimization efficiency matters. However, the core idea remains a straightforward extension of existing gradient-based textual optimization rather than a fundamental breakthrough.", "essential_references_not_discussed": "I think essential references are mostly discussed, except one concurrent work named HessianGrad:\n@misc{\nzhang2025hessiangrad,\ntitle={HessianGrad: Optimizing {AI} Systems with Hessian-Aware Textual Gradients},\nauthor={Peiyan Zhang and Haibo Jin and Leyang Hu and Xinnuo Li and Liying Kang and Man Luo and Yangqiu Song and Haohan Wang},\nyear={2025},\nurl={https://openreview.net/forum?id=0hc7iQLhCt}\n}", "other_strengths_and_weaknesses": "To me the major weakness is a clear discussion of the observed second order effects and the specific ways to incorporate the second order effect (what type of prompt, what type of workflow and LLM, what is the cost of such a call, etc.)", "other_comments_or_suggestions": "-", "questions_for_authors": "Repeating here for completeness: For the second-order effects, what type of prompt or what type of workflow and LLM did the authors used?  What is the additional cost of computing the second order effects? What type of second-order effects did the authors observe in the distribution of feedback?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8S1lAK3re", "reviewer": "Reviewer_L3Xh", "summary": "The manuscript proposes Revolve, a new method that enhances LLM-based optimization by simulating second-order dynamics for self-evolving agents. Existing gradient approximation methods use textual feedback to approximate first-order gradients but are less effective for long-horizon optimization. Revolve addresses this by modeling response evolution over multiple iterations, which captures higher-order refinements for more stable and informed adjustments. This method has been tested on various tasks, it consistently outperforms existing baselines like CoT, TextGrad, and Reflexion across diverse LLM backends from Llama-3.1-8B to GPT-4o. The evaluation also shows that Revolve enhances efficiency that reduces total runtime by up to 50%. By shifting textual optimization from a purely feedback-driven process to a structured, trajectory-aware approach, Revolve enables more generalizable and scalable adaptation in AI agent systems.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed evaluation of our work, for acknowledging the soundness of our work, the clarity of the writing, the coverage of related literature, and the comprehensiveness of our experiments. We'll address your concerns in our following response.\n\n**Q1. New experiments on the ProTeGi baseline.**\n\nWe thank the reviewer for suggesting a comparison with ProTeGi. We have now included ProTeGi as a baseline in our prompt optimization experiments. To adapt it to our setting, we used GPT-4o to generate high-quality few-shot exemplars for LLaMA 3.1 8B.\n\nThe updated results are shown below:\n\n|  Dataset   | | | | | Accuracy % |\n|  ----  | ----  |----  |----  |----  |----  |\n|   | CoT | ProTeGi | TextGrad | M-TextGrad | REVOLVE|\n| Object Counting  | 65.0 |68.0 |77.0 |80.0 |83.0 |\n| GSM8K  | 84.6 |84.6  |84.6  |84.6  |84.6  |\n\nWe observe that while ProTeGi yields slight improvements over standard CoT prompting on Object Counting, our method achieves higher gains. On GSM8K, all methods perform similarly, likely due to task saturation. These results further validate the effectiveness of REVOLVE in improving prompt optimization, particularly in more challenging settings.\n___\n**Q2. A typo that doesn’t impact clarity too much but ought to be addressed: in page 7, row 353, the results should reference Table 2 instead of Table 5, as Table 5 contains the complete results.**\n\nThank you for pointing this out. We have corrected the reference in the revised manuscript.", "claims_and_evidence": "After thoroughly check the paper, I find that the claims presented in the paper are well-supported by both theoretical justification and empirical validation: \n\nClaim 1:\n\nMethodology: Revolve enhances LLM-based optimization by simulating second-order effects.\n\nEvidence:\n\n1.\tThe mathematical framework in Section 3.4 formalizes how Revolve captures response trajectory dynamics.\n\n2.\tSection 3.5 clarifies that Revolve does not compute second-order derivatives numerically but instead approximates such effects through structured response tracking.\n\n3.\tEmpirical loss curves in Figure 2 illustrate that Revolve mitigates stagnation and stabilizes optimization, whereas first-order methods exhibit oscillatory behavior.\n\nClaim 2:\n\nAccuracy: Revolve outperforms existing baselines across multiple tasks.\n\nEvidence: Evaluation results in Section 4 support this claim, e.g., a 29.17% performance gain over state-of-the-art baselines in code optimization.\n\nClaim 3:\n\nEfficiency: Revolve optimizes more efficiently by reducing total runtime.\n\nEvidence: Appendix G reports that Revolve achieves faster convergence while maintaining stable performance improvements.\n\nClaim 4: \n\nGenerality: Revolve generalizes effectively across LLMs.\n\nEvidence: Comprehensive multi-model evaluations confirm its adaptability, with results detailed in the universality analysis in Section 4.1.", "methods_and_evaluation_criteria": "The methodology is well-structured, and the evaluation criteria are comprehensive and appropriate:\n\n1.\tThe method is tested across various key tasks: solution optimization, prompt optimization, and code optimization, where each requires iterative refinement and long-horizon reasoning. \n\n2.\tThe experiments leverage diverse benchmarks, e.g., BBH, MMLU, and LeetCode Hard, which covers a range of reasoning complexities. \n\n3.\tBaseline comparisons with CoT, TextGrad, and Momentum-Enhanced method provide a fair assessment across different optimization paradigms. \n\n4.\tThe study also evaluates Revolve across multiple LLM architectures to demonstrate its generalizability beyond specific models. \n\n5.\tComputational efficiency is analyzed to ensure that performance improvements are not achieved at excessive computational cost. \n\nOverall, the experimental setup and evaluation methodology are well-designed, with no major concerns.", "theoretical_claims": "The derivations in this paper are logically sound and align with empirical results, with no evident inconsistencies.", "experimental_designs_or_analyses": "Please refer to methods and evaluation criteria part.", "relation_to_broader_scientific_literature": "I think this approach would connect to broader trends in test-time adaptation and self-refinement. Beyond improving accuracy, the paper contributes to efficient LLM adaptation, balancing performance with computational efficiency, which is a critical challenge in large-scale AI. Its strong generalization across diverse LLMs and tasks positions Revolve as part of a broader effort to enhance AI adaptability and inference-time optimization.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tOriginality: The idea that modeling response evolution to simulate second-order optimization effects in textual adaptation is interesting and holds broad significance for both LLM research and practical applications. Besides the methodological contribution, the paper marks a conceptual shift in LLM optimization. It moves beyond purely feedback-driven updates by incorporating structured response tracking, which bridges textual optimization with second-order principles. \n\n2.\tClarity: This paper is overall well-structured and easy to follow. The general setup in the section of methodology helps readers to understand the proposed Revolve framework. Additionally, the detailed information on task-specific implementations in the Experiment section, along with supporting information in the Appendix, facilitates implementation and reproducibility.\n\n3.\tEvaluation: Extensive evaluation across multiple tasks demonstrates Revolve's substantial performance improvement compared to baseline methods across various LLM backbones.\n\n4.\tSignificance: Revolve's design exhibits impressive generality across various tasks. It has great chances to benefit researches in diverse directions.\n\nWeaknesses:\n\n1. The authors are recommended to compare Revolve with ProTeGi [1] in prompt optimization for a more thorough comparison.\n\n[1] Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023). Automatic Prompt Optimization with “Gradient Descent” and Beam Search (No. arXiv:2305.03495). arXiv. http://arxiv.org/abs/2305.03495\n\n2. A typo that doesn’t impact clarity too much but ought to be addressed: in page 7, row 353, the results should reference Table 2 instead of Table 5, as Table 5 contains the complete results.", "other_comments_or_suggestions": "Please refer to weaknesses part.", "questions_for_authors": "Please refer to weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: REVOLVE introduces a novel approach to enhance LLM-based optimization by simulating second-order dynamics, moving beyond the limitations of purely first-order feedback methods like TextGrad. The proposed method captures higher-order information by tracking the evolution of responses over multiple iterations, enabling more stable and informed adjustments, which is particularly beneficial for long-horizon optimization tasks. Reviewers generally agree on the merits of the proposed method. We highly recommend that authors incorporate this feedback into the next revision.
Decision: Accept (poster)