metaTextGrad: Automatically optimizing language model optimizers
Authors: Guowei Xu
Mert Yuksekgonul
Carlos Guestrin
James Zou
Venue: NeurIPS 2025 poster ()
Keywords: programming models
prompting techniques
meta-learning
LLM optimizer
Primary area: deep_learning
TL;DR: We propose an approach to flexibly optimize LLM optimizers.

Abstract: Large language models (LLMs) are increasingly used in learning algorithms, evaluations, and optimization tasks. Recent studies have shown that using LLM-based optimizers to automatically optimize model prompts, demonstrations, predictions themselves, or other components can significantly enhance the performance of AI systems, as demonstrated by frameworks such as DSPy and TextGrad. However, optimizers built on language models themselves are usually designed by humans with manual design choices; optimizers themselves are not optimized. Moreover, these optimizers are general purpose by design, to be useful to a broad audience, and are not tailored for specific tasks. To address these challenges, we propose metaTextGrad, which focuses on designing a meta-optimizer to further enhance existing optimizers and align them to be good optimizers for a given task. Our approach consists of two key components: a meta prompt optimizer and a meta structure optimizer. The combination of these two significantly improves performance across multiple benchmarks, achieving an average absolute performance improvement of up to 6% compared to the best baseline.

metaTextGrad: Automatically optimizing language model optimizers
Guowei Xu Tsinghua University
Mert Yuksekgonul Stanford University
Carlos Guestrin Stanford University
James Zou Stanford University
Abstract
Large language models (LLMs) are increasingly used in learning algorithms, eval- uations, and optimization tasks. Recent studies have shown that using LLM-basee optimizers to automatically optimize model prompts, demonstrations, predictions themselves, or other components can significantly enhance the performance of Al systems, as demonstrated by frameworks such as DSPy and TextGrad. However, optimizers built on language models themselves with manual design choices; optimizers themselves are not optimized. Moreover, these optimizers are general purpose by design, to be useful to a broad audience, and are not tailored for specific tasks. To address these challenges, we propose metaText which focuses on designing meta opttiipp further enhance existing optimizers and align them to be good optimizers for given task. Our approach consists of two key components: meta prompt optimizer and meta structure optimizer. The combination of these two significantly improves perfor- mance across multiple benchmarks, achieving an average absolute performance improvement of up to 6% compared to the best baseline.
usually designed by humans
1 Introduction
Large language models (LLMs) are increasingly used in learning algorithms, optimization, and evaluation tasks 1 challenges. Firstly, many of these algorithms are still hand-crafted to a considerable extent, requiring substantial human expertise and effort to design and implement effectively. Second, LLMs notably sensitive to the specific wording and structure of their instructions 6 making it tedious to improve them effectively be used in learning algorithms.
3 4 J. However, algorithms that incorporate LLMs often face significant
Many studies have explored prompt optimization approaches to automatically design better prompts and enhance the performance of L.Mss. For example, algorithms such OPRO 3, MIPRO 77). and TextGrad 1 have introduced optimizers based on LLMS that support automatic prompt optimization. However, these optimizers often fixed, applying the same optimization strategies with the same optimizer prompts independent of the task, lacking a process to align with the specific task. Importantly, these optimizers tasks and be used by large user bases. Furthermore, different optimizers likely excel at different types of optimization tasks; thus, achieving the best of all optimizers introduces either choice be made among them or a strategy to ensemble them. There is no method to automatically design and improve the optimizers to call based on the characteristics of task.
general purpose by design, be useful many different downstream
In this work, we explore how to automatically optimize both the structure and the prompt of optimizers. Specifically, assume that all LLM calls are black-box calls, where the internal states of the LLM,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Task
Accur 53%
Optimized Program
Meta Optimizer
Question: How many Bages
Candidate
Optimized Optimizer
Optimizer
Answer:
624
Optimizers
Program
Average Acouracy:459
Average Acouracy:4
V TextGrad
DSPy
Figure 1: Illustration of the meta-optimization process. meta-optimizer optimizes LLM optimizers by aligning them with specific tasks through task interaction while leveraging the strengths of different optimizers to propose a more effective optimizer.
as well as any information such as model gradients, are inaccessible, and only the inputs to and outputs of the LLM are observable. For the task to be optimized, we require only a small training dataset containing some input-output pairs and evaluation metric the user wants to improve upon.
We propose using meta-optimizer to automatically optimize the optimizer, with the objective of identifying improved optimizers such that the program optimized by it achieves the best performance on the given task. To this end, we introduce two types of meta-optimization strategies: the meta prompt optimizer and the meta structure optimizer. The meta prompt optimizer focuses on refining the prompts of the LLM optimizers to enhance their effectiveness and better align them with specific tasks. Meanwhile, the meta structure optimizer is designed to automatically determine the optimal combination and sequence of different optimizers or modules based on the characteristics of the task.
By combining these two types of meta-oppimmzers, we propose components into unified framework. Specifically, given a of input optimizers, metaTextGrad first performs prompt optimization for each optimizer independently. Then, it explores the combi- nation and sequencing of these optimizers to construct a more effective composite optimizer. We conducted experiments on multiple benchmarks, and the results demonstrate that our meta-optimized optimizers consistently outperform the existing ones.
which integrates both
Overall, we summarize our contributions as follows. First, we introduce the concept of meta- optimization, highlighting that existing LLM optimizers often require further task alignment and effective combination through meta- ceta-oppimiizr to maximize their potential. Second, we develop two types of meta-optimizers: the meta prompt optimizer and the meta structure optimizer. Building on these, we propose metaTextGrad, which integrates both types of meta-optimizers into a unified framework. Lastly, experimental results on multiple benchmarks demonstrate that our method significantly outperforms baseline approaches in both performance and generalization.
2 Problem Statement
Here, we will follow the notation from 77. Consider an LLM program that may contain multiple LLM calls forming pipeline, with each call using different prompt. The pipeline structure of program and the prompt corresponding to each LLM call can be learnable and optimized by an LLM optimizer.
The task of the LLM optimizer is find the optimal program given a training dataset D (with pairs of inputs and outputs y), and an evaluation metric μ..
arg max
Σ (Z.g)ED
(((((,)))
D
Existing methods attempt to approximate solutions to this optimization problem from different perspectives. Optimizers such as MIPRO (7), OPRO [ and TGD [9] aim to find better prompts, while optimizers like ADAS [10 focus exploring structures. We unify the existing optimizer

---

algorithms into general framework, as outlined in Algorithm 1 the same type as of M, are designed by humans, and their structure and prompts are fixed.
Importantly, these optimizers are
Specifically, each optimizer should be capable of performing four operations:
1. Initialize: Given the training dataset and the program to be optimized, prepare the training scheme.
Suggest improvements the program, which may involve modifications the pipeline structure, prompts, or other aspects.
Propose:
3.Update:
Update the current optimal program based on the evaluation results of the improved program and determine the next proposal.
4.
Extract(ptiinzzedPrrrnnn Return the best program found so far.
Algorithm Inner Loop: Optimize & with Optimizer M
Input: Optimizer Initial Program 4, Max Iterations
2: Input: Training Data D. Validation Data Doat, Metric μ
3: Output: Φ² i.e. the optimized version of 4
4: Inittaalizee ②)) do
5: for
6: @x Prropooee)
7: σ
- [Daat] Lanee (1).)
8: M.Uppatee Pk.
9: end for
10: (Ø' (0",0") M.ErrecOOpimmizeePrrgaanm)
11: return (0"
The problem we investigate is how to further enhance existing optimizers and align them to be effective optimizers for given task, instead of relying human-written optimizers. To this end, we introduce the concept of a meta-optimizer. Let the optimized program obtained by an optimizer M be denoted x- M. oottimizeeD, Φ). The optimization objective of the meta-optimizer is to find improved optimizers such that the program optimized by this optimizer performs better on the corresponding task. In particular, the optimization problem formalized as:
M arg maax M
p(MMoptimize(D) (x),)) (2.))ED
Σ
D
In optimization, good initializations often contribute to improved performance, in continuous or discrete optimization alike [2] Ideally, the meta-optimization framework should leverage the existing, manually designed optimizers and achieve further improvements. Therefore, the input to the meta-optimizer can include one more existing optimizers as initialization, ultimately producing an optimized optimizer. The detailed process is illustrated in Algorithm 2
Algorithm 2 Meta-Optimization of Optimizers
I: Input:
Mcta-Optimizer M
2: Input: Max Ieta-ittrration J, Max Inner-Iterations
3: Input: Initial optimizers (L) (2))... ...., (y.,)...,, ()
Input: Training Data Validation Data D
5: Input: Metric μe, Initial Program
6: Output: Optimized optimizer M
7: M. Initialize (D, (
8: for <- to J do
9: M, < Propose)
10: (2),a,) < InnerLoop [M,, 1, Drat,
11: M. M.Update(M,,o)
end for
13: M + M.EExtracOppimiizooopppnnnel)
14: return M
Specifically, the meta-optimizer combines different input optimizers to propose new optimizers. The newly proposed optimizers execute the process described in Algorithm I to obtain the inner loop

---

optimization results. Based on the performance of the newly designed optimizers, the meta-optimizer updates the current best optimizer and determines the proposal for the next iteration (outer loop).
However, similar to the problem of optimizing a program, finding an optimal optimizer using a meta- optimizer is generally intractable optimization problem. Therefore, it is necessary to introduce appropriate parameterizations and simplifications to the problem. In our work, we primarily focus on two types of parameterizations for the meta-optimizer:
automatically optimizing the prompts in the optimizer to align it with the given task (Meta Prompt Optimizer),
automatically optimizing the combination of different optimizers based on the characteristics of the task, forming new composite optimizer (Mela Structure Optimizer).
Collectively, meta-optimize using both parameterizations to obtain better optimizers.
3 metaTextGradi Automatically Optimizing Language Model Optimizers
In this section, we first introduce the theoretical insights behind the meta optimizer. Next, we further analyze and illustrate our motivation through concrete examples. Finally, we present the metaTextGrad pipeline, which consists of the meta prompt optimizer and the meta structure opti- mizer.
3.1 Theoretical Insight
We present the theoretical motivation for meta optimization, highlighting the importance of aligning the optimizer with the target task. In particular, is shown that when both the training and test sets are sampled from the same underlying data distribution, an optimizer properly aligned via meta-leariing on the training set will, with high probability, produce programs on the test set whose accuracy closely approaches that of the optimal optimizer. In contrast, an optimizer that has been optimized on the training set lacks such theoretical guarantee.
We begin by introducing the necessary notation. Let J X + [0, 1] denote the accuracy metric, where X is the space of natural language outputs. Let D be a distribution over natural language inputs x. Let &y be optimizer parameterized by 0 O. Given input and access to p. 4g makes T zeroth-order queries and returns an optimized output æy.
Define the loss of the optimizer input L(0, x) 1 u(ZT) 1 - (z) T)). The
population loss is given by R(0) = Eg-n [L0, }. J. Let S = {xI..... En ~ D" denote a dataset sampled from D, and define the empirical loss S Rg(
Σ T(0.
Let 0g = arg minşce Rs be the optimizer parameters obtained via meta- meta-learning on the training set S, and let 0 = arg mingee R(0) be the globally optimal optimizer under distribution D. We can now state the following thcorem:
Theorem 1. Let S1 and S₂ be datasets sampled independently from distribution D, with sizes and m, respectively. Then, with probability at least 1 the optimizer Os, trained S1 satisfies:
Rs, 0 < R((*)
log(6/8)
log(6/5)
76
2m
The proof of Theorem iis based on Hoeffding's inequality, and the full derivation is provided in Appendix B The theorem highlights the necessity of performing metaaoptimizztion.
In contrast, an optimizer On that hasn't been optimized the training dataset has theoretical guarantee. We can still guarantee that Rs, (0)) similar to R(fp), but the guarantee says nothing about how large R(O,) is. If the optimizer is bad average for this task, it will still be bad S₂ with high probability. This provides a theoretical foundation for the design of metaTextGrad.
3.2 Motivating Example
Existing optimizers such TextGrad and DSPy are manually designed by humans with the goal of performing well across broad distribution of tasks, and they indeed demonstrate strong average performance.

---

Meta Optimizer
Inner Optimization Loop
Initial Optimizers
Optimized Optimizer
Initial Program
DSPPP
1. Structure
Propose new optimizer
TextGrad Structure
You optimization system whose goal is ensure that pipeline generates correct
Proposer
TextGrad TGD
Proposed optimizer
Update best optimizer
You part an optimization system that improves
Updater
Optimized Program
2.TGD
Figure 2:
Illustration of metaTextGrad. metaTextGrad combines meta prompt optimizer and meta structure optimizer. Given a set of optimizers, metaTextGrad performs optimization in two steps. First, it individually refines each optimizer by optimizing its prompts better align with the task. Then, it combines the different prompt-optimized optimizers to construct the final optimizer.
For example, the prompt used by the TextGrad TGD optimizer is as follows:
TextGrad TGD optimizer Prompt
You are part of optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable.
As can be seen, this prompt is indeed highly general, with phrasing such as 'improve prompts, solutions to problems, code, any other text-based variable'.
However, in some cases, such penerol-purpose prompt fails to effectively optimize model performance. While the optimizers in both TextGrad and DSPy can obtain learning signals from task-specific evaluators, these signals tend to noisy and sparse
First, when feedback is provided solely in the form of scalar scores rather than textual guidance, becomes sparse, making optimization substantially more challenging. Second, although optimizers such as TextGrad's s TGD can accept textual gradient feedback, such feedback is often highly noisy, making difficult for the optimizer learn effectively. For example, the evaluator feedback received by the TGD optimizer may look like the following:
Feedback Received by the TGD optimizer
To improve the prompt for the executer and enhance the objective function, consider the following feedback: 1. Explicit Criteria Definition*:; The prompt should explicitly instruct the executer define the criteria for optical activity at the beginning of the response. This can prevent ambiguity and ensure that the executer uses the correct scientific principles. For example, the prompt could include a directive "List the criteria for optical activity before analyzing each compound." 2. **Data Verification Directive**; Incorporate a step in the prompt that requires the executer verify the input data against reliable sources. This could be phrased as "Cross-check the properties of each compound with trusted chemical database before proceeding with the analysis." #*Structured Logical Reasoning**: Encourage structured approach to reasoning by breaking down the analysis into distinct steps. The prompt could suggest a format like "For each compound, first identify chiral centers, then assess symmetry, and finally determine optical activity".
5

---

Feedback Received by the TGD optimizer (continued)
4. **Cooss-efeeeenn Encouragement ... By incorporating these elements into the prompt, the executer can be guided to produce more accurate and reliable responses, thereby improving alignment with the ground truth answer and enhancing the objective function.
Here, the evaluator feedback includes suggestions such as 'List the criteria for optical activity before analyzing each compound', which are specific single problem instance. Such feedback is clearly noisy. Since a generic optimizer relies solely on feedback to guide its updates, it is likely to incorporate such suggestions into the optimized LLM program prompt, which be detrimental to the overall performance of the program.
Yet, in reality, we can choose to let the LLM optimizer adapt specific task distribution in order to achieve better performance. This is because if the distribution of programs generated by the task-specific optimizer is more aligned with the task requirements, the difficulty of finding the optimal program will be significantly reduced, and the impact of noise in the evaluator's signal will greatly mitigated, even when the feedback is noisy.
We illustrate this using the BBH Dyck Languages task as example, showing that aligning the LLM optimizer to specific task distribution can lead to improved performance. For instance, consider the following spec-fp optimizer prompt:
A Task seek-specific Optimizer Prompt
You are part of an optimization system specialized in improving prompts for bracket matching and sequence completion tasks. Your role to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets (, <, O). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures, (3) handling multiple bracket types simultancously, and (4) ensuring complete closure of all open brackets. You should critically analyze how the prompts can better guide the model to track open brackets, maintain proper nesting order, and systematically complete sequences. Consider incorporating pattern recognition strategies and explicit validation rules in the improved prompts. Your improvements should lead to more reliable and accurate bracket sequence completions.
can observed that the task task- specific optimizer leads to a shift in the distribution of LLM programs it tends to optimize, making it more likely to generate content that aligns with key task requirements, such as producing programs that satisfy the requirement preserving the LIFO (Last In First Out) order, etc. As result, even if the evaluator feedback is somewhat noisy or sparse, the LLM program optimized by the optimizer can still perform well, and is more likely to generate critical statements such as: Explicitly push each opening symbol onto the stack and pop it when a corresponding closing symbol encountered. After processing each symbol, describe the current state of the stack, focusing unmatched opening symbols. In contrast, if generic optimizer is used, it becomes difficult to generate effective prompts under noisy or sparse feedback conditions.
This demonstrates that adapting to new task distribution is meaningful. To avoid adapting to each new task distribution by hand, we meta-learn how to adapt, which is the core motivation of met metaTexttrad.
3.3 Pipeline
3.3.1 Meta Prompt Optimizer
LLM optimizers are usually designed by humans with manual design choices. As a result, the optimizers themselves are not optimized aligned with given task. We aim to optimize the prompts of LLM optimizers to further enhance their effectiveness and align them with tasks.
Below is a brief introduction to the implementation method of the meta prompt optimizer. The detailed pseudocode can be found in Appendix A. To Initialize, the meta prompt optimizer runs round of optimization the validation dataset evaluate and record the initial performance of

---

the optimizer. To Propose, the meta prompt optimizer randomly samples data examples from the training dataset and analyzes the general characteristics of the task type. Based on the current best optimizer prompt, it proposes an improved prompt that is more aligned with the task. To Update, the optimized optimizer undergoes an inner loop optimization test on the validation dataset. If the test results outperform those of the previously best optimizer, the optimizer is updated. To paarractQpmiized(pimimmi
the meta prompt optimizer returns the best optimizer learned so far. Among these steps, Propose is the core of the meta prompt optimizer and the only stage where LLM calls are invoked. For detailed prompts, refer to Appendix D.I
3.3.2 Meta Structure Optimizer
variety of LLM optimizers have been proposed to optimize program structures, prompts, and other components. The meta structure optimizer is designed to automatically optimize the combination and ordering of different optimizers based on the characteristics of the task.
Below, we briefly introduce the working principles of the meta structure optimizer. The detailed pseudocode implementation can be found in Appendix A.2 To Initialize, the meta structure optimizer runs one round of optimization for each input optimizer on the validation dataset, selects the highest score and the corresponding optimizer as the initial best value. To Propose, we provide the meta structure optimizer with a set of reference optimizers. If previously optimizd, setter-performing optimizers exist, they are also included. Based on this, the meta structure optimizer integrates and proposes an improved optimizer. To Update, the meta structure optimizer evaluates whether the proposed optimizer shows improved performance on the validation dataset and updates the current best optimizer and score. To the best optimizer learned so far. The Propose stage is the only stage with LLM calls. For detailed prompts, please refer to Appendix D.2
the meta structure optimizer returns
3.3.3 metaTextGrad
In the previous two sections, we introduced the meta prompt optimizer and the meta structure optimizer. metaTextGrad is composed of these two components. Specifically, after receiving a of input optimizers, metaTexttrad first performs prompt optimization for each optimizer individually and then explores the combination of different optimizers form better composite optimizer.
4 Experiment
4.1 Experimental Setup
In this section, we the existing optimizers from DSPy and TextGrad baseline methods. We evaluate our approach and the basclines on multiple benchmarks, including BBH [13] [4].
MMLU [15), and GPQA (16). To ensure reproducibility, we provide the prompts and structures of the learned programs in Appendix E
Baselines. We used zero-shot CoT, few-shot CoT 17]. self-consistency [18] best-of-N 19 20
MIPROv2 77 TextGrad TGD ① and ADAS-TC TG 10 as baselines. Zero-shot CoT relies on direct chain-of-thouge reasoning, whereas MIPRO and TextGrad's TGD optimizer refine the program's prompt. ADAS is an algorithm for automatically searching and optimizing the program's structure. We implemented this algorithm within TextGrad as baseline, referred to as ADAS-TG.
We evaluate our method on four widely used, challenging, and diverse benchmarks: BBH Word Sorting, BBH Dyck Languages 13 14,, MMLU Abstract Algebra [15], and GPQA Diamond 16.. For detailed settings, please refer to Appendix C Due to the non-determinism of
Benchmarks.
LLM APIs (21). the test accuracy for each benchmark is averaged over five random seeds.
In our experiments, we consider three different levels of LLM calls: (1) LLM calls within the program itself, (2) LLM calls made by the optimizer while refining the program, and (3) LLM calls made by the meta-optimizer when optimizing the optimizer. As shown in Section 4.3 the frequency of these calls decreases significantly across these levels. This hierarchical structure allows for cost-effective resource allocation: the program should use a relatively economical model, the optimizer can leverage more capable model, and the meta-optimizer should employ the best available model. Consequently,
7

---

Method
Word Sorting
Dyck Languages
GPQA Diamond
Abstract Algebra
Average
Val
Test
Val
Test
Val
Test
Val
Test
Val
Test
Vanilla prompting methods
Zero-shot CoT
0.46
0.55
0.06
0.05
0.32
0.34
0.74
0.70
0.40
0.41
8-shot CoT
0.50
0.52
0.14
0.19
0.32
0.35
0.65
0.71
0.40
0,44
Self-consistency (8)
0.47
0.52
0.10
0.12
0.40
0.42
0.76
0.70
0.43
0.44
Best (8)
0.48
0.52
0.14
0.17
0.37
0.77
0.74
0.44
0.46
TextGrad optimizers
TGD Optimizer
0.54
0.55
0.10
0.10
0.34
0.35
0.76
0.71
0.44
0.43
ADAS-TG
0.58
0.58
0.21
0.16
0.36
0.37
0.75
0.70
0.48
0.45
DSPy optimizers
Zem-shht MIPROv2 0.57
0.55
0.19
0.16
0.43
0.38
0.76
0.77
0.49
0.47
8-shot MIPROv2 0.52
0.57
0.33
0.26
0.37
0.34
0.74
0.65
0.46
Meta-optimized optimizers
metaTextGrad
0.60
0.65
0.42
0.37
0.45
0.40
0.78
0.71
0.56
0.53
Table 1: Accuracy (%) of GPT-4o-ming on benchmarks. Bold indicates the best result, and underlined text represents the second- best.
our experiments, we use GPT--4-mini for LLM calls within the program, GPT-4o for the MIPROv2 and TGD optimizers, and the ol model for the structure optimizer and the meta-optimizers.
4.2
Main Results
As shown in Table [ we achieve up to an 11 % absolute performance improvement across these datasets, with an average performance significantly surpassing existing optimizers. Our method outperforms both TGD and ADAS-TG, which serve as the base candidates for metaTextGrad, across all benchmarks and achieves the best performance on most them. It's worth noting that the programs optimized by the meta-optimized optimizers also exhibit interesting properties.
(1) The optimized optimizer is more aligned with specific tasks. For example, in the BBH Dyck Languages task, the generated program includes components such as n rype analyzer, and a stack validator, which closely match the nature of the task. In contrast, unaligned optimizer tends to propose more generic and broadly applicable structures.
(2) The optimized optimizer is more effective in handling finer details. For instance, in multi-step LLM calls, passing both the overall problem and subproblems to each LLM call helps maintain global understanding throughout the process. This behavior is frequently observed in the optimized optimizer, whereas the initial optimizer tends to pass only the subproblems to each subpart.
(3) The meta veta-oop optimizer generally improves efficiency. Although we allocate six opti- mization steps per training epoch, we observe that meta-optimized optimizers, due to their stronger task alignment, often achieve significant improvements within the first 1-2 steps. In contrast, other optimizers show more gradual improvements.
Please refer to Appendix
D for the optimized programs and Appendix for the optimized optimizers.
4.3
Cost analysis
In this section, we examine the trade-off between effectiveness and computational cost.
First, we analyze the token usage at different levels within single epoch of meta optimization. This analysis supports our design choice of using models with different capabilities at different levels. As shown in Table the token usage per optimization epoch on the MMLU Abstract Algebra dataset reveals that higher-level components require significantly fewer tokens than lower-level ones. This justifies our hierarchical design. For comparison, a single round of zero-shot CoT requires approximately 140k tokens, indicating that the overall token consumption of our optimization pipeline remains within reasonable and practical range.

---

Level
Tokens
Model
Performance
Cost
Program level
400k
O-shot CoT (4o-mini)
0.05
0.14S
Optimizer level
100k
Ours (4o-mini)
0.37
0,44S
Meta-optimizer level
2.5k
0-shot CoT (4o)
0.18
0.52$
Table 2: Token analysis on Abstract Algebra.
Table 3: Cost analysis on Dyck Languages.
In addition, evaluate the cost and performance of the zero-shot CoT approach using both GPT-4o- mini and GPT-4o on the BBH Dyck Languages dataset, and compare them our optimized approach applied to GPT-4o-mini. As shown in Table 3 our method achieves the best performance BBH Dyck Languages while incurring a lower cost than GPT-4o. This demonstrates that with effective prompt and structure optimization, smaller model can outperform the zero-shot performance of larger model. These findings highlight the practical applicability and scalability of our approach.
4.4 Transferability of the optimized optimizer across models and datasets
In this section, we evaluate the transferability of our optimized optimizer across different language models and datasets. As shown in Table 4 our method trained on GPT-4o-mini achieves superior performance compared to unoptimized baselines when evaluated on Claude 3 Haiku. Furthermore, as illustrated in T our optimizer trained on the GPQA diamond dataset also transfers effectively to the Abstract Algebra dataset. These results demonstrate that our meta-optimized optimizer exhibits strong transferability across models and datasets.
Method (Claude Haiku)
Dyck Languages
Method
Abstract Algebra
Val
Test
Val
Test
Zero-shot CoT
0.07
0.10
Zero-shot CoT
0.74
0.70
TextGrad optimizers
TextGrad optimizers
TGD Optimizer
0.10
0.04
TGD Optimizer ADAS-TG
0.76
0.71
ADAS-TG
0.35
0.34
0.75
0.70
Optimizers optimized on GPT-44-mini
Optimizers optimized on GPQA diamond
metaTextGrad
0.32
0.35
metaTextGrad
0.78
0.77
Table 4: Transferability of the optimized opti- mizer across models.
Table 5: Transferability of the optimized opti- mizer across datasets.
4.5 Analysis of the effectiveness of each meta optimizer
In this section, we analyze the contributions different components of the proposed meta optimizer. As shown in Table 6 we find that all meta optimizers improve performance the BBH Dyck Languages benchmark. Optimizers optimized using either method outperform the original optimizer. Among them, the meta prompt optimizer achieves the best improvement when applied ADAS-TG.
Split 0-shot CoT TGD ADAS-TG TGD (0) ADAS-TG (0) Struct (0) metaTextGrad
Val
0.06
0.10
0.21
0.21
0.42
0.24
0.42
Test
0.05
0.10
0.16
0.24
0.37
0.16
0.37
Table 6: Analysis of the effectiveness of each meta optimizer on Dyck Languages, TGD (O), ADAS- TG (O), and Struct (O) respectively denote the TGD and ADAS-TG optimizers enhanced by the meta prompt optimizer, and the optimizers enhanced by the meta structure optimizer.
Here, metaTextGrad produces the same results as the optimized ADAS-TG because the meta optimizer did not find better option during the meta structure optimization. So, the optimized ADAS-TG was selected as the best result. Notably, the best meta optimizer varies across benchmarks due to task-specific differences. Despite this, all meta optimizers can effectively enhance the performance of given optimizer.

---

5 Related Work
5.1 Prompt optimization
Prompt optimization has proven crucial for improving LLM performance. Initial strategies, such as few-shot learning and in-context learning, demonstrated careful prompt design could significantly boost LLM performance [22]. Techniques like chain-of-thought reasoning [T7] and ensemble methods (33 also emerged as popular ways structure prompts for effective problem-solving. However, hand-crafted approaches are limited in their utility. Efforts to automate prompt optimization have led to the development of gradient- grasient-based approaches [2 25] These methods, however, require access to model parameters, which limits their application to open-source models.
To address these constraints, alternative approaches have been proposed that leverage LLMs them- selves as prompt optimizers. APE [26) was among the first to introduce the concept of automatic instruction generation and selection. (77] introduced the concept of meta prompt, demonstrating systematically designing pettapsrmpps can improve prompt quality. DSPy 4 proposed a systematic approach for optimizing LLM programs, integrating structured optimization techniques.
Another line of research explores optimization methods based on textual feedback. ProTeGi 28 first introduced the concept of textual gradients, highlighting that LLMs themselves can serve as form of a loss function to guide the improvement of LLM programs. Building on this idea, TextGrad [[] developed a structured textual gradient descent framework, demonstrating its applicability and effectiveness across multiple disciplines and domains. Concurrently, OPTO (29] proposed a related approach in which the optimizer receives an execution trace alongside feedback on the generated output. Semantic gradient descent [30] refines textual gradient descent by enhancing feedback signals.
However, the LLM optimizers proposed in these methods remain fixed during the optimization process, limiting their effectiveness and adaptability. Our approach addresses this limitation by leveraging meta-optimizer to align LLM optimizers with the task.
5.2 Classical meta-learning
Gradient-based meta-learning algorithms such as Model-Agnostic Meta-Learning (MAML) KI First-Order MAML, Reptile (32), and ANIL [33] cast learning to learn as finding a good initialization that can be fine-tuned with only handful of gradient steps. MAML jointly optimizes across tasks through a bi-level procedure, explicitly encouraging large improvements after one or two inner-loop updates. Reptile shows that simpler first-order update suffices, while ANIL's ablation studies reveal that the bulk of MAML's gains stem from feature reuse rather than rapid weight adaptation, allowing the inner loop to be removed for all but the task task-specific head. Despite their successes, these methods utilize fixed optimizers. In contrast, we focus on optimizing L-s-ed optimizers rather than classical ones, and we meta-learn the optimizer so that the optimization strategy is aligned with each new task, rather than merely learning a good initialization.
Conclusion
In this paper, we propose metaTextGrad, meta opttmizttiiiztoonn framework that enhances existing LLM optimizers by aligning them more effectively with tasks. Our method introduces two key components: the meta prompt optimizer, which refines optimizer prompts for better task adaptation, and the meta structure optimizer, which determines the optimal combination of different optimizers. By integrating these two components, metaTexttrad improves the efficiency of LLM optimizers, leading to better performance across a diverse range of benchmarks.
Looking ahead, there are several promising directions for future research. First, even the meta optimizer we proposed can be optimized. In particular, our meta optimizer is designed by ourselves, instead of learned by data, and there are techniques in the meta learning literature that could be adapted to allow this [34 335. Second, future work can explore different ways to parameterize the optimizers. This study primarily focuses on refining optimizer prompts and their composition, but meta-optimizers could also be leveraged to automatically enhance the optimization algorithms employed by existing optimizers. We believe that optimizing L.L optimizers is a crucial step toward further improving the performance and task alignment capabilities of L.L--driven systems and has the potential to provide valuable insights to the research community.
10

---

References
[1] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.0799. 2024.
[2] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[3] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024.
[4] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari. Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into state-of-the-an pipelines. In The Twelfth International Conference on Learning Representations, 2024.
[5] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[6] Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. Can large language models be trusted for evaluation? scalable meta-evaluation of Ilms evaluators via agent debate. arXiv preprint wrXiv:2401.16788 2024.
[7] Krista Opsahl-Ong. Michael Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs, 2024.
[8] Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2022.
[9] Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint satisfaaionn lens on factual errors of language models. The Twelfth International Conference on Learning Representations, 2024.
[10] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint 20l.:24008.04335. 2024.
[11] Qian Li, San- Yang Liu, and Xin-She Yang. Influence of initialization on the performance of metaheuristic optimizers. Applied Soft Computing, 91:106193, 2020.
[12] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference machine learning. pages 1139-1 1147. PMLR, 2013.
[13] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thoughhe can solve them. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, Toronto, Canada, July 2023. Association for Computational Linguistics.
[14] Aarohi Srivastava, Abhinav Rastogi, and Abhishek Rao et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.

---

[15] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021.
[16] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: graduate-level google-pros proof q&a benchmark. In First Conference on Language Modeling, 2024.
[17] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01124824244833 2022.
[18] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le. Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, Self-consistnney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learming Representations, 2023.
[19] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[20] Steve Webb. The physical basis of imrt and inverse planning. The British journal of radiology, 76(910):678-689, 2003.
[21] Sherman Chann. Non-determinism in gpt-4 is caused by sparse moe. https://152334h
hitthb..///bggg/oooodeeeeemiiita gptt-// 8 2023.
[22] Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi. Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpont tuning? case study in medicine. arXiv preprint arXiv:2311.16452 2023.
[23] Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! survey collaborative strategies in the era of large language models, 2024.
[24] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4222 4235, Online, November 2020. Association for Computational Linguistics.
[25] Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web conference 2022, pages 2022.
[26] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023.
[27] Qinyuan Ye, Maxamed Axmed, Reid Pryzant, and Fereshte Khani. Prompt engineering a prompt engineer. arXiv preprint 2r3 11-22311..0566 1.05661, 2023.
[28] Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 7957-7968, Singapore, December 2023. Association for Computational Linguistics.
[29] Ching-An Cheng, Allen Nie, and Adith Swaminathan. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llm.. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 71 71596-71642. Curran Associates, Inc., 2024.
12

---

[30] Wenyi Wang, Hisham A. Alyahya, Dylan R. Ashley, Oleg Serikov, Dmitrii Khizbullin, Francesco Faccio, and Jürgen Schmidhuber. How correctly do semantic backpropagation on language- based agentic systems, 2024.
[31] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17. page 1126-1135 JMLR.org, 2017.
[32] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
[33] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representationss 2020.
[34] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
[35] Jürgen Schmidhuber. Evolutionary Principles in Self-Referential Learning. PhD thesis, Techni- cal University Munich, 1987.
13

---

A Implementation Details
In this section, provide the implementation of the meta optimizers. The LLMCall component generates responses based on the meta optimizer's prompt. The detailed prompts will provided in the next section of the appendix.
A.1 Meta Prompt Optimizer
Algorithm 3 Meta Prompt Optimizer
1:
function M Initialize (D, M)
2:
store(D)
.M" <- .M
4:
o² 0
5: return
6: end function
7: function M. Propose(
question, answer D
9: proposedOptimizerr.e + (proopl M", question, answer)
10: return proposedOptimizer
11: end fuuction
function M. M. Up..utteM,,))
13:
iT >> o² then
14:
M + M
15:
t
16: end if
return
18:
end function
19:
function M. ExxerecOOppiiiiccppppiiieee
20:
return M
21:
end function
A.2
Meta Structure Optimizer
Algorithm 4 Meta Structure Optimizer
1:
function M (nitialiizeeD, (LI)))
store (D, (M)
3:
.M" None
4:
o" +
5: refurn
6: end function
function M Propose()
8: proposedOptimizer +- LpMoall A1,, (M,
return proposedOppimizer
10: end function
11: function M. M.Uppatc(M,o)
12: if o" then
13: M M
<-a
15: end if
16: return
17:
end function
function M. Extrac(Opiimizzo(ppiiiieee)
19:
return M
20:
end function
A.3
Meta Optimizer
For the meta optimizer, the execution process consists of two steps: (1) Each optimizer is individually optimized using the meta prompt optimizer. (2) The optimized optimizers are further refined using the meta structure optimizer.
14

---

B Proof of Theorem 1
Theorem (Restatement of Theorem [ Let S1 and S2 be two datasets sampled independently from distribution D, with sizes and m, respectively. Then, with probability at least 1 8, the optimizer 0s, trained S satisfies:
Rs, ( S R(A") +
log(6/6)
(6/(6/)) 2m
Proof. According to Hoeffding's 's inequality, for any 0 < < 1:
Pr S [((B) Rs, (0)) > cn( 1
(4)
where:
(0)
10/(2/6)
(S)
2n
Using inequality 4 we can bound the population risk of the learned optimizer. With probability at least 1
3
R(O)= - Rs. (0) + en 8 < Rs, (0*) + enl R(0') + 2 26
En(g)
(6)
where 0 = arg ming R(0).
With union bound, with probability at least - 8:
8 [) s R(0*) + 2n ( 5 Rs. (0) - R() Em (
em mm(g)
(7)
In conclusion, with probability at least d over the draws of S₁ and Sy:
Rs, (0) S R(0") + 2 log(6/8)
log(0/6)
(8)
C Details Experimental Benchmark Setup
The BBH task 13 14 is a challenging benchmark that requires precise reasoning across various tasks. We select two subsets from the BBH benchmark: BBH Word Sorting and BBH Dyck Languages. The BBH Word Sorting task requires the language model to sort a given set of words in order, while the BBH Dyck Languages task involves providing string composed of various types of brackets and asking the model to determine the characters needed to complete the bracket pairing. We the same vrain/validation/test splits in TextGrad [I] (i.e., 50, 100, and 100 instances for training, validation, and testing) and follow the TextGrad approach by using GPT-4o to evaluate whether the model's output is correct, based the predicted and ground truth answers.
BBH.
The MMLU benchmark [15] evaluates a model's ability to answer questions across wide range of scientific disciplines. We selected the MMLU Abstract Algebra dataset and created training, validation, and test sets consisting of 10, 50, and 40 questions, respectively. We assess model accuracy using exact matching. determining correctness by applying regular expression check whether the model's response contains "Answer: [A-D]" and matches the ground truth. However, we observed that the DSPy baselines struggle to adhere to this output format. accommodate this, we use GPT-4o to evaluate the correctness of responses generated by DSPy-based methods.
MMLU.
GPQA Diamond. The GPQA Diamond benchmark [16] evaluates the model's ability to solve graduate-level, Google-proot questions. The benchmark consists of 198 questions, which we split into training, validation, and test datasets containing 30, 100, and 68 questions. Similar to MMLU, we use exact matching to determine response accuracy by applying regular expression to check whether the model's answer follows the format "Answer: [A-D]" and matches the correct answer. For the DSPy baselines, we use GPT-4o to evaluate the correctness of
responses.
15

---

D Prompts of Meta Optimizers
In this section, we provide the prompts used the propose stage the meta prompt optimizer and the meta structure optimizer. The propose stage is the only part of the meta optimizer that involves an LLM call; all other functions consist of direct numerical updates or result retrieval, as explained in pseudocode in Appendix
D.1
Meta Prompt Optimizer
Below, we present the prompt used for the meta prompt optimizer.
Meta Prompt Optimizer
Task Requirement
A TextGrad optimizer optimizes TextGrad pipeline so that the pipeline can generate better outputs based on inputs.
A TextGrad pipeline consists of several agents, each of which has specific role in the optimization process, which defined by different prompts. You will be given general task description of optimizer and the specific task the pipeline aims to solve.
Your task is to propose improved optimizer task description so that the optimizer can better optimize the pipeline for the given task.
# Optimizer Code
Here is the source code of the optimizer: (just for reference) (optimizer_ Saweniieerssuuee_eeeeeee soure__cdee] The task of the optimizer
Specifically, the pipeline aims solve
this
kind of question:
(exam-
ple_set..ge set.get__qesseeee An example of the task is provided here:
typeO).
Question:
Answer: sxtample_answedt
The LLM optimizer wants to improve LLM pipeline to solve such kind of problems. # Current Task Description
Here is the current task description of the optimizer, which you can improve: (opti- mizer_prompt]
# Your task
You should identify what the optimizer should pay attention to in order to improve the (optimizer ttper of the pipeline for solving the given task.
Conduct a detailed analysis of the given example, and respond in the following format:
"jsonn
"improved_tae task liiikdddsssstn'..... "...", ".". Your improved task description for the optimizer
D.2
Meta Structure Optimizer
Meta Structure Optimizer
Task Requirement
A TexttGrd optimizer optimizes TextGrad pipeline so that the pipeline can generate better outputs based on inputs.
The structure and prompts of current optimizers fixed, and you will be given the source code of some optimizers.
Your task is propose an improved optimizer code to better optimize the pipeline. This can be achieved by integrating the reference codes of the given different optimizers and merging them into new optimizer.
16

---

Meta Structure Optimizer (Continued)
# Task Details
You will be provided with implementations of several optimizers, each annotated with their respective purposes.
Carefully read through their functions and implementation details. Your task is to integrate the different implementation approaches provide more optimized solution. In this context, the step() function refers to the process of performing a single optimization step.
You may notice that different optimizers are suitable for different purposes and should be applied in a specific order.
For example, if the pipeline aims to optimize the structure, this optimization will overwrite the prompts of each previously optimized component, so the structure optimization should be executed first.
Your implementation of the step( function should take the order into account, and each call to the step( function should only optimize single aspect of the pipeline because the gradient context is not reusable.
Therefore, you need implement a logic that the optimizer strategies in a sequential manner, with each strategy being applied self.epoch times before moving to the next one.
different optimizing
(optimizer_prone pproppr_ppppp)
# Code Reminder
You should include the following necessary imports at the beginning of the code: import inspect
import copy
from typing import Type, List, Union
from collections import defaultdict
from textgrad.varian variable import Variable
from textgrad import logger
from textgrad. engine import EngineLM
from textgrad. wwt..oopmmee...iiiizzz....
import Optimizer, gewgggrattadddeeeeeeee context_tteeeeeeeeeeeeeeeee text
from textgrad. modeel import Pipeline
from autograd import LMMlll
from textgrad.come import validate cirerreennnn__eeeeeeeeeeeeeeeeee _oroe_________ Bel__oo_______eeeee
get_default
from
import construct_ tgd___dd______p pro_pp, SYPELINESSSTEEM_PRRE PREFIXX
OPTIMIZER SEAMIIEE_SYSTEEE SYSTEM_PROMPPT,
HIPELINESSYSTER SUFFIX Note
You required to build features from the others, instead of starting from scratch.
of the existing optimizers and improve it by integrating
You should only include the ImprovedOptimizer (inherited exactly from Optimizer) class in the code, and no other classes or functions.
You should implement all the necessary functions and attributes in the ImprovedOptimizer class to ensure that the code can be executed without errors.
You should modify input signatures of the init_____ and step functions when implementing the ImprovedOptimize class.
Please output the complete improved Python code, and make sure to call the improved class "ImprovedOptimizer" remember to also include all necessary attributes of the class so that the code can be executed without errors.
Do not include any additional comments or unnecessary text in the output.
17

---

E
Optimized Programs
To facilitate the reproducibilityy of our work, we provide the program generated by the optimizer optimized with metaTextGrad.
E.1
BBH Word Sorting Benchmark
BBH Word Sorting
class SimplePipeline (Pipeline):
task_description =
"You will answer reasoning question. Think step by step. "The last line of your response should the following
format: Answer: VEVALUE'
"where VALUE is the answer to the question."
def initt__ (self, engine: Union [EngineLM, str] = None, path=None); init_ init
super
path-
self. planner_prommppppeeeeeeaa Variable(
"Create step by step plan for the question. Provide each step on a new line.",
requires_grad-Trus role_descrippiion=pplaaee prompt"
self pllanerr = tg. BlackboxLLMMsslf... engine, self calanner_prompee
self subsolver_prompt Variable(
"Solve this sub-step in detail
without giving the final answer.",
requires-grad-Tiee lere_descriptioonasubbeee solver prompt"
self. subsolver tg. BlaekbooLLLMMself..... engine,
self subsolver_prompee
self www.fllannnnnpaae Variable(
"""Combine all reasoning into a coherent final response, ending with the format: Answer: $VALUE""",
requires_grad-True, tole_descriptiion=finall answer prompt"
self finalsolver tg. BlackboxLLMMseef engine, self filalllaaonnlllll
def _step_sppll cite_ssplitrrueeeeee text: str):
return re. sp.stl \n*,, text sext.strip()
def fsrwa question: Variable) -> Variable:
plan self planner (question)
steps = Split() (self sttp:l__ssepsspitttiui plan) sub_ solutiins []
for step in steps:
sub_solutions. append ) aggrogated
Aggregate () (sub_solutions
final_ self finalsolver (Aggregate () ([question,
aggregated_reasse rresoonnnnn))) return final_answer answer
18

---

E.2
BBH Dyck Languages Benchmark
BBH Dyek Languages
class SimplePipeline (Pipeline): [imppeeeppelime(ppeelene::: task_descripp cask_description = """You will answer a reasoning question. Think step by step. The last line of your response
should be of the following format:
'Answer: SVALUE' where VALUE is the answer
question. """
def (selff engine: Union [EngineLM, str] None, path-None); (engine (engine=engine, path-path) if not path:
self www.iinnnllpppl. = tg. BlackboxLLM
self. engine,
Variable(
"""Break down the Dyck sequence validation into detailed steps. Format as numbered steps: 1), 2), etc. """,
cole__escrripiion"ccrr creates detailed analysis plan" role_
self. type_analyzer = tg B3.BlackboxLLM
self engine,
Variable(
"""Identify and categorize bracket types.
List each type and its corresponding closing bracket. Format: 'Types:
[pairs]'nm!
Soquires_grad-Trig
role_ vole_description aaappiiiiinnaayeeeesssses bracket types and pairs"
self stack_valideee tg. BlackboxLLM
self self.engh engine,
Variable(
"""Simulate stack operations for bracket matching. Show stack state after each operation.
End with 'Answer: Valid/Invalid' requires_grad-True,
role_ role_desciippioo"""errrrro stack-based validation"
self
= tg. BllackboxLLM
self. engine,
Variable(
"""Analyze nesting hierarchy. Check if inner brackets close before outer brackets.
End with 'Answer: Proper/Improper requires.grad-Trrue, roleedescripption""nall nesting hierarchy"
19

---

BBH Dyek Languages (Continued)
self @epth_checckee tg. BlaakboxxLLM
self. engine,
Variable(
"""Calculate mazimum nesting depth and verify balanced structure. End with 'Answer:
Balanceddaadddeeeee Yes/No '""",
role doll_description="ee checks nesting depth and balance"
self sequence_ validatto = tg. BlackboxLLM
self self.enp engine,
Variable(
"""Validate sequence completenes and correctness.
End with 'Answer: Complete/Incompleee
requires_grad-Thne role golledeescriptioo=="vllaaaa sequence completeness"
self final_evaluator tg. BlackboxLLM
self. engine,
Variable(
"""Synthesize all analysis results and determine if this a valid Dyck sequence.
End with 'Answer: Yes/Nlo
requires_graddTruu
role_descripte mmakes final validity determination'
)
def (srw, question: Variable) Variable:
plan self seattialpplaner(( (question) analysis_steps Split() (lambda x: re. splii(tr x), plan)
type_analysis self self.type__... awa.aaayyyeeauestetea
stack_valle validation self. stwsssstaaallls validator( Aggregate() ([question, aope_analssis]
basting_analysee self nesting.amalyzer(
Aggregate ([question, stack_validation))
depth_analysie analysis = self deppt_echeceeeeeeeeeeeeeeeeeeeee Aggregate() ([question, aasting_analysis).
sequence_
requence_valliattion validation self sequence, validator (
Aggregate() (lquestion, (ppth_analysis]]
20

---

BBH Dyek Languages (Continued)
final_resulk self final_evaluutorr
Aggregate ([
question,
stack_validation, nesting_analysian depth_analysin
@equence_validattoe
})
return Extract() () (r" ArnAneer: (.+),, final_resulf
E.3
GPQA Diamond Benchmark
GPQA Diamond
class SimplePipeline (Pipeline): (Pipeelln)l task_descripptin = """You will answer reasoning question. Think step by step. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is question." """
the answer
def __init__ engine: Union [EngineLM, str] None, path-None): super _nnittit____ (engine-enging path-path) if not path:
self executer textgrad BlackboxLLM self belf.engine engine,
textgrad Variable(
Bextgrad.Variabled
"""You will answer multiple choice question. Begin by identifying and listing all key details and constraints from the problem statement. Use explicit reasoning steps to outline the solution, incorporating relevant scientific principles such reaction mechanisms or stereochemistry. Verify your initial conclusions by cross- pross-checking each step against the problem's requirements. Consider alternative answers and evaluate why they might be correct or incorrect.
Provide a clear justification for your answer choice, and rate your confidence in the final answer on scale from your rationale. The last line of your response should be of the following format:
10, explaining
'Answer: $VALUE' where VALUE is one of ABCD. """, Sequires_grad-Trng role delle_descriptio==""" prompt for the executer, which aims solve the task"
def (sel,, question: Variable) -> Variable: return self executer (question)
21

---

E.4 MMLU Abstract Algebra Benchmark
MMLU Abstract Algebra
class SimplePipeline ( (Pipeline):
task_ desskdeecriition = "Y"You will answer reasoning question. Think step by step. last line your response should be of the following format:
:nswwe:: $VALUE' where VALUE is the answer to the question. """
def __innit__ engine: Union [EngineLM, str] None, path=None): super _wniitttt_ti__iiiii (engiiitennnnnneeeeeee (engine-enging engine, path-path. if not path:
self executer textgrad BlackboxLLM
self engine, textgrad.Variable
textgrad Variable(
"""You will answer multiple choice question. Analyze each statement separately and provide your reasoning. Use clear, logical steps, verifying each sub-component of the problem systematicallyy Present each statement distinctly using bullet points or numbers, ensuring the final conclusion is clearly separated from the statement evaluations. Include any assumptions or contect necessary for each conclusion. After evaluating each statement, reezamine your conclusions
to confirm their correctness. Introduce a summary verification step before concluding. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is one of ABCD. """
role deeeeddecriptionno "pommp for the executer, which aims solve the task"
def fsrward(self, question: Variable) -> Variable: return self, executer (question)
F
Optimized Optimizers
F.
Optimized TGD Optimizer Demo
The TGD optimizer generic, so its optimized program offers only general guidance like "ensuring that every opening bracket has corresponding closing bracket," without task-specific strategies. In contrast, the optimized TGD aligns closely with the task, emphasizing ideas like "preserving the LIFO (Last In First Out) order of nested structures." Since the optimized TGD optimizer is better aligned with the task, its optimized program also effectively implements the LIFO principle, making it more effective. This is why task-specific prompt optimizer necessary.
TGD Optimizer prompt
You are part of an optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, any other text-based variable.
22

---

Optimized program prompt
You will answer reasoning question. Think step by step, ensuring logical consistency and accuracy. Explicitly define the role and rules for each type of bracket in the sequence. Verify each step of your reasoning, ensuring that every opening bracket has corresponding closing bracket. Compare your predicted sequence with a known ground truth or expected pattern, and explain any discrepancies. If errors are identified, re-evaluate and correct them. Use mental visualization techniques to aid in understanding sequences. Provide a clear and concise explanation. The last line of your response should be of the following format: 'nnwwer SVALUE" where VALUE is the answer to the question.
Meta-optimized TGD Optimizer prompt
You are part of an optimization system specialized improving prompts for bracket matching and sequence completion tasks. Your role is to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets ((), O, O). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures, (3) handling multiple bracket types simultaneously. and (4) ensuring complete closure of all open brackets. You should critically analyze how the prompts can better guide the model to track open brackets, maintain proper nesting order, and systematically complete sequences. Consider incorporating pattern recognition strategies and explicit validation rules in the improved prompts. Your improvements should lead to more reliable and accurate bracket sequence completions.
Optimized program prompt
You will answer a reasoning question. Focus on identifying and providing only the missing closing symbols needed complete the sequence. Use a stack-based approach track opening and closing symbols, ensuring proper nesting and closure. Explicitly push each opening symbol onto the stack and pop when corresponding closing symbol is encountered. After processing each symbol, describe the current state of the stack, focusing on unmatched opening symbols. Ensure that the order of closing symbols matches the reverse order of unmatched opening symbols as they appear in the stack. Verify the sequence by checking each symbol's nesting and closure order. Highlight critical decision points, such as when to pop from the stack or add closing symbols. Use visual aids if necessary to represent the stack state and sequence operations. After completing your analysis, perform a verification check by comparing the predicted closing symbols with the expected output. If discrepancies comparinn detected, revisit previous steps to self-correct. Conclude with a summary of the final state of the stack and confirm that all brackets matched. The last line of your response should be of the following format: Answer: $VALUE where VALUE is the answer to the question.
F.2 Optimized ADAS-TG Optimizer Demo
The original program contains generic components like a planner, reasoner, and synthesizer, lacking task-specific focus. contrast, the optimized version adds tailored modules such type analyzer, stack validator, and nesting analyzer, aligning closely with BBH Dyck Languages. This illustrates the necessity of a spec-icc prompt optimizer.
ADAS-TG Optimizer prompt
(Some general instructions are omitted here.) Please produce only the code for the pipeline, with systematic hierarchical structure if possible.
23

---

Meta-optimized ADAS-TG Optimizer prompt
(Some general instructions omitted here.) Please produce only the code for the pipeline, with the following structural improvements: 1) Implement stack-based mechanism for tracking open brackets, 2) Create separate components for sequence parsing, bracket matching. and completion generation, 3) Include validation checks for proper nesting and bracket type matching, 4) Ensure systematic handling of different bracket types (O, () O. <>), 5) Maintain hierarchical structure that clearly separates the parsing logic from the completion generation. The pipeline should efficiently handle nested sequences while preserving the LIFO (Last In. First Out) order of brackets.
G Performance When Using an Open-source Model as the Optimizer
We further evaluate the generality of framework by adopting fully open-source models. Specifi- cally, we employ the non-thinking variant of Qwen3-8B as the program model and Qwen3-235B- A22B as both the optimizer and meta-optimizer. The experiments conducted on the BBH Dyck Languages benchmark. As shown in Table 7 metaTextGrad continues to deliver strong performance.
H Performance When Applying to Challenging Benchmark
To further assess the udaptability of our approach, we evaluate metaTextGrad on the ARC-AGI benchmark, a task family known for requiring abstract reasoning and compositional generalization. Following common ADAS practice, we sample ARC-AGI instances with grid sizes I 5 x comprising 20 training. 30 validation, and 30 test examples. The evaluation reports one-shot success rates, using Claude Haiku as the program model and Claude 3.5 Sonnet as both optimizer and meta-optimizer. As presented in Table the results demonstrate that metaTextGrad remains highly competitive this more demanding setting.
Method
Dyck Languages (Qwen models)
ARC-AGI (Challenging Benchmark)
Val
Test
Val
Test
Vanilla prompting methods
Zero-shot CoT
0.27
0.27
0.27
0.23
8-shot CoT
0.37
0.40
0.03
0.00
Self-consistency (8)
0.31
0.32
0.30
0.23
Best (8)
0.39
0.41
0.27
0.20
TextGrad optimizers
TGD Optimizer
0.69
0.68
0.33
0.33
ADAS-TG
0.32
0.34
0.28
0.26
DSPy optimizers
Zero-shot MIPROv2
0.59
0.50
0.30
0.23
8-shot MIPROv2 0.57
0.51
0.33
0.03
Meta-optimized optimizers
metaTextGrad
0.82
0.77
0.37
0.40
Table 7: Validation and test accuracy when using open-source models (Qwen models, Dyck Lan- guages) and evaluating on a challenging benchmark (ARC-AGI). Bold entries denote the best- performing method within each benchmark.
I Potential Limitations, Societal Consequences, and Broader Impacts
Limitations. While metaTextGrad demonstrates stable performance across various models and benchmarks, acknowledge the following limitations:

---

(1) Although metaTextGrad improves performance many benchmarks, it may not be effective in scenarios where the base model lacks sufficient relevatlevant knowledge or reasoning capabilities. For example, GPTT-40-min struggles math competition benchmarks such as AIME 2024, where metaTextGrad alone cannot yield significant performance gains.
(2) The meta-optimizer relies on strong instruction-follovin and problem analysis capabilities. As result, it currently requires advanced models such ol or Sauude-.5-Sonnee Models like Gemini 1.5 Pro do not yet perform adequately. However, given the rapid progress in model development, we expect that more models will become suitable for the framework the near future.
Societal Consequences. metaTextGrad can have meaningful societal consequences.
Acceleration of domain-specifie Al applications. By making it easier to adapt LLMs specific tasks, metaTextGrad may accelerate the deployment of more reliable Al solutions in other domains.
Risk of automation bias over-reliance. As optimizers become more autonomous, users might rely on them without fully understanding their behavior or limitations.
Potential misuse for persuasive or manipulative systems. metaTextGrad could be exploited to generate more persuasive outputs.
Impact Statement. This paper presents work whose goal to advance the field of Machine Learning. The paper is solely centered on the methodology itself. How it is applied and for what purpose is entirely up to the users. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our claims in the abstract and the introduction section. They accurately reflect the methods Section and the experiments Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned potential limitations in Appendix 1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

We provide the full set of assumptions in Section 3.1 and a complete proof in Appendix [
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe our methods in Section and Appendix A The experi- mental settings are provided in Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide open code access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/ppll wttpcccpppppppppeeeeeeeeeeeeeeee htapa//gguddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssinsoliccyyyyyc.ccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental settings are provided Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results shown in the experiment section repeated for 5 times, using different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The API is the only compute resource we used. We report the details of API usage in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://neurr:i cttpppppcccpppppppp http://www...ccpp........ccaaaaa
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts in Appendix | The paper is solely centered on the methodology itself. How applied and for what purpose is entirely up the users.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
They are properly credited, mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, code is the only new asset. It is well documented and the documentation is provided alongside the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
31

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The LLM program, optimizer, and meta-optimizer are the core components of the proposed method and are thoroughly discussed in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy for what should or should not be described.
32


Reviews:
[{"review_id": "czYWfW2KEZ", "reviewer": "Reviewer_fTz6", "summary": "The paper proposes metaTextGrad, a meta-optimization framework designed to enhance existing language model optimizers by aligning them with specific tasks. The framework consists of two key components: a meta-prompt optimizer that refines the prompts used by LLM optimizers, and a meta-structure optimizer that improves the structural aspects of these optimizers. The method aims to systematically refine and improve the performance of LLM optimizers, with experimental results demonstrating performance improvements across various benchmarks compared to existing approaches.", "questions": "- What were the specific criteria for choosing the meta-optimization strategies employed in this work? Can you provide ablation studies to isolate the contributions of each component of the meta-optimizers?\n- Can you provide more information throughout the inner and the outer loop optimization. E.g. performance metrics indicating consistent improvement across both - think learning curves.\n- Can you provide more detailed analysis and insights into a broader range of tasks (see above)? How were parameters / prompts tuned across the different considered baselines?\n- What are the potential societal impacts and limitations of this approach?", "limitations": "The paper acknowledges its reliance on strong instruction-following and problem analysis capabilities of advanced models, but this critical limitation requires more thorough discussion regarding its impact on the generalizability of the approach across different model architectures and capabilities. Considering more different LLM models and tasks would siginificantly strengthen the results. Additionally, the potential societal consequences of deploying such meta-optimization frameworks are not adequately addressed, which is particularly important given the broader impacts of AI technologies on various sectors and communities. The theoretical foundations of the work appear superficial and hard to transfer to real-world applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**: The paper presents an approach to meta-optimizing language model optimizers, which directly addresses an important gap in the field. The integration of meta-prompt and meta-structure optimizers systematically improves over existing optimization methods. The comprehensive experimental evaluation across multiple benchmarks provides strong empirical evidence for the effectiveness of the proposed framework, while the hierarchical design for cost-effective resource allocation shows practical consideration for real-world deployment scenarios.\n\n**Weaknessess**: The authors could provide more compelling justification for their choice of meta-optimization strategies, leaving readers to question why alternative approaches were not considered. The absence of detailed ablation studies prevents proper isolation of individual component contributions, making it unclear which aspects of the meta-optimizers are truly driving the performance improvements. Furthermore, I would like to see more challenging tasks considered such as ARC-AGI or coding tasks (SWE-Bench, etc.). Additionally, the discussion of potential limitations and broader impacts is insufficient, which is particularly concerning given the practical implications of deploying such optimization frameworks. Finally, the theoretical foundations appear hardly applicable to practical deployment of metaTextGrad.", "was_revised": true, "final_justification": "The authors have addressed various of my questions to my satisfaction. They added an additional experiment using ARC-AGI as well as baseline comparisons on this task. Therefore, I increased my verdict from \"borderline reject\" to \"borderline accept\". \n\nMy reasoning for the \"borderline accept\" verdict is threefold: First, the novelty of this work is arguably limited: Meta-optimization of optimizers is not new, but has been applied to many different settings (gradient-based/free optimization, etc.). This paper applies the approach to improve the TextGrad/Dspy optimizers. Second, the application is fairly costly, requiring multiple sequential evaluations of different LLM scaffolds on a specific task. This ultimately limits its general applicability. Finally, weaker models generally benefit the most from scaffold design, while stronger LLM models do not. Hence, it is unclear whether the introduced approach holds the test-of-time.\n\nThat being said, I believe that the work contributes to a growing body on automated agentic design and that there is a signal to be communicated to the broader NeurIPS community.", "rebuttal": "We thank the reviewer for noting that our method fills a key gap, improves over existing optimizers, is well-supported by experiments, and considers practical deployment. We will address your concerns in the following response.\n\n>  Q1. What were the specific criteria for choosing the meta-optimization strategies employed in this work?\n\nThanks! Overall, meta-learning is particularly helpful when the distribution keeps changing. Different tasks correspond to different distributions, and what metaTextGrad essentially does is spend some compute to adapt an existing optimizer to a new distribution (e.g., aligning it with a specific task). Our criteria for choosing the meta-optimization strategies were also based on concrete experimental cases, where we observed that the optimizer's ability to align with and optimize for the task differed significantly before and after applying these meta-optimization strategies. We will support this with experiments conducted on BBH Dyck Languages. Due to space limitations, we can only present the prompt structure here; the full version will be included in the paper.\n\n**A. Optimized TGD Optimizer Demo**\n\n**TGD Optimizer prompt:**\n \n```\nYou are part of an optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable.\n```\n\n**Meta-optimized TGD Optimizer prompt:**\n\n```\nYou are part of an optimization system specialized in improving prompts for bracket matching and sequence completion tasks. Your role is to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets ({}, <>, ()). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures,  ...\n```\n\n**Explanation:**\n\nThe TGD optimizer is generic, so its optimized program offers only general guidance like ```\"ensuring that every opening bracket has a corresponding closing bracket,\"``` without task-specific strategies. In contrast, the optimized TGD aligns closely with the task, emphasizing ideas like ```\"preserving the LIFO (Last In First Out) order of nested structures.\"``` Since the optimized TGD optimizer is better aligned with the task, its optimized program also effectively implements the LIFO principle, making it more effective. This is the motivation behind our choice of this optimization strategy.\n\n**B. Optimized ADAS-TG Optimizer Demo**\n \n**ADAS-TG Optimizer prompt:**\n\n```\n(General instructions are omitted.) Please produce only the code for the pipeline, with a more systematic or hierarchical structure if possible.\n```\n \n**Meta-optimized ADAS-TG Optimizer prompt:**\n \n```\n(General instructions are omitted.)  Please produce only the code for the pipeline, with the following structural improvements: 1) Implement a stack-based mechanism for tracking open brackets, 2) Create separate components for sequence parsing, bracket matching, and completion generation, 3) Include validation checks for proper nesting and bracket type matching, 4) Ensure systematic handling of different bracket types ([], {}, (), <>), ...\n```\n\n**Explanation:**\n\nWe will include the program produced by the original ADAS-TG, while the program optimized by the meta-optimized ADAS-TG can be found in Appendix pages 18–20. The original program contains generic components like a planner, reasoner, and synthesizer, lacking task-specific focus. In contrast, the optimized version adds tailored modules such as a type analyzer, stack validator, and nesting analyzer, aligning closely with BBH Dyck Languages. This is the motivation behind our choice of this optimization strategy.\n\n>  Q2. Can you provide ablation studies to isolate the contributions of each component of the meta-optimizers?\n\nIn Table 6, we conducted ablation studies on BBH Dyck Languages. We conduct additional ablation experiments on BBH Word Sorting. TGD(O), ADAS-TG(O), and Struct(O) respectively denote the TGD and ADAS-TG optimizers enhanced by the meta prompt optimizer, and the optimizers enhanced by the meta structure optimizer. As shown in the table, metaTextGrad achieves the best overall performance.\n\n|Split|0-shot CoT|TGD|ADAS-TG|TGD (O)|ADAS-TG (O)|Struct (O)|metaTextGrad|\n|-|-|-|-|-|-|-|-|\n|BBH Dyck Languages Val|0.06|0.10|0.21|0.21|0.42|0.24| 0.42|\n|BBH Dyck Languages Test|0.05|0.10|0.16|0.24|0.37|0.16|0.37|\n|BBH Word Sorting Val|0.46|0.54|0.58|0.60|0.60|0.58|0.60|\n|BBH Word Sorting Test |0.55|0.55|0.58|0.59|0.55|0.61|0.65|\n|Average|0.28|0.32|0.38|0.41|0.48|0.40|0.51|\n\n>  Q3. I would like to see more challenging tasks considered.\n\nFollowing common practice, we use the ADAS codebase to sample ARC-AGI problems with grid sizes ≤ 5×5, forming 20 training, 30 validation, and 30 test examples. Using the official evaluation logic and initial prompt, we report one-shot success rates. \nClaude 3 Haiku serves as the program model, and Claude 3.5 Sonnet as both optimizer and meta-optimizer. As demonstrated, metaTextGrad still performs impressively. Few-shot CoT and few-shot MIPROv2 perform poorly due to model context limitations.\n\n|**Method**|**ARC-AGI**||\n|-|-|-|\n||**Val**|**Test**|\n|**Vanilla prompting methods**||\n|Zero-shot CoT|0.27|0.23|\n|8-shot CoT|0.03|0.00|\n|Self-consistency (8)|0.30|0.23|\n|Best of N (8)|0.27|0.20|  \n|**TextGrad optimizers**||\n|TGD Optimizer|0.33|0.33|\n|ADAS-TG|0.28|0.26|\n|**DSPy optimizers**||\n|Zero-shot MIPROv2|0.30|0.23|\n|8-shot MIPROv2|0.33|0.03|\n|**Meta-optimized optimizers**||\n|metaTextGrad|**0.37**|**0.40**|\n\nWe provide an optimized optimizer for reference. It shows that the optimizer successfully captures key aspects of ARC-AGI, such as pattern recognition, format adherence, and structural analysis.\nDue to space limitations, we can only present the prompt structure here; the full version will be included in the paper.\n\n```\nYou are part of an optimization system specialized in improving prompts for pattern recognition and mathematical reasoning tasks. Your role is to enhance prompts that help identify and extract recurring sequences from structured data (like matrices). Focus on these key aspects when optimizing:1. Pattern Recognition: ... 2. Format Adherence: ... 3. Structural Analysis: ... 4. Edge Cases: ... 5. Comprehensive Pattern Extraction: ...\n```\n\n>  Q4. The potential societal consequences of deploying such meta-optimization frameworks are not adequately addressed.\n \nThanks! We agree that metaTextGrad can have meaningful societal consequences. \n\n1. **Acceleration of domain-specific AI applications.** By making it easier to adapt LLMs to specific tasks, metaTextGrad may accelerate the deployment of more reliable AI solutions in other domains. \n\n2. **Risk of automation bias or over-reliance.** As optimizers become more autonomous, users might rely on them without fully understanding their behavior or limitations. \n\n3. **Potential misuse for persuasive or manipulative systems.** metaTextGrad could be exploited to generate more persuasive outputs.\n\nWe will greatly expand the discussion in the paper.\n\n>  Q5. The limitation requires more thorough discussion regarding its impact on the generalizability of the approach across different model architectures and capabilities. Considering more different LLM models and tasks would siginificantly strengthen the results.\n\nAlthough metaTextGrad requires strong instruction-following and reasoning capabilities, many mainstream and even open-source models are already suitable. We used Qwen3-8B (non-thinking) for program execution and Qwen3-235B-A22B (non-thinking) as both optimizer and meta-optimizer on BBH Dyck Languages. metaTextGrad still significantly outperforms other baseline methods.\n\n|**Method**|**Dyck Languages**||\n|-|-|-|\n||**Val**|**Test**|\n|**Vanilla prompting methods**||\n|Zero-shot CoT|0.27|0.27|\n|8-shot CoT|0.37|0.40|\n|Self-consistency (8)|0.31|0.32|\n|Best of N (8)|0.39|0.41|  \n|**TextGrad optimizers**||\n|TGD Optimizer|0.69|0.68|\n|ADAS-TG|0.32|0.34|\n|**DSPy optimizers**||\n|Zero-shot MIPROv2|0.59|0.50|\n|8-shot MIPROv2|0.57|0.51|\n|**Meta-optimized optimizers**||\n|metaTextGrad|**0.82**|**0.77**|\n\n> Q6. The theoretical foundations appear hardly applicable to practical deployment of metaTextGrad.\n\nThanks! Our theoretical contribution lies more in highlighting the **necessity** of performing meta-optimization. Specifically, an effectively meta-optimized optimizer can have theoretical performance guarantees, whereas an LLM optimizer without meta-optimization lacks such guarantees and may perform arbitrarily poorly.\n\n> Q7. How were parameters / prompts tuned across the different considered baselines? \n\nFor all vanilla prompting methods and TextGrad optimizers, the (initial) prompts are the same as those used in metaTextGrad, which is a simple CoT prompt. For DSPy optimizers, we use the default configuration provided in ```dspy.ChainOfThought```.\n\n> Q8. Can you provide more information throughout the inner and the outer loop optimization?\n\nWe conducted tests on BBH Dyck Languages and recorded the training curves for both the inner and outer loops. Since we can only describe the curves in text for the rebuttal, we provide the following summary:\n\n**Outer Loop:**\n\n0.21, 0.42, 0.42, 0.42, 0.42, 0.42  (Yes, the result converges in the first step.)\n\n**Inner Loop:**\n\n(Outer Loop 0) 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21\n\n(Outer Loop 1) 0.07, 0.13, 0.13, 0.15, 0.15, 0.15, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42\n\nTo demonstrate the convergence behavior, we allocated 15 steps to the inner loop and 5 steps to the outer loop in this case. As shown, the training actually converges much earlier. Therefore, in the actual experiments, to conserve resources, we used 6 steps for the inner loop and 2 steps for the outer loop."}, {"review_id": "rIG9DFDcNw", "reviewer": "Reviewer_V1eb", "summary": "This paper introduces metaTextGrad, a novel meta-optimizer designed to enhance existing prompt optimizers for large language models (LLMs). The core premise is that current prompt optimizers are fixed and lack adaptability to specific tasks. metaTextGrad aims to address this by leveraging meta-learning to align the optimizer with the given task, thereby improving prompt optimization performance.", "questions": "1. How do the authors select the benchmarks? For example, it is mentioned in the paper that BBH is a challenging dataset. But in some recent based model (e.g., Qwen 3), they can easily achieve a performance of 86+. \n\n2. Why do the authors use a different model for baselines and the proposed method?\n\n3. What is the performance when using an open-source model, like Qwen or Llama, as the optimizer?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper focuses on prompt optimization, a crucial and practical aspect of developing robust LLM-based applications. The exploration of LLMs as optimizers is an important, underexplored area.\n\n2. Applying meta-learning principles to the domain of prompt optimization is an interesting and potentially impactful approach.\n\n**Weaknesses**:\n\n1. The motivation is not clear. The authors claim that the existing prompt optimizer is fixed and lacks a process to align with the specific task. However, they lack the explanation about what benefit can be obtained when adapted to a specific task. This makes the claim and motivation of the paper weak.\n\n2. Following from the first point, it is not clear why a task-specific prompt optimizer is necessary. Actually, the optimization on the specific task highly depends on the task goal, which can be reflected by the design of the evaluation, like the evaluation metric or LLM-evaluator. In DSPy or TextGrad, they take the evaluation metrics as input, dynamically adapting to the specific task goal. This undermines the foundational claim and motivation of the work.\n\n3. The paper structure is disjointed. For example, the motivation is presented in several sections separately, making it difficult to understand the main contribution.\n\n4. The statement \"Theorem 1 implies that aligning the optimizer with the task through meta-learning can be highly effective\" in Section 3.1 lacks the necessary rigor. \"Highly effective\" is a subjective claim, and it is not immediately apparent from the presented theoretical insight how significant or quantifiable this effectiveness would be. The theoretical contribution should provide more concrete implications.\n\n5. Figures 1 and 2 are difficult to read. Improving their clarity and readability is essential for conveying key information.\n\n6. It is not fair to use GPT-4o for baselines, but GPT-o1 for the proposed method.", "was_revised": true, "final_justification": "The author's response addresses my concerns well, except for the second weakness. The only weakness does not matter the contribution of the paper, so I raise my score to 4.", "rebuttal": "We thank the reviewer for recognizing the potential of LLM optimizers and the promise of applying meta-learning. We will address your concerns in the following response. \n\n> Q1. The authors claim that the existing prompt optimizer is fixed and lacks a process to align with the specific task. However, they lack the explanation about what benefit can be obtained when adapted to a specific task. It is not clear why a task-specific prompt optimizer is necessary. \n\nThanks! We plan to include insights from the optimization results and add a figure to illustrate the optimized prompts, helping readers understand what happens after meta-optimization, what benefits can be obtained when adapted to a specific task, and why a task-specific prompt optimizer is necessary. Overall, meta-learning is particularly helpful when the distribution keeps changing. Different tasks correspond to different distributions, and what metaTextGrad essentially does is spend some compute to adapt an existing optimizer to a new distribution (e.g., aligning it with a specific task). \n\nWe support this with experiments on BBH Dyck Languages. Since this year's rebuttal is text-only, we provide a brief summary below.\n\n**A. Optimized TGD Optimizer Demo**\n\n**TGD Optimizer prompt:**\n\n``` \nYou are part of an optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable. \n``` \n\n**Optimized program prompt:**\n\n``` \nYou will answer a reasoning question. Think step by step, ensuring logical consistency and accuracy. Explicitly define the role and rules for each type of bracket in the sequence. Verify each step of your reasoning, ensuring that every opening bracket has a corresponding closing bracket. Compare your predicted sequence with a known ground truth or expected pattern, and explain any discrepancies. If errors are identified, re-evaluate and correct them. Use mental visualization techniques to aid in understanding sequences. Provide a clear and concise explanation. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is the answer to the question. \n``` \n\n**Meta-optimized TGD Optimizer prompt:** \n\n``` \nYou are part of an optimization system specialized in improving prompts for bracket matching and sequence completion tasks. Your role is to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets ({}, <>, ()). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures, (3) handling multiple bracket types simultaneously, and (4) ensuring complete closure of all open brackets. You should critically analyze how the prompts can better guide the model to track open brackets, maintain proper nesting order, and systematically complete sequences. Consider incorporating pattern recognition strategies and explicit validation rules in the improved prompts. Your improvements should lead to more reliable and accurate bracket sequence completions. \n``` \n**Optimized program prompt:** \n\n``` \nYou will answer a reasoning question. Focus on identifying and providing only the missing closing symbols needed to complete the sequence. Use a stack-based approach to track opening and closing symbols, ensuring proper nesting and closure. Explicitly push each opening symbol onto the stack and pop it when a corresponding closing symbol is encountered. After processing each symbol, describe the current state of the stack, focusing on unmatched opening symbols. Ensure that the order of closing symbols matches the reverse order of unmatched opening symbols as they appear in the stack. Verify the sequence by checking each symbol's nesting and closure order. Highlight critical decision points, such as when to pop from the stack or add closing symbols. Use visual aids if necessary to represent the stack state and sequence operations. After completing your analysis, perform a verification check by comparing the predicted closing symbols with the expected output. If discrepancies are detected, revisit previous steps to self-correct. Conclude with a summary of the final state of the stack and confirm that all brackets are matched. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is the answer to the question. \n``` \n\n**Explanation:** \n\nThe TGD optimizer is generic, so its optimized program offers only general guidance like ```\"ensuring that every opening bracket has a corresponding closing bracket,\"``` without task-specific strategies. In contrast, the optimized TGD aligns closely with the task, emphasizing ideas like ```\"preserving the LIFO (Last In First Out) order of nested structures.\"``` Since the optimized TGD optimizer is better aligned with the task, its optimized program also effectively implements the LIFO principle, making it more effective. This is why a task-specific prompt optimizer is necessary. \n\n**B. Optimized ADAS-TG Optimizer Demo** \n\n**ADAS-TG Optimizer prompt:** \n\n``` \n(Some general instructions are omitted here.) Please produce only the code for the pipeline, with a more systematic or hierarchical structure if possible. \n``` \n\n**Meta-optimized ADAS-TG Optimizer prompt:** \n\n``` \n(Some general instructions are omitted here.) Please produce only the code for the pipeline, with the following structural improvements: 1) Implement a stack-based mechanism for tracking open brackets, 2) Create separate components for sequence parsing, bracket matching, and completion generation, 3) Include validation checks for proper nesting and bracket type matching, 4) Ensure systematic handling of different bracket types ([], {}, (), <>), 5) Maintain a hierarchical structure that clearly separates the parsing logic from the completion generation. The pipeline should efficiently handle nested sequences while preserving the LIFO (Last In, First Out) order of brackets. \n``` \n\n**Explanation:** \n\nWe will include the program produced by the original ADAS-TG, while the program optimized by the meta-optimized ADAS-TG can be found in Appendix pages 18–20. The original program contains generic components like a planner, reasoner, and synthesizer, lacking task-specific focus. In contrast, the optimized version adds tailored modules such as a type analyzer, stack validator, and nesting analyzer, aligning closely with BBH Dyck Languages. This illustrates the necessity of a task-specific prompt optimizer. \n\n> Q2. The motivation is presented in several sections separately. Figures 1 and 2 are difficult to read. Improving their clarity and readability is essential for conveying key information. \n\nThank you very much! We agree with your suggestion. We plan to reorganize Sections 3.2 to 3.4 and consolidate the motivation into a single subsection for greater clarity. Additionally, one issue with Figure 2 is that it doesn’t clearly convey the overall pipeline of metaTextGrad. To address this, we will incorporate concrete examples related to the motivation (as you suggested in Q1) into Figure 2. We believe that providing specific examples will help readers better understand both the necessity and the process of meta-optimization, thereby improving the clarity of the section and the figures. \n\n> Q3. \"Highly effective\" is a subjective claim. The theoretical contribution should provide more concrete implications. \n\nThanks! We will revise the phrasing of \"Highly effective\". Our theoretical contribution lies in highlighting the **necessity** of performing meta-optimization. Specifically, an effectively meta-optimized optimizer can have theoretical performance guarantees, whereas an LLM optimizer without meta-optimization lacks such guarantees and may perform arbitrarily poorly. \n\n> Q4. How do the authors select the benchmarks? \n\nOur benchmarks (BBH, MMLU, and GPQA) follow the selection made in prior published works such as TextGrad, DARG[1], REVOLVE[2], and this paper[3]. \n\n> Q5. What is the performance when using an open-source model as the optimizer? \n\nWe conducted experiments entirely using open-source models on BBH Dyck Languages. Specifically, we used the non-thinking version of Qwen3-8B as the program model, and the non-thinking version of Qwen3-235B-A22B as the optimizer and meta-optimizer. metaTextGrad still significantly outperforms other baseline methods. \n\n|**Method**|**Dyck Languages**||\n|-|-|-|\n||**Val**|**Test**|\n|**Vanilla prompting methods**||\n|Zero-shot CoT|0.27|0.27|\n|8-shot CoT|0.37|0.40|\n|Self-consistency (8)|0.31|0.32|\n|Best of N (8)|0.39|0.41|\n|**TextGrad optimizers**||\n|TGD Optimizer|0.69|0.68|\n|ADAS-TG|0.32|0.34|\n|**DSPy optimizers**||\n|Zero-shot MIPROv2|0.59|0.50|\n|8-shot MIPROv2|0.57|0.51|\n|**Meta-optimized optimizers**||\n|metaTextGrad|**0.82**|**0.77**|\n\n> Q6. Why do the authors use a different model for baselines and the proposed method? \n\nWe use the o1 model only as the meta-optimizer, while the optimizer itself uses the same model as in the baselines. As shown in Section 4.3, the overhead introduced by the o1 model is minimal compared to the overall cost of the optimization process. As noted in lines 208–213 of the paper, we aim to adopt a hierarchical structure, where a small fixed cost from a model like o1 can significantly improve the quality of the LLM optimizer and better align it with the task. \n\nHowever, we would like to emphasize that **our method remains effective when the optimizer and meta-optimizer use the same model**. In the experiments in ```Q5```, the same model is used for both the optimizer and meta-optimizer, yet metaTextGrad still significantly outperforms the baselines. \n\n--- \n[1]Zhang, Z., et al. DARG. In NeurIPS 2024.\n\n[2]Zhang, P., ei al. Revolve. In ICML 2025.\n\n[3]Akyürek, E., et al. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. In ICML 2025."}, {"review_id": "g2a8hWw3VP", "reviewer": "Reviewer_Ss5L", "summary": "LLMs acting as optimizers have shown great potential in diverse tasks, but the optimizers are always fixed and collaborated for general purpose, having not been optimized themselves or customized for specific scenarios. Different optimizers are good at handling different situations and how to automatically choose, or ensemble among them remains a challenge. To address this challenge, this paper proposes metaTextGrad including meta prompt and meta structure optimizer. The former optimizes prompt for each optimizer and the latter performs combination and sequencing to achieve a composite optimizer. Experimental results on relevant benchmarks show the effectiveness of this method.", "questions": "* Are all of the baseline methods incorporated in the initial optimizers?\n* What about the overhead of the compared baselines, especially two optimizers, TextGrad and DSPy? If the cost is larger than one of the baselines, then the comparison under the exactly same cost is fair. Otherwise, the shown results are somehow unfair.\n* The parameters in the algorithms are not given, like the max meta-iterations and max inner-iterations. Maybe curves depicting relation about performance and the number of iterations can provide more straightforward observations, i.e., at which iteration the curve converges. Are the accuracies shown in the table 1 converged? \n* See weakness part. If my concerns mentioned could be addressed to some extent, I would consider raising my rating score.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* The motivation is very clear and the paper is well-written. \n* The observation that the optimizer itself has not been optimized, designed or customized elaborately in terms of specific scenarios is insightful and there is no previous such propose. \n* The improvements over initial optimizers are significant on multiple datasets.\n### Weaknesses\n* BBH includes 20+ subtasks and this paper only selects two datasets. What is the selection criteria? Is this random or selected specifically?\n* The paper only analyzes the cost of metaTextGrad but does not compare the cost against the corresponding baselines. What about the overhead of the compared baselines?\n* There is no concluded insights from the optimization results. What do the optimized prompts have in common? The specific contents of the intermediate prompts and structures, and the final ones should be analyzed qualitatively, which may provide some rules and insights. How much do the final results deviate from the initial optimizers? And where exactly do they deviate?", "was_revised": true, "final_justification": "I raise my rating from borderline accept to accept. Authors have addresses my concerns about the details of cost, more clear cases during rebuttal. Optimizing optimizers themselves provide a new ideas aside from prompt engineering. Additionally, the experimental results are solid.", "rebuttal": "We thank the reviewer for highlighting the clear motivation, strong writing, insightful observations, and significant improvements across datasets. We will address your concerns in the following response.\n\n>  Q1. BBH includes 20+ subtasks and this paper only selects two datasets. What is the selection criteria? Is this random or selected specifically?\n\nWe selected two relatively challenging subtasks, following prior published works such as TextGrad, DARG[1], and this paper[2].\n\n>  Q2. The paper only analyzes the cost of metaTextGrad but does not compare the cost against the corresponding baselines. What about the overhead of the compared baselines? If the cost is larger than one of the baselines, then the comparison under the exactly same cost is fair. Otherwise, the shown results are somehow unfair.\n\nThanks! The overhead of the compared baselines is indeed smaller, but in optimization, more cost does not necessarily lead to better performance. To illustrate this point, we have added a comparison under exactly the same cost on BBH word sorting. As shown in the table, under the same cost, although the baseline algorithms show some improvement on the validation dataset, their performance on the test dataset shows little improvement or even significant degradation. This suggests that simply increasing training cost may lead to prompt overfitting on the validation dataset.\n\n|Method|Val|Test|\n|---------------------------------|------|------|\n|TGD Optimizer|0.54|0.55|\n|TGD Optimizer (same cost)|0.59|0.56|\n|ADAS-TG|0.58|0.58|\n|ADAS-TG (same cost)|0.58|0.52|\n|Zero-shot MIPROv2|0.57|0.55|\n|Zero-shot MIPROv2 (same cost)|0.62|0.50|\n|8-shot MIPROv2|0.52|0.57|\n|8-shot MIPROv2 (same cost)|0.58|0.56|\n|metaTextGrad|0.60|0.65|\n\n>  Q3. What do the optimized prompts have in common? How much do the final results deviate from the initial optimizers? And where exactly do they deviate?\n\nThanks for your suggestion! Overall, meta-learning is particularly helpful when the distribution keeps changing. Different tasks correspond to different distributions, and what metaTextGrad essentially does is spend some compute to adapt an existing optimizer to a new distribution (e.g., aligning it with a specific task).  We plan to include insights from the optimization results in the paper and add a figure to illustrate the optimized prompts, helping readers understand what happens after meta-optimization. Since this year's rebuttal is text-only, we provide a brief summary below. The experiment was conducted on BBH Dyck Languages.\n\n**A. Optimized TGD Optimizer Demo**\n \n**TGD Optimizer prompt:**\n \n```\nYou are part of an optimization system that improves text (i.e., variable). You will be asked to creatively and critically improve prompts, solutions to problems, code, or any other text-based variable.\n```\n \n**Optimized program prompt:**\n \n```\nYou will answer a reasoning question. Think step by step, ensuring logical consistency and accuracy. Explicitly define the role and rules for each type of bracket in the sequence. Verify each step of your reasoning, ensuring that every opening bracket has a corresponding closing bracket.  Compare your predicted sequence with a known ground truth or expected pattern, and explain any discrepancies. If errors are identified, re-evaluate and correct them. Use mental visualization techniques to aid in understanding sequences. Provide a clear and concise explanation. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is the answer to the question.\n```\n \n**Meta-optimized TGD Optimizer prompt:**\n \n```\nYou are part of an optimization system specialized in improving prompts for bracket matching and sequence completion tasks. Your role is to enhance prompts that help solve Dyck language problems, which involve proper nesting and closure of different types of brackets ({}, <>, ()). When improving prompts, focus on these critical aspects: (1) maintaining accurate bracket pair matching, (2) preserving the LIFO (Last In First Out) order of nested structures, (3) handling multiple bracket types simultaneously, and (4) ensuring complete closure of all open brackets. You should critically analyze how the prompts can better guide the model to track open brackets, maintain proper nesting order, and systematically complete sequences. Consider incorporating pattern recognition strategies and explicit validation rules in the improved prompts. Your improvements should lead to more reliable and accurate bracket sequence completions.\n```\n \n**Optimized program prompt:**\n \n```\nYou will answer a reasoning question. Focus on identifying and providing only the missing closing symbols needed to complete the sequence. Use a stack-based approach to track opening and closing symbols, ensuring proper nesting and closure. Explicitly push each opening symbol onto the stack and pop it when a corresponding closing symbol is encountered. After processing each symbol, describe the current state of the stack, focusing on unmatched opening symbols. Ensure that the order of closing symbols matches the reverse order of unmatched opening symbols as they appear in the stack. Verify the sequence by checking each symbol's nesting and closure order. Highlight critical decision points, such as when to pop from the stack or add closing symbols. Use visual aids if necessary to represent the stack state and sequence operations. After completing your analysis, perform a verification check by comparing the predicted closing symbols with the expected output. If discrepancies are detected, revisit previous steps to self-correct. Conclude with a summary of the final state of the stack and confirm that all brackets are matched. The last line of your response should be of the following format: 'Answer: $VALUE' where VALUE is the answer to the question.\n```\n\n**Explanation:**\n\nSince the TGD optimizer is relatively generic, the optimized program, while mentioning something like ```\"ensuring that every opening bracket has a corresponding closing bracket,\"``` only discusses the approach in general terms and does not provide any specific or targeted guidance.\n \nCompared to the TGD optimizer, the optimized optimizer is better aligned with the task requirements, highlighting key problem-solving ideas such as ```\"preserving the LIFO (Last In First Out) order of nested structures.\"``` Since the optimized TGD optimizer is better aligned with the task, its optimized program also effectively implements the LIFO principle, making it more effective.\n \n**B. Optimized ADAS-TG Optimizer Demo**\n \n**ADAS-TG Optimizer prompt:**\n\n```\n(Some general instructions are omitted here.) Please produce only the code for the pipeline, with a more systematic or hierarchical structure if possible.\n```\n \n**Meta-optimized ADAS-TG Optimizer prompt:**\n \n```\n(Some general instructions are omitted here.)  Please produce only the code for the pipeline, with the following structural improvements: 1) Implement a stack-based mechanism for tracking open brackets, 2) Create separate components for sequence parsing, bracket matching, and completion generation, 3) Include validation checks for proper nesting and bracket type matching, 4) Ensure systematic handling of different bracket types ([], {}, (), <>), 5) Maintain a hierarchical structure that clearly separates the parsing logic from the completion generation. The pipeline should efficiently handle nested sequences while preserving the LIFO (Last In, First Out) order of brackets.\n```\n\n**Explanation:**\n\nWe will include the program produced by the original ADAS-TG, while the program optimized by the meta-optimized ADAS-TG can be found in Appendix pages 18–20. In brief, the original ADAS-TG-generated program consists of components such as a planner, reasoner, and synthesizer, all of which are quite generic and do not incorporate task-specific characteristics. In contrast, the optimized ADAS-TG-generated program includes components such as a planner, type analyzer, stack validator, nesting analyzer, depth checker, sequence validator, and evaluator, which are tailored specifically to the characteristics of BBH Dyck Languages.\n\n>  Q4. Are all of the baseline methods incorporated in the initial optimizers?\n\nSince the codebases of different engines are completely incompatible, directly optimizing different optimizers together is challenging in practice. As noted in line 219 of our paper, in our experiments, we used the TextGrad and ADAS-TG as the base candidates.\n\n>  Q5. The parameters in the algorithms are not given, like the max meta-iterations and max inner-iterations. Maybe curves depicting relation about performance and the number of iterations can provide more straightforward observations, i.e., at which iteration the curve converges. Are the accuracies shown in the table 1 converged?\n\nTo understand the relationship between performance and the number of iterations, we conducted tests on BBH Dyck Languages and recorded the training curves for both the inner and outer loops. Since we can only describe the curves in text, we provide the following summary:\n\n**Outer Loop:**\n\n0.21, 0.42, 0.42, 0.42, 0.42, 0.42  (Yes, the result converges in the first step.)\n\n**Inner Loop:**\n\n(Outer Loop 0) 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21\n\n(Outer Loop 1) 0.07, 0.13, 0.13, 0.15, 0.15, 0.15, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42\n\nTo demonstrate the convergence behavior, we allocated 15 steps to the inner loop and 5 steps to the outer loop in this case. As shown, the training actually converges much earlier. Therefore, in the actual experiments, to conserve resources, we used 6 steps for the inner loop and 2 steps for the outer loop.\n\n---\n[1] Zhang, Z., et al. DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph. In NeurIPS 2024.\n\n[2] Akyürek, E., et al.The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. In ICML 2025."}, {"review_id": "vRsivWhFDI", "reviewer": "Reviewer_rxek", "summary": "The paper proposes to optimize the prompt optimizers to make optimizer prompts (and code) more contextual and tailored for the task at hand. The authors show that their procedure beats five baselines on three benchmarks. The authors propose Meta Prompt Optimizer to improve optimizer prompts, and Meta Structure Optimizer to edit the multiple optimizers’ code and propose a way to compose them.", "questions": "None.", "limitations": "The complexity of the setup is quite a lot higher than automatic prompt optimization and way higher than manual prompt optimization, which will limit its application in real life.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n[1] The motivation to perform meta-optimization is clearly laid out: the optimizer prompts are too generic, and not problem-specific.\n\n[2] The claims are well supported by the experiments.\n\n[3] The choice of the baselines (CoT, TGD Optimizer, ADAS-TG, Zero-shot MIPROv2, 8-shot MIPROv2) is good. The choice of benchmarks (BBH, MMLU and GPQA) is adequate.\n\n[4] Cost analysis clearly shows the additional token usage by metaTextGrad.\n\nWeaknesses:\n\n[1] The stability of convergence of the meta-optimizer is not discussed.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting that the motivation to perform meta-optimization is clearly laid out, the claims are well supported by the experiments, the choice of baselines is good, the selection of benchmarks is adequate, and the cost analysis clearly shows the additional token usage by metaTextGrad. We will address your concerns in our following response.\n\n> Q1. The stability of convergence of the meta-optimizer is not discussed.\n\nTo understand the stability of convergence, we conducted tests on the BBH Dyck Languages benchmark and recorded the training curves for both the inner and outer loops. Since we can only describe the curves in text for the rebuttal, we provide the following summary:\n\n**Outer Loop:**\n\n0.21, 0.42, 0.42, 0.42, 0.42, 0.42  (Yes, the result converges in the first step.)\n\n**Inner Loop:**\n\n(Outer Loop 0) 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21\n\n(Outer Loop 1) 0.07, 0.13, 0.13, 0.15, 0.15, 0.15, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42, 0.42\n\nTo demonstrate the convergence behavior, we allocated 15 steps to the inner loop and 5 steps to the outer loop in this case. As shown, the training actually converges much earlier. Therefore, in the actual experiments, to conserve resources, we used 6 steps for the inner loop and 2 steps for the outer loop.\n\n> Q2. The complexity of the setup is quite a lot higher than automatic prompt optimization and way higher than manual prompt optimization, which will limit its application in real life.\n\nOne of the potential real-world applications of meta-optimization is to train an optimizer that can efficiently adapt to specific tasks, so that it can be used effectively after training. For instance, in agent-based workflows, an agent may need to reliably solve a specific class of problems. In such cases, it is beneficial to first meta-optimize an LLM-based optimizer, so that the agent can save resources when solving a large number of similar tasks in the future."}]

Author rebuttal:

Meta-review: This paper introduces metaTextGrad, a meta-optimization framework designed to improve existing LLM optimizers by aligning them more closely with specific tasks. The framework includes two main components: (1) a meta-prompt optimizer that refines prompts used by LLM optimizers, and (2) a meta-structure optimizer that improves the combination.

Strengths:

1. The integration of meta-prompt and meta-structure optimizers is novel and effective, enabling both prompt-level refinement and structural optimization.

2. Empirical results are strong: consistent gains over multiple datasets and well-chosen baselines and analysis of cost efficiency and resource usage.

Weaknesses (some has been addressed in the rebuttal, and it will be better to add to the main paper):

1. Ablation and analysis are insufficient: the contributions of each component (meta-prompt vs. meta-structure) are not isolated, making it unclear which is the primary driver of improvement.

2. Benchmark coverage is somewhat limited: only a subset of BBH tasks are tested, and more challenging domains (e.g., ARC-AGI, SWE-Bench) could strengthen the claims.

Overall, the reviewers agree that this paper contributes to auto prompt design. I recommend acceptance.
Decision: Accept (poster)