Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Authors: Ching-An Cheng
Allen Nie
Adith Swaminathan
Venue: NeurIPS 2024 poster (2024)
Keywords: Optimization
Back-Propagation
Automatic Differentiation
LLM
Language Feedback
Execution Trace
Primary area: optimization_for_deep_networks
TL;DR: Framework for efficient optimization of heterogenous parameters in general computational workflows

Abstract: We study a class of optimization problems motivated by automating the design and update of AI systems like coding assistants, robots, and copilots. AutoDiff frameworks, like PyTorch, enable efficient end-to-end optimization of differentiable systems. However, general computational workflows can be non-differentiable and involve rich feedback (e.g. console output or user’s responses), heterogeneous parameters (e.g. prompts, codes), and intricate objectives (beyond maximizing a score). We investigate end-to-end generative optimization – using generative models such as LLMs within the optimizer for automatic updating of general computational workflows. We discover that workflow execution traces are akin to back-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization. Formally, we frame a new mathematical setup, Optimization with Trace Oracle (OPTO). In OPTO, an optimizer receives an execution trace along with feedback on the computed output and updates parameters iteratively. We provide a Python library, Trace, that efficiently converts a workflow optimization problem into an OPTO instance using PyTorch-like syntax. Using Trace, we develop a general LLM-based generative optimizer called OptoPrime. In empirical studies, we find that OptoPrime is capable of first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, etc., and is often competitive with specialized optimizers for each domain. We envision Trace as an open research platform for devising novel generative optimizers and developing the next generation of interactive learning agents. Website: https://microsoft.github.io/Trace/.

Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Ching-An Cheng" Microsoft Research thingancemicrosoff.com
Allen Nie Stanford ani anie@cs.c stanford. edu
Adith Swaminathan"
Netflix
awwaminathhneeeetllixcccc
Abstract
We study a class of optimization problems motivated by automating the design and update of Al systems like coding assistants, robots, and copilots. Difo frameworks, like PoTch,, enable efficient end-to-end optimization of differentiablee systems. However, general computational workflows can be non-differentiable and involve rich feedback (e.g. console outpul or user's responses), heterogeneous parameters (e.g. prompts, codes), and intricate objectives (beyond maximizing score). We investigate end-to-end generative optimization using generative models such as LLMs within the optimizer for automatic updating of general computational workflows. We discover that workflow execution traces are akin back-pro pack-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization. Formally, we frame setup, Optimization with Trace Oracle (OPTO). In OPTO, an optimizer receives an execution trace along with feedback on the computed output and updates parameters iteratively. We provide Python library. Trace, that efficiently converts workflow optimization problem into an OPTO instance using PyTorch-like syntax. Using Trace, we develop general LLM- based generative optimizer called OptoPrime. In empirical studies, we find that OptoPrime is capable of first-order numerical optimization, prompt optimization, hyper parameter tuming, robot controller design, code debugging, etc., and is often competitive with specialized optimizers for each domain. We envision Trace an open research platform for devising novel generative optimizers and developing the next generation of interactive learning agents. Website: https: httts://microsoftt thub. io/Trocce
mathematical
Introduction
Computational workflows that integrate large language models (LLMs), machine learning (ML)
models, orchestration, retrievers. tools, etc., power many state-of-the-art AI applications [1]: from chatbots [2], coding assistants [3], robots [4], to multi-agent systems [5]. However designing computational workflow requires laborious engineering because many heterogeneous parameters (e.g. prompts, orchestration code, and ML any erroneous behaviors of the workflow persist unless developer manually updates it.
are involved. Moreover, after deployment
We study a class of optimization problems motivated by automating the design and update of compu- tational workflows. Computational workflows produce optimization problems with heterogeneous parameters, rich feedback (e.g. console output and user's verbal responses), and intricate objectives (beyond maximizing a score). Moreover, workflow can have interdependent steps (e.g. adaptive orchestration. feedback control loops) and/or involve non-differentiable, operations (e.g. ML models, simulations) whose behavior cannot be succinctly captured. As result, the structure of the computation may change as the parameters and the inputs of the workflow vary.
fami-black-box. stochastic
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Due to its complexity, computational workflow optimization is usually framed as a black-box [6] or algorithm configuration [7] problem, and is tackled by general techniques like Bayesian Opti- mization [8], Exolutionary Algorithms [9], Reinforcement Learning (RL.) [10] using scalar scores as feedback. But one observation of scalar feedback alone does not provide an improvement signal, so these algorithms are very inefficient when the parameter space is large (e.g. codes or natural language prompts). Recently LLM-based optimizers [11-16] have been proposed as generative optimizers to improve efficiency, leveraging the prior of generative models learned from large pre-training corpora to optimize complex prompts and codes. In this paper, we investigate how generative optimization can be applied more broadly and systematically to optimize general computational workflow end-to-end Appendix B discusses related works in generative optimization.
1.1 Toward Efficient End-to-End Optimization of Computational Workflows
Inspired by back-propagation [17]. we take end-to-end approach to computational workflow optimization. AutoDiff frameworks [18, 19] have scaled the back-propagation algorithm to optimize differentiable workflows (e.g., neural networks) with billions of parameters. We extend the idea of AutoDiff and design a new framework, Trace, for jointly optimizing all of the heterogeneous parameters in general computational workflows, which may not be differentiable.
Trace treats computational workflow as graph like a neural network, where nodes are either inputs, parameters or the results of computation steps, and directional edges denote how nodes are created from others. But, instead of gradients, Trace propagates the execution trace workflow (which records the intermediate computed results and how they are used), via the notion of minimal subgraph (see Section 3.3). We show that propagating the exccution trace subsumes back- proppaaaatn for differentiable workflows, and remains applicable even for non-differentiable workflows. Viewing workflow as a graph and using its execution trace is standard for software engineering: for instance, human developers use such traces to debug distributed systems [20]. Our novel insight is that execution traces also enable end-to-end generative optimization, because they provide the information needed relate rich feedback to the parameters in general workflows.
1.2 Example of Trace in Action
Trace is available as a Python library with an API inspired by Py Torch [19]. A user declares the parameters to be optimized using a trainable flag, decorates the workflow with node and bundle wrappers, and runs a Trace optimizer - just like how they would declare and train neural networks.
Consider building an Al agent for the Battleship game (Fig. 1). components (reason and act), which are chained together to react to different board configurations. The Battleship environment provides feedback (binary reward in texts) to tell if the agent's 's action hit the hidden ships, and the goal is to hit all hidden ships as fast as possible. Consider how a human programmer might approach the problem. They may run the policy and change the code based on the
The agent's 's policy (Fig. 2a) has two
Strene benct wwweeaaeeetttteeeeeee
Strace, www.lsttrrannbee-rrree res8en(ee3F - 4371
2003 IF
try parenttal case rrw. col sal
serrcending
Defauit lree, cal.
pasillan iisisloss foord
precittty recesstt Vic
Etrace banccagt poscectraaeeeee (arenaiaeTTT
giee
we.
ratur best 1991
10.
Learn to enumerate all squares
Balance unexplored squares
Only guess |0. 01
adjacent to previous hits
% or Stips Hit
60
20
0
Training Iberations
Trace (Ous)
OPRO
Enumeration
Figure 1: Learning Example in Battleship. Trace automatically optimizes heterogeneous parameters (multiple codes) to train an agent to Battleship. Mcans and standard errors computed over 10 random seeds.

---

class Pol Polssy(trace. (oleey(rrace.ooull))
def foreard(self, nap): plan self reason (top)
policy Policy()
Hoard [Node] (oe
Heason (PanantooNiode) def (esount))
parars policy par paraneesss
outout seit.act(naa, plan)
optinizer trace. Ogtinizer
return output
5ya make("Battlessa ('Batcleship-ve))
burdd (aaace.brdla(trannnaa (tratnable--True) rww.enessssiii naр)
board
while done:
feedback
Act (ParameteCooale def actin
Given Battleship analyze the gane
8 Forward
Execution fredbock
target (beacy(
return [0, 0]
board. feedback. done
anv.atep (ata) except TraceExacucionérre as
atrace. ( def (self, nap, plan):
Action [Nudel [0,
feedback = str(e)
target e. exccepioonnnee
Given a map and plan, select target coordinate....
Backward pass and update
eptimizer.zerg zero. feecbaak)c
and return
eptimizer.bate backuerd packverd(targge feedback)
optinizer. step
Fxr
Feodback
(a) We write a trainable policy in (b) We then use PyTorch-like optimiza- (c) Trace automatically Python using Trace operators.
tion syntax to train the policy.
records execution DAG.
Python Code of the Battleship Example. To build a self- aelf-and agent with Trace, we only need to annotate some empty functions (reason, act) and set an optimizer following PyTorch semantics. For space, we trim the docstrings of the empty functions with "..." and list them in Appendix J. Trace then builds a DAG the workflow executes and updates the parameters (see Fig. 1 for the result).
Figure 2:
observed feedback. They may rewrite the code a few times to try different heuristics to solve this problem. They will fix any execution errors (e.g., out-of-bounds exceptions) by using stack traces.
Our Trace framework accomplishes the programmer's goal automatically without adding complexity the Python code. The user declares reason and act trainable (Fig. 2a) and then runs the agent in a PyTocchllike training loop (Fig. 2b). During the execution, Trace records a directed acyclic graph (DAG) (Fig. 2c) and uses it to compute the execution trace for optimization. Trace also automatically catches errors (c.g.. syntax/semantic errors) and can use them as feedback. In Fig. 1, we show what the agent learns as Trace optimizes* its policy, where the learned policy is evaluated on new randomly generated games for generalization. The agent can quickly improve its performance and learn strategies that are increasingly complex. Remarkably, there is no mention of the specific Battleship environment API, nor details on how the functions reason and act should behave or adapt in Fig. 2a. The Trace optimizer figures out all the details through interactions as the computational graph unfolds and the feedback on the output is observed. Beyond code as parameters in this example, we also have experiments in Section where prompts and other heterogenous parameters are optimized.
1.3 A New World of Optimization
The design of Trace is based on new mathematical setup of iterative optimization, which we call Optimization with Trace Oracle (OPTO). In OPTO, an optimizer selects parameters and receives a computational graph well as feedback on the computed output. Trace is tool to efficiently convert the optimization of computational workflows into OPTO problems in practice.
We argue that framing computational workflow optimization as OPTO can lead to faster convergence than black-box approach. We present constructive proof: We design generative optimizer called OptoPrime, for OPTO. OptoPrime turns OPTO to sequence of pseudo- algorithm problems. In each iteration of OPTO, we format the execution trace and output feedback as aseudo-algorithm question and present it to an LLM for solution (GPT-4 using ReAct-Cort CooT prompt listed in Appendix G). In experiments, we apply OptoPrime unchanged to many disparate applications like prompt optimization, first-order numerical optimization, hyper- syper-parameter tuning, and robot controller design. We find that the general purpose OptoPrime is competitive with specialized optimizers for each domain, e.g., achieving 10% higher accuracy on BigBenchHard [21] when optimizing a DSPy [22] program compared to their hand-designed optimizer.
ceneral-purposs efficient
Trace, OPTO, and OptoPrime together provide the first tractable algorithm for optimizing general computational workflows end-to-end. The Trace framework a) leverages a workflow's 's structure and b) can incorporate rich feedback beyond scores, extending AutoDiff to complicated, non-differentiable computational workflows. With Trace, we conjecture that "training deep agent networks" (which fluidly mix computation of tensors, LLMs, and other programmable tools) will soon be possible.
We use new general-
general-purpose purpose LLM-based optimizer OptoPrime that we detail in Section 4.
3

---

2
Optimization with Trace Oracle
OPTO is the foundation of Trace. In this section, we define this graph- abstraction of iterative optimization and discuss how OPTO covers various computational workflow optimization problems.
Preliminary We review the definition of a computational graph (see Fig. 2c). A computational graph g DAG, where node represents an object (such as tensors, strings, etc.) and an edge denotes an input-output relationship. We call a node without parents root and a node without children a leaf, which are the inputs and outputs of the computational graph. In the context of optimization, some inputs are marked as trainable parameters, which are denoted as (Xo). For node X, its parents are the inputs to an operator that creates X. The descendants of node X are those that can be reached from X following the directed edges; the ancestors are defined conversely. Without loss of generality, we suppose that all computational operators have a unitary output'. In this way, we can associate the operator that creates the child node with the child node, and the full computation can be represented compactly as a DAG without explicitly representing the operators. The exccution trace of a computational workflow is defined as the sequence of operations and their exccution results invoked when computing the output from a set of inputs; execution traces can be expressed as a computational graph as defined above.
2.1 Problem Definition of OPTO
OPTO is an abstract setup of iterative computational work- flow optimization. An OPTO problem instance is defined by tuple (9,w, T), where θ is the parameter space, w the context of the problem, and T is Trace Oracle. In each iteration, the optimizer selects a parameter 0 E e, which can be heterogeneous. Then the Trace Oracle T returns a trace feedback, denoted as T = (f.g), where is the execution trace represented as a DAG (the parameter is contained in the root nodes of g). and f is the feedback provided to exactly one of the output nodes of g. Finally, the optimizer uses the trace feedback r to update the parameter according to the context w and proceeds to the next iteration, as shown in Fig. 3.
8 Gak fugi f2-82
Av f3.83
f2182
Figure 3: Iterations of OPTO. When O E e is selected, the Trace Oracle T returns trace feedback = (f. g). where 9 is compu- tational graph using as input and f is the feedback given to the output of g.
In OPTO, the output feedback f is generic, such as scores, gradients, hints/explanations expressed in natural language, and console messages. The context w provides invariant information to interpret the output feedback f as well as any known iide-information, e.g., desired properties of the parameters. The context is fixed for an OPTO problem instance (similar to an instruction, or problem definition), whereas the output feedback f can change with the parameter E and the resulting computation g. For example. may be "Minimize a loss function" and f is loss. Alternatively, w can be open-ended, like "Follow the feedback" and f describes how an output should be changed. In Section 3.2, we discuss how to define the context and output feedback when constructing OPTO problems in practice. this paper, we focus on OPTO problems where f and w can be expressed compactly in text. This covers a wide range of problems [23], including those with scalar feedback.
OPTO differs from black-box setup in that the execution trace g shows the computational path toward the output, which provides information to construct a parameter update direction from f and w. In the minimization example above, when the execution trace g is missing, it is unclear how the parameter can be improved given only a point evaluation of f. On the other hand, with documenting the functions used to create the output, an update direction (e.g.. a gradient) can be derived. We highlight that the structure of the computational graph returned by the Trace Oracle T can be different each iteration the general case (as in Fig. 3) because some workflows can change with different inputs and parameters.
To ground the OPTO setup, we show how OPTO is related to some existing problems with examples. We discuss other examples like hyperparameter tuning and multi-agent systems in Appendix C.
Example 1 (Neural network with back-propagation)) The parameters are the weights. g is the neural computational graph and f is the loss. An example context can be "Minimize loss". The back-propagation algorithm can be embedded in the OPTO optimizer, e.g., an OPTO optimizer can use 7 compute the propagated gradient at each parameter, and apply a gradient descent update.
A multi-output operator can always be modeled by a single-output operator and single-output indexers.

---

Example 2 (RL). The parameters are the policy. is the trajectory (of states, actions, rewards) resulting from running the policy in a Markov decision process; i.e., g documents the graphical model of how a generated action is applied to the transition, which then returns the observation and reward. can be a success flag. w can be "Maximize return (cumulative rewards)" or "Maximize success".
Example (Prompt Optimization of an LLM Agent). The parameters are the prompt of an LLM workflow. g is the computational graph of the agent and f is the feedback about the agent's 's behavior (which can be scores natural language). can be "Maximize score" "Follow the feedback".
3 Trace: The Next AutoDiff
We design framework, Trace, to bring OPTO from abstract concept to reality. Trace provides a lightweight Python' tool to implement the Trace Oracle of OPTO for optimizing computational workflows. Through the OPTO framing, Trace separates the design of optimizers and domain-specific components so that general-purpose purpose optimizers can be built that work across diverse domains.
3.1 Design of Trace
Trace is designed based on two primitives:
node is the wrapper of Python objects. When wrapped, a Python object is registered as a unique node in the computational graph of Trace. A node be set trainable, which would make the node a parameter in OPTO. In addition, when using node to declare a parameter, one can also describe (in natural language) constraints that the parameter should obey.
bundle is the decorator to turn Python methods into operators. When function is decorated, its docstring and source code are recorded as the definition of the operator, which infer how the output feedback should change the parameters. Moreover, functions decorated by bundle can be set t trainable as well, which means that the code of the decorated method becomes a parameter."
For any workflow, using Trace involves the following steps (see Fig. 2). First, the user declares the workflow's parameters using node and bundle, and defines the workflow's 's conceptual blocks as operators using bundle. Then the user creates OPTO optimizer (such as OptoPrime in Section 4), and optionally provides the context w for the problem. (A default context w OptoPrime is "Follow the feedback").. In addition, the user defines mechanism to provide feedback to the computed result (e.g., scores, natural language suggestions, etc.), in analogy to defining a loss function in neural network training. After the declaration, optimization via Trace repeats the following: /) Execute the decorated workflow. As it runs, a DAG is built in the backend, logging the computed results and their connections. 2) Initiate the propagation of the output feedback to the parameters by calling backward. d. (Any execution error is also treated as feedback; see Appendix D.) Internally, Trace extracts the minimal subgraph g connecting the parameters and the output and sends the OPTO optimizer the trace feedback T = (J,g). 3) Call the OPTO optimizer's step method to update the parameters.
3.2 Using Trace Primitives for Effective Execution Tracing
There many ways represent computational workflow as a computational graph, from abstract- ing the entire process as one big operator to listing all low-level steps as operators in the graph. In Trace, the level of abstraction is decided by how bundle applied, as all steps underneath bundle are abstracted as one operator. The design of bundle allows tracing most Python codes, except for those modifying the content of an object reference in place. However, such a case can be avoided by first duplicating the object and then applying the modification to the copied object, similar to how recurrent neural network is implemented.
Different abstraction choices trade off the graph complexity and the description needed for each operator. Abstracting everything into single operator makes trivial graph but requires more descriptions to faithfully capture the workflow. On the other hand, not all details matter in optimiza- tion, so exposing every low-level operator can make the graph unnecessarily cluttered. The best representation depends on the application and OPTO optimizer at hand. This problem, we believe, similar the design of neural network architectures. Here, we suggest defining the operators to
"The current implementation in Python. The DAG-based design Trace can be adapted to other programming languages also, just like AutoDiff libraries that are available C++. TThi would add extra parent (i.e., the trainable code) the computed child node.
5

---

mimic the whiteboard system diagram of the computational workflow. This level of abstraction in our experiments strikes a good balance between the ease of documentation and the graph size.
Apart from architecture design, another design question is what information goes into the context versus the description of each operator? For a single problem, there is difference in principle; one can choose to provide details of all operators in g through the context w. However, this will require manually crafting a context for every workflow. We suggest instead providing description of the operators when they are defined using bundle. Then Trace will automatically generate the workflow -seeciff information while the same context shared across many workflows.
3.3 Backward Feedback Propagation: Realizing the Trace Oracle of OPTO
Trace uses recursive graph traversal algorithm (Al- gorithm 1) to propagate feedback in the reversed topo- logical ordering. With different propagators, Algo-
Algorithm 1 Backward Message Passing
Input: Node oulput, feedback f. propagator P
1: < P.init(f)
rithm can implement various serward-backwarr schemes including back-propagation. 6 We propose
2:
,T)
3: queue t Miinieap ((uuputt)
general propagator, Minimal Subgraph Propagator
4: while queue is not empty do
(MSP), in Algorithm MSP propagates the trace feedback t (f.g), where g is implemented as a priority queue. Running Algorithm with MSP (Al- gorithm 2) together implements the Trace Oracle of OPTO, which extracts the minimal subgraph between parameter nodes and output.' ? connecting the parame- ters and an output. Appendix E proves the following:
5: node + queue.pop()
6: feedback +- P.propagattemaades for parent in node.parents do
7:
8:
T T feedbacckk (parent)
9:
parenl.add_ aww.aadddeeebceeekee Feedback(node,, 7) if parent & queue then
10:
11:
queue.push(parent)
Theorem 1. For a graph with N nodes and maximum degree W, Algorithms I and 2 have time complexity O(WN" N log N) and space complexity N)
Algorithm Minimal Subgraph Propagator
Input: A child node node
// The pseudo code implements propagate.
By contrast, back- peoppaggagntinn has time and space complexities of O( ) and (d) respectively, where d is the maximal dimension of tensors. The difference is because in the most general setting of computational graphs and feedback, the propagated feedback (no matter how it is represented) does not have a constant size and needs the full subgraph.
ini t(f) returns (J, (t).
1: + {node) U(parend in pareetaaents)
2: for (S1.9) in node. feedback do
3: 9 + UU9
4: f
// all f. are the same.
5: return {p (f,9) for p in mode.parent
Theorem 2. For generic computational graphs of N nodes, in the worst case, the propagated feedback needs description length (2N) to construct improvement direction.
Despite the worst case complexity of MSP, in practice the difference is negligible. Since MSP only involves merging priority queues of references, most actual computation happens in the forward pass (and also the optimizer's step method). However for very large problems with millions of nodes in the minimal subgraph, we anticipate that computational issues of MSP could arise.
4 Design of the First OPTO Optimizer
We introduce an LLLM-based generative optimization algorithm OptoPrime for any text-based OPTO problem. We believe that this is one of many possible optimization algorithms for these problems and there is a large space to be explored for identifying efficient optimization methods for OPTO.
Subgraph Representation One core challenge of designing an LLM-based OPTO optimizer is how to represent the execution trace subgraph y (which can involve various graph structures and heterogenous data) to LLMs, in a way that LLMS can understand and reason about the downstream effects of parameter updates. We leverage the LLMs' remarkable coding and debugging ability [3].
II propagtop,,,,,,n, the message is the gradient V. and the propagate function returns J, Σ, V, to its ith parent, where J; is the Jacobian to the ith parent and and the V gradient received from the jih child. 'The minimal subgraph gx-+ between nodes X' and a node is defined as gx.y == X U12, PIZ e ancestors (Y), 7 e descendants( X), X E X, P E parents( 7), P E parents(Y)).
'The space complexity refers to the extra space needed for the backward pass, not including the forward pass.

---

We present the trace feedback (f. g) computed by Trace as a gpeudoaalorithng problem: the subgraph g expressed as a report of code execution with information about the computed values and descriptions of operators in g. Then we prompt the LLM to update the parameters in g based on feedback f given to the output. Fig. 4 shows an exam- ple. It is crucial to note that even though the lines look like an actual program, is not the real program but the computational graph defined by bundle (see Section 3.2).
#Code:
bar
y add (b,
2 = mul [a. y)
#Definitions:
[mul] This
nultiply aperatar.
[add] This an add operator
[bar] This is a nethod that does
negative scaling.
#inputs:
b=1.0
#0thers:
a=2.8
y=3.0
#Dutput
Parameter Update We prompt the L.LM with a ReAct- CoT style prompt (Appendix G.2) in one query, asking to generate reasoning based on the graph, and suggestion on the parameter changes. If the suggestion can be extracted from the response, we update the parameters.
z=6.0
#Variable
x=-1,0
#Feebback:
Output should be larger
Figure 4: An example pseudo-code report generated by Trace for a program of x Node () and the objective of max,
Optimization Memory OptoPrime optimizes most workflows reasonably well using just the traced graph and feedback, but it can run into issues when single feed- back alone is not informative (e.g.. the output feedback is rewards, but there is no description of how the rewards are generated). For robustness, we have a basic memory module in OptoPrime, which tracks the past parameter-feedback pairs as in-context examples. See Appendix G for details.
bar * (bar (x)+
5 Experiments
We evaluate the Trace framework with OptoPrime. We implement the statc-of-the-art LLM optimizer OPRO [13] baseline; in comparison with OptoPrime, OPRO does not use the execution trace but relies on the memory of parameter and feedback pairs. For these experiments, we use GPT-4- O125-Preview. We run the experiments on standard PC with 16 GB RAM, and Trace introduces no measurable overhead on executing the workflow. We also conduct experiments to compare Trace and OptoPrime with a concurrent AutoDiff-like framework, TextGrad [24], which was released after Trace was submitted to NeurIPS. We show that TextGrad can be easily implemented as an optimizer in Trace, and OptoPrime achieves similar or better performance than TextGrad while using much less computation time. In the rest of this section, we will denote Tracc+OptoPrimee simply as Trace. We report the token usages of all approaches in all experiments in Appendix I.I.
5.1 Validating with Numerical Optimization
First, we want to validate if OptoPrime can solve classical differentiable optimization problems, since they are a special case of OPTO. Consider the problem of min, [A(x)) - y"1 for a target y". We construct synthetic task environment that randomly creates y² and the computational graph of h with arbitrarily complex connections between numerical variables (see Appendix I.3 for details). We evaluate OptoPrime (denoted Trace) and variant that does not see the graph (Trace Masked); both the optimizers do not use memory. The output feedback "The output should be <larger/smaller>" (this feedback has the same information as the gradient w.r.t. h). We compare also the performance of Adam optimizer [25]. We run 30 trials over different randomly generated problems, All methods see the same randomness. Trace able to match the best-in-class Adam; on the other hand, without access to the full computational graph, the feedback-alone optimizer struggles to find x* (Figure 5a).
5.2 Tuning Hyperparameters to Orchestrate Complex Systems
We tested Trace traffic control problem, which is an instance of hyper- ppprr-pareettrr tuning. We used UXSim [26] to simulate traffic at a four-way intersection, where the trainable parameters are 2 integers in [15, 90). which are the green light duration for each direction of traffic flow. The feedback is the estimated delay experienced by all vehicles due to intersections, and the goal of an optimizer to minimize the delay using the fewest number of traffic simulations. To this end, this optimizer must find the right trade-off for temporally distributed and variable demands. In Fig. 5 we report the performance of a SOTA heuristic from the traffic control literature, SCATS [27] as well as two black-box optimization techniques: Gaussian Process Minimization (GP) [8] and Particle Swarm Optimization (PSO) [28]. All methods use the same starting parameters. Trace denotes OptoPrime using memory, and Trace NoMem denotes OptoPrime without memory. We report further details in
7

---

Number Optiadaation
Troffe Optinilattnnn
Trutly Oytinitaaee
Thace
Maviaed
23-
Pyleeck - Adaas
PAD
175
Traus Stocked
Delee Te
109
25
(a) Numerical Optimization. (b) Trace vs. Other optimizers.
(c) Ablations of Trace.
Figure 5: Numerical Optimization and Traffic Optimization Results.
Appendix 1.4. GP and PSO appear bad because 50 iterations are insufficient for their convergence; given enough iterations, both will eventually perform well. Trace is quickly competitive with the SCATS heuristic, whereas OPRO is not. Moreover, perform well for this task. But we note that Trace consumes extra overhead compared to other methods, since Trace has to materialize the resulting computation graph and query an LLM with effectively longer prompt than that of OPRO.
find that memory crucial for Trace to
5.3 Unifying Prompts and Functions Optimization
Many LLM agents today, e.g., specified by LangChain [29] and DSPy [22], have many components. These libraries provide optimization tools to optimize a small portion of their workflows, predom- inantly the prompt that goes into an LLM call. However, for building self-adapting agents that can modify their own behavior, only allowing the change to one part of workflow but not others can be limiting. In this experiment, we test Trace's 's ability in joint prompt optimization and code generation. Specifically, we optimize a given DSPy-based LLM agent and tune its three components: the meta-prompl prompt_template template, function create_ ppompt that modifies the prompt with the current question, and a function extract aarsaettaaaarrrrrrr that post-processes the output of an LLM call.
We set up an end-to-end prompt-and-code optimization pipeline. We use an automatic evaluation function to compare the LLM's output with the ground truth, which requires the LLM agent to generate outputs not only with the correct answer but also in the correct format (following the guidelines of [30]). We use the Big-Bench Hard dataset [21] (15 examples for training, 5 for validation, and the rest for testing). We compare Trace with DSPy's s COPRO module (which optimizes the meta-prompt). In Table 1, we show that Trace is able to optimize DSPy program beyond what DSPy's 's COPRO optimizer can, especially on algorithmic tasks. This result shows how Trace can concretely improve upon existing LLM prompt optimization libraries. We show learned codes in Appendix J.
BBH all (23 tasks)
NLP (12 tasks)
Algorithmic (11 tasks)
BBH all (23 tasks)
NLP (12 tasks)
Algorithmic (11 tasks)
0-shot
O-shot
DSPy
41.6
53.8
32.6
DSPy + CoT
70.4
73.7
68.0
DSPy-PO
55.3
69.0
45.2
DSPy-PO CoT
71.6
73.9
70.0
Trace 59.5
70.9
51.1
Trace + CoT
78.6
75.8
80.6
Table 1: End-to-end workflow optimization for an LLM henchmark (Big-Bench Hard) in 0-shot setup. CoT refers to Chain-of- prompting and PO refers to DSPy's 's own prompt optimizer (COPRO). We use Trace optimize a DSPy program, starting from the same program and prompt template specified by DSPy.
5.4 Long-Horizon Robot Manipulator Control
We test the ability of Trace to optimize long-horizon workflows with complex dependencies and "back-propagate through time". We use Trace train the controller code (in Python) for simulated Sawyer robot manipulator. We use the Meta- environment from LLF-Bench [23] as the simulator and consider three tasks: Reach, Pick-place, and Push. For each task, LLF-Bench provides task instruction and meaning of the action space, which we use as the context w of Python dictionary vectors, indicating the end-effector
the OPTO problem. The observation position, the goal position, the gripper status, etc. The action space is a 4-dimensional vector to control the relative position of the end-effeccoo and the gripper state. In each time step, the L.LF-Bench Meta-World World simulator returns the observation and natural language feedback to guide the robot. An

---

(a) Reach (Test)
(b) Pick-place (Test)
(c) Push (Test)
Figure 6: Learning the feedback control policy (code) for simulated Sawyer manipulator in LLF-Bench Meta-World In each iteration (x-axis), episode of rollout (10 steps) is performed, and then the policy updated. The mean and standard error the success rate over seeds are shown.
episode ends if the robot successfully solves the problem or because of time-out. We consider an episodic training setting. The initial condition for all iterations training is the same. We evaluate the learned policy terms of success, starting from 10 held-out initial conditions. The task horizon is 10 steps, which is sufficient for task completion, and each training iteration has one rollout. The output feedback in OPTO textual representation of task success. In addition the controller code, we also decorated the reset and step functions of the gym environment so that the entire rollout can be traced end-to-end. We compare Trace with OPRO; because of the streaming OPTO setting, our OPRO implementation only proposes one candidate in each iteration, which is then evaluated and provided with the output feedback.
The experimental results are summarized in Fig. 6. Trace denotes OptoPrime using memory, and Trace NoMem denotes OptoPrime without memory. We show learned code in Appendix J. OptoPrime is clearly the top-performing optimizer, especially the version with memory. OPRO is able to solve Reach at the start, but its performance degraded over iterations (this instability was mentioned in [13]) and gets a similar performance as OptoPrime (without memory) in Push. To validate that the performance of OptoPrime is indeed due using the execution trace, include ablation where we mask out the execution trace. which leads to significant decline in performance and stability. This experiment features the most complex graph structures among all the experiments. The experimental results here are quite impressive, showing that Trace is able to learn sophisticated control logic in dozens of interactions, not only working on the training initial conditions but also on held-out testing ones too. We discuss some limitations in Appendix I.6.
5.5 Comparison with TextGrad
After the submission of our work to NeurIPS, another AutoDiff-like framework, TextGrad [24], was released, which shares the same goal of end-to-end optimizing AI workflows as Trace. In comparison, TextGrad propagates text feedback, whereas Trace propagates minimal subgraphs (see Section 3.3). The graph-based design of Trace, which separates the tracing infrastructure and optimization algorithms, makes it more flexible. In fact, we easily implemented TextGrad as an optimizer the Trace framework, but the reverse not possible (because TextGrad couples the infrastructure and the optimization algorithm together). In addition, unlike TextGrad, Trace supports jointly optimizing heterogeneous parameters and can be applied to directly trace given computational workflow without the need to rewrite the workflow using pre-defined templates. Please see Appendix H for more discussion comparing the two frameworks.
In this experiment, we apply Trace to directly decorate the evaluation code released with the TextGrad library and optimize the parameters following their training/evallationn pipeline lyne-by-line. This experimental design makes the comparison fair by allowing each optimizer to access the same LLM APIs around the same time, and showcases the flexibility of Trace framework to optimize any com- putational workflow. We pick the Solution optimization [24, Table 2] and Prompt optimization [24, Table 3] for the reasoning tasks experiments. Please see [24] for details on the exact setup. We compare OptoPrime, TexIGrad", and reimplementation" 10 of TextGrad as an optimizer in Trace. We find that all these algorithms achieve similar success rates in these experiments. One noticeable difference is that OptoPrime is about 3x faster wall-clock time than TextGrad since OptoPrime makes a single call to LLM in each optimization step, whereas TextGrad calls linear to the graph's size.
The numbers the original paper cannot reproduced exactly despite using the released TextGrad code. All algorithms in Trace see the same graph, which is slightly different from the graph of TextGrad.

---

OptoPrime (Trace)
TextGrad (24-10-30)
TextGrad (Trace)
TextGrad (Reported)
Time
Time
MMLU-MMchinne Learning
86.6 (0.2)
1.7 (0.6)
86.1 (0.5)
3.5 (1.1)
86.3 (0.2)
88.4
MMLU- Colllee Physics
94.1 (0.8)
1.2 (0.3)
93.1 (0.7)
2.3 (0.4)
93.3 (0.6)
95.1
Google-proof QA
59.6 (1.3)
12.2 (1.4)
53.2 (0.6)
19.5 (1.9)
54.0 (0.7)
55.0
BBH Counting
89.4 (0.1)
55.9 (4.5)
89.2 (1.2)
142.9 (9.3)
87.6 (1.7)
91.9
BBH Word Sorting
71.6 (3.1)
82.5 (10.1)
72.0 (0.4)
211.1 (16.8)
71.4 (2.5)
79.8
GSM8K
82.5 (0.1)
82.4 (0.6)
82.0 (0.2)
81.1
Table 2: Comparison between Trace and TextGrad. The optimizer is GPT-40-2024-088-0 and the student
model is GPT-35-turbor 106. The results show the mean and the standard crror of success rate of the last iterate computed by 5 seeds. The experiment time reported is in minutes (the time involves not just training but also validation and testing by running TextGrad's original pipeline); the time of GSM8K experiment is omitted as the experiment time (>8hrs) is determined primarily by the evaluation not optimization.
6 Limitations
We highlight that Trace OPTO and OptoPrime are first step towards end-to-end generative opti- mization and building self-adapting workflows. They have limitations in their current form. OPTO captures rich feedback, but it is important to specify solution concept as well as the feedback source. We provide guidance for feedback design in Section 3.2 and discuss notions of optimality in Appendix F. We believe designing feedback will be important as designing loss function in deep learning, both of which are open research questions. Also, Trace cannot convert all computational workflows into OPTO problems, e.g., stateful functions that modify their state in place cannot be represented as a DAG without modification, and distributed/parallel computing workflows are incom- patible with the current implementation (though in theory Trace run in asynchronous way so long as the overall graph does not end up with cycles). Finally, while Trace is designed to be generic and future-proof, the OptoPrime optimizer is preliminary. Although we demonstrated that OptoPrime could work well with moderate-size graphs, it is not a provably optimal algorithm and uses more tokens than OPRO, though, in our experiments, OPRO's performance does not improve even when given a large token budget. The debugging ability and context limits of the LLM used in OptoPrime crucially determine the scale of problems that we can practically address today. Consequently, more research is needed for designing token-efficient generative optimization algorithms.
Conclusion and Future Work
We created Trace that can convert a computational workflow optimization problem into an OPTO problem, and we demonstrated tractable OPTO optimizer, OptoPrime. This is just a first step towards a new paradigm of end-to-end generative optimization, with exciting avenues for future work. We discuss a few selected ones below. Please see Appendix A for longer discussion.
In OptoPrime, we connect optimization an LLM's reasoning capability. Techniques that have been proposed to improve LLM reasoning, c.g. Chain-of-Thougee [31], Few-Shot Prompting [32], Tool Use [33], and Multi-Agent Workflows [5] could also help improve OptoPrime or design new OPTO optimizers. We conjecture that hybrid workflow of LLM and search algorithms, can enable a truly general-purpose OPTO optimizer. Along the way, we must settle on how to delineate the agent vs. the optimizer. How to trade off the generality of optimizer vs. crafting side-information in the context w to achieve task spee-fie performance is an open question.
In Trace, we chose specific propagator (MSP), which maximally preserves information for general computation graph. We can instead specialize for specific computations, e.g. to accommodate very large graphs with a hierarchical graph representation. Going a step beyond the basic memory module we experimented with in OptoPrime, we anticipate that an optimizer that can reason about how a workflow will behave under counterfactual parameter settings can be more efficient than OptoPrime and can enable a wwvide-and-congeee study the theoretical properties of OPTO (such as optimization landscape and complexity). We hope our preliminary effort Appendix F can provide some guidance. Finally, this paper, we focused on output feedback and context that can be compactly textualized. We anticipate that computational workflows with rich non-textual contexts and output feedback will also benefit from automatic generative optimization through appropriate applications of Trace (e.g., with VLMs).
approach to OPTO. More research needed to
10

---

Acknowledgments and Disclosure of Funding
We would like to thank John Langford, Ahmed Awadallah, Jennifer Neville, Andrey Kolobov, Ricky Loynd and Paul Mineiro for thought-provoking discussions. We would also like to thank Tobias Schnabel, Rujjie Zheng. Wanqiao Xu, and Kaiwen Wang for their valuable feedback on an early draft of this manuscript. Additionally, we thank Anqi Li, Omar Khattab, David Hall, Yifan Mai, Bryan He, Yash Chandak, Emma Brunksill, and Dawen Liang for their suggestions and feedback. The work was partially done while Adith Swaminathan was at Microsoft Research.
References
[1] Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quiney Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models compound Al systems. https:/ //bai: http://baaabbbbeeeeeeeeeee berkeley. cdu/bllog/222//02/118/ compound-ai-systene 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint S10::::3030087744 2023.
[3] Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simis- ter, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In SIGPLAN International Symposium on Machine Programming, page 21-29, 2022.
[4] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference Robotics and Automation, pages 9493 9500, 2023.
[5] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08. 08.0000..555... 2023.
[6] Andrew R Conn, Katya Scheinberg, and Luis N Vicente. Introduction derivative-free optimization. SIAM, 2009.
[7] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, pages 507-523. Springer, 2011.
[8] Peter Frazier. Bayesian optimization. Recent Advances in Optimization and Modeling of Contemporary Problems, pages 255-278, 2018.
[9] Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation, 1(1):1-23, 1993.
[10] Josep Ginebra and Murray K Clayton. Response surface bandits. Journal of the Royal Statistical Society Series Statistical Methodology, 57(4):771-784, 1995.
[11] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis. Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In ICLR, 2023.
[12] Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. In EMNLP, pages 7957-7968, 2023.
[13] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR, 2024.
[14] Tobias Schnabel and Jennifer Neville, Prompts as programs: structure-aware approach to efficient compile-time prompt optimization. arXiv preprint arXiv:2404.02319, 2024.
11

---

[15] Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (STOP): Recursively self coff-impoviin code generation. arXiv preprint arXiv:2310.02:54, 2023.
[16] Allen Nie, Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. The importance of directional feedback for L..M-based optimizers. arXiv preprint arXiv:2405. 16434, 2024.
[17] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(6088) 1986.
[18] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. Journal of Muchine Learning Research, 18(153): (4153))1-43, 2018.
[19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: an imperative style, perforrefee deep learning library. In NeurIPS, pages
2019.
[20] Karthik Nagaraj, Charles Killian, and Jennifer Neville. Structured comparative analysis of systems logs to diagnose performance problems. In USENIX Symposium on Networked Systems Design and Implementation, pages 353-366, 2012.
[21] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging BIG-Bench tasks and whether chain-of-thougle can solve them. In Findings of the ACL, pages 13003-13051, 2023.
[22] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq. Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023.
[23] Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LLF-BBnch: Benchmark for interactive learning from language feedback. arXiv preprint arXiv:2312.06853, 2023.
[24] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad: Automatic "differentiation" via text. arXiv preprint 240.00466. 2024.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[26] Toru Seo. UXsim: An open source macroscopic and mesoscopic traffic simulator in python a technical overview. arXiv preprint arXiv:2309.17114 2023.
[27] Courtney Slavin, Wei Feng, Miguel Figliozzi, and Peter Koonce. Statistical study of the impact of adaptive traffic signal control on traffic and transit performance. Transportation Research Record, 2356(1): 1):117-126, 2013.
[28] James Kennedy and Russell Eberhart. Particle swarm optimization. In ICNN, volume 4, pages 1942-1948, 1995.
[29] LangChain Team. Langchain tracing. https://blog
2023.
[30] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating L.LMs as agents. In ICLR, 2024.
[31] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models. In NeurIPS, pages 24824-24837, 2022.
12

---

[32] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[33] Timo Schick, Jane Dwivedi-You Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In NeurIPS, 2023.
[34] James Wexler. Artificial Intelligence in Games: A look at the smarts behind Lionhead Studio's 's "Black and White" and where it can and will go in the future. https://wwwt cs. rochester. edu/-brown/242// http://///////////aaa///
/gwm/s.asssss.. pdf, 2002.
[35] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh R
N. Zeyuan Chen, Jianguo Zhang. Devansh Arpit, Ran Xu, Phil L Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient optimization. In ICLR, 2024.
[36] Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Côté. Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimiza- tion of stacked LLMs using variational inference. In NeurIPS, 2023.
[37] Mingehen Zhuge, Wenyi Wang. Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jurgen Schmidhuber. Language agents as optimizable graphs. arXiv preprint arXiv:2402.16883 2024.
[38] Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, and Xuanjing Huang. Are large language models good prompt optimizers? arXiv preprint 220.0000 2024.
[39] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. PromptAgent: Strategic planning with language models enables expert-level prompt optimization. In ICLR, 2024.
[40] Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Siyuan Lu, Yaliang Li, and Ji- Ji-Rong Wen. Unleashing the potential of large language models as prompt optimizers: An analogical analysis with gradient-basee based model optimizers. arXiv preprint
2024.
[41] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In ICLR, 2024.
[42] Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Guojun Peng, Zhiguang Cao, Yining Ma, and Yue-Jiao Gong. LLaMoCo: Instruction tuning of large language models for optimization code generation. arXiv preprint arXiv:2403.0113/1 2024.
[43] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In ICLR, 2024.
[44] Michael R. Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. arXiv preprint arXiv:2312.04528, 2023.
[45] Gábor Bartók, Dean P Foster, Dávid Pál, Alexander Rakhlin, and Csaba Szepesvári. Partial m classification, regret bounds, and algorithms. Mathematics of Operations Research, pages 967-997, 2014.
[46] Laurent Hascoet and Mauricio Araya-Polo. Enabling dsiren checkpointing strategies in reverse-mode automatic differentiation. arXiv preprint cs/0606042, 2006.
[47] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots Truncated back- propagation for bilevel optimization. In AISTATS, pages 1723-1732 2019.
[48] Ching-An Cheng, Jonathan Lec, Ken Goldberg, and Byron Boots. Online lcarning with continuous variations: Dynamic regret and reductions. In AISTATS, pages 2218-2228. 2020.
[49] Sham Machandranath Kakade. On the Sample Complexity of Reinforcement Learning. PhD thesis, University College London, 2003.
13

---

A Perspective: Deep Agent Workflows
We posit that the current practice of manually engineering computational workflows to build AI systems is analogous to programmers in the early 2000s hand-coding neural network weights to create engaging AI characters in video games [34]. Just like AutoDiff enabled the automatic and scalable optimization of deep neural networks with billions of parameters, we believe that Trace is the first step towards automatic and scalable optimization of "Deep Agent Workflows" to power even more capable AI systems. However, there are several limitations of the current implementation of Trace that need to be addressed to build Deep Agent Workflows.
When designing interactive AI systems that learn from their interactions, we need to define the parameters and feedback of the system. Parameters are the internal attributes that can be updated by the learning algorithm employed by the system. Feedback are the things observed and recorded by the system as a product of its interactions, and that provide signal for learning. Trace enables the development of new learning algorithms (e.g. through OptoPrime) that incorporate rich feedback to update heterogenous parameters. In contrast, Aufo for deep neural networks uses numerical feedback (e.g. rewards loss functions) to optimize numerical parameters (c.g. tensors). Black-box optimization techniques (e.g. Reinforcement Learning) can use numerical feedback to optimize heterogenous parameters (e.g. codes, typer-parameters as well tensors), though they are inefficient. In the experiments, we saw that Trace was more efficient than black-box methods by using a generalization of back-propagation. Finally, Trace can use rich feedback (e.g. language) extract more signal for learning.
What can be traced? Trace cannot convert all computational workflows into OPTO problems. Workflows with recursive bundle operators or those requiring distributed/paralle computing are not compatible with the current implementation. A future work would be to expand the Trace implementation to support these scenarios. In addition, the current implementation of Trace does not trace the execution within an operator defined by bundle, though in principle this is possible. There are also ambiguities in how an existing workflow can be traced and represented as DAG. One example is when there is some sub- wor-workllow following an if condition and another one following else. One can choose wrap the entire code, including if and else, by bundle as a single operator. On the other hand, one can also just wrap the sub-workflows and not trace the if condition nor represent it as part of the DAG. (That is, suppose the if condition is true; from the DAG, one cannot see the alternate path under else). The latter choice has flavor of applying the boolean condition, whereas the first choice enables pack-propagation through also the logical condition. We summarized some design considerations in Section 3.2. We foresee the choice of what to trace and how to trace in building Deep Agent Workflows will be an on-going research problem, similar to neural network architecture design.
on
Where do get rich feedback? The OPTO framework captures an abstraction of rich feedback, called trace feedback (the execution trace and the output feedback), but it requires specification of operator descriptions and output feedback source to guide the optimization effectively. OPTO can be more efficiently solvable than black-box problems only when the trace feedback provides information beyond reward signals; otherwise, information-wise, OPTO is easier than black-box problems. The Trace framework automates the generation of the execution trace in OPTO, when users of Trace decorate the workflow end-to-end. Currently, OptoPrime uses the docstring of operators in the execution trace to understand the operators; nonetheless, Trace logs also the source code of the decorated methods, which can also be used in the future to design optimizers that uses more details of the operators. In our experiments, we focus on output feedback that is automatically generated. In other contexts. e.g. users interacting with chatbots, we can natively gather natural language feedback or synthesize feedback [16] based on raw observations. We also anticipate that feedback in the form of images (e.g. users' gestures) or videos (e.g. videogame player showing a desired correction to agent's 's behavior) will be readily available and can be used as the output feedback in the OPTO framework. While Trace can handle this feedback, the current design of OptoPrime does not handle non-textual content (either in parameters, inputs or feedbacks). We anticipate future work along the lines of [16] on enhancing feedhack design and developing guidelines for designing more informative and directive optimizer and feedback mechanisms.
14

---

How to design adaptive optimizer for general OPTO problems? The proposed OptoPrime optimizer shows the possibility of designing a single optimization algorithm to solve range OPTO problems from diverse domains. However, we remark that OptoPrime is just the first step; it is akin to the vanilla gradient descent algorithm, which shows a proof of concept but is not scalable for large problems. The current design of OptoPrime has several scalability limitations due to its summary approach that updates parameters through one call to an LLM. While we show in Section 1, that OptoPrime can compete with ADAM in small problems, OptoPrime is not as computational efficient and cannot scale as well as ADAM; therefore, for large-scale numerical problems, Trace and OptoPrime with the current design does not replace classical Auto OptoPrime is also limited by the ability of LLMs. It has difficulty in handling parameters or nodes that cannot be compactly represented in text, which prevents it from optimizing neural network weights, reasoning with large, stateful objects like a database. Similarly, it likely cannot handle large graphs (with thousands of nodes) the moment, such a large graph would result in huge context which may be beyond what LLMS can understand and reason about reliably. This limitation is aggravated when dealing with noisy feedback systems, we need to present multiple graphs and feedback in the context once. We also do not know how to rigorously define the concept of step size, which however we expect is important to handle noisy or local feedback. We need further research on graph simplification and representation, to reduce complexity and improve the efficiency of feedback propagation. Lastly, right now OptoPrime represents the constraints on parameters as part of the text description, but we have observed that LLMs do not always follow it. An effective workaround is to implement constraint checking in the workflow to throw exceptions (which are then handled as feedback). More specialized constraint handling techniques is an interesting research direction (e.g. "projecting" OPTO solver proposals for parameters onto their feasible sets), but they are not implemented in OptoPrime yet.
B Related Work
Framework for Computational Workflows Frameworks such as LangChain [29], Semantic Kernels, AutoGen [5], DSPy [22] allow composing computational workflows and provide hand- engineered optimizers to tune an LLM's context (i.e. prompt templates, few shot examples, or tool libraries) using scalar feedback with black- box search techniques. They support tracing of the workflow to aid in profiling, debugging and visualization. In contrast, Trace uses tracing for automatic optimization, and constructs a different representation of the computational graph which suited for that purpose. Moreover, Trace is designed to be general-purpont and agnostic to the underlying frameworks of computational workflows users choose. In principle, one can apply Trace decorate and tune a workflow based on mix of Autogen, LangChain, DSPy codes. In fact, our experiments in Appendix use workflows declared using both AutoGen and DSPy.
Optimization of Graphs of LLM Workflows There are multiple efforts to optimize the computa- tional graph of LLM workflows, which is special case of the OPTO problem. These algorithms focus on optimizing prompts. SAMMO [14] is an example for prompts that uses additional graph structure to make the optimizer efficient. SAMMO represents the prompt parameter itself as program so as to enable more efficient black-box search through the space of programs. DSPy [22] can optimize directly the prompts the few-shot examples to include using scalar reward feedback. Retroformer [35] uses another small language model (LM) to provide suggestions/feedbacck (i.e. changing prompts) to improve the behavior of an actor LLM, where the small LM is tuned by offline RL. Deep Language Networks [36] view all of the prompts in an LLM workflow as tunable parameters and jointly optimizes them. They discovered that optimizing each parameter in isolation instead produces subpar results. [37] frames LLM systems as graph where nodes are operations and edges are messages/connectionss. (Note that this is different from the DAG used in Trace; here nodes are messages and edges are input-output of operators) and optimizes for the connection on edges (binary variables) by REINFORCE using scalar reward feedback and prompts by LLMs They optimize each component separately without considering each other; for example, the prompts are optimized individually without considering the graph topology or how they are used down the road. We suspect this approach can be less stable. Their prompt optimization part also does not take output feedback, but simply use an LLM to self-check whether the prompt meets the need of generating desired functions the user specified. In contrast to these works, through the OPTO framing, Trace supports joins optimization of all parameters (prompts, hyperparameters, codes) with rich feedback, and is agnostic to graph structures (c.g., changing these parameters can dynamically change the graph
15

---

structure and connections between nodes). Users of Trace are free to specify which parameters they want automatically optimize via online interactions.
LLM-Optimizers for Prompts and Codes There is huge and fast growing literature using LLMs as optimizers to improve prompts [12, 11, 38-40] or codes [41, 15, 42, 38]. Different from the works mentioned above, here the focus has been on isolated problem (e.g.. changing the format) rather than considering non-trivial workflow or agent with multiple components like above. They do not consider optimizing prompts or codes as one component of a bigger workflow (c.g. implementing an autonomous agent), which is harder and requires the right credit assignment. In addition. these LLM-based optimizers, including OPRO [13], often propose only principles of how prompts should be designed and requires crafting problem specific prompts (as opposed to single optimization prompt that can be applied to different problems). For adapting them to new problems, users need to design new prompts. Trace can also be applied to optimize trivial OPTO problems where the returned graph has just single node of the parameter (which are the scenarios considered by these works). Nonetheless, the main focus of this paper is to study how optimization can be done efficiently as the graph becomes nontrivial and for diverse applications. Trace achieves this by using the abstract OPTO problem framing. Since OPTO encapsulates domain specific info in the graph, it enables designing fully instantiated optimizers that can be applied to multiple problems, rather than just principles which then requires hand crafting prompts for individual problems like previous works.
behavior of single LLM or improving the code generation in the anestion-answeriin
LLM-Optimizers for Hyperparameters Recent works like [43, 44] use LLMs to optimize numer- ical hyperparameters, as an alternate to Bayesian optimization. Here in the experiments we show that Trace + OptoPrime also can effectively learn hyperparamters, faster than Bayesian optimization. The main difference between Trace and the aforementioned work is the representation of the prob- lem. Trace, we provide the graph to the representation), and we consume rich language feedbacks on the output, both of which accelerates hyper-parameter optimization.
optimization (through the alggoritm
OPTO Related Setups ОPTO is a generalization of partial monitoring games [45]. If there exists a latent loss function that the feedback f adheres to (e.g. as in [23]). those OPTO instances can be written as partial monitoring game. However OPTO admits a more general notion of feedback f, and we discuss solution concepts for them in Appendix F. On the other hand, OPTO can be also viewed as a special case of Learning from Language Feedback (LLF) setup defined in [23] with observations
as the trace feedback. This is framing of a meta LLF problem. In the LLFBench experiments of this paper (Section 5), we show Trace can be used to learn policy for LLF problems grounded to an application too.
WorldWorld
Back-propagation has been shown to be a very effective tool in optimizing differential computational workflows. Our design of Trace is inspired by back paek-propagation and the ease of use of the AutoDiff framework PyToorch [19]. Nonetheless, we highlight that back- propagation (Backward Mode Differentiation) is not the only AutoDiff algorithm. For example, the gradient can be computed in a forward mode (Forward Mode Diffcrentiation) as well, and there are also techniques of Checkpointing [46] and Truncated Back-Propagation approximation [47] for efficiency. What are the equivalent ideas of these methods for general computational workflows? We think this is an interesting future research direction.
AutoDiff' and Back-propagattinn
C Examples of OPTO
To ground the OPTO setup, we show how OPTO related to some existing problems with examples. Example 4 (Neural network with back prok-propagation). The parameters are the weights. 9 is the neural computational graph and f is the loss. An example context can be "Minimize loss". The pookppropagttion algorithm, in view of the OPTO formulation, is embedded in the OPTO optimizer. For example, an OPTO optimizer here is a composition of pack-propggttion and gradient descent, where back- propppropagttion takes T to compute the propagated gradient at the parameter.
Example (Code Debugging). The parameters are the codes. و denotes the stacked trace and f is the error message returned by compiler. can be "Make no error".
16

---

Example 6 (RL). The parameters are the policy. 9 is the trajectory (of states, actions, rewards) resulting from running the policy in Markov decision process; that is, g documents the graphical model of how an action generated by the policy, applied to the transition dynamics which then returns the observation and reward, etc. f can be the termination signal or a success flag. w can be "Maximize return" or "Maximize success".
Example (Hyperparameter Tuning of ML Pipeline). The parameters are e.g. learning rates and architectures. g describes the stages of the ML pipeline and the evaluation on the validation set, and f is the validation loss. w can be "Minimize validation error".
Example (Prompt Optimization of an LLM Agent), The parameters are the prompt of an LLM workflow. is the computational graph of the agent and f is the feedback about the agent's 's behavior (which can be scores or natural language). w can be "Maximize score" "Follow the feedback".
Example 9 (Multi-Agent Collaboration). The parameters are each agent's prompts. y describes the entire conversation flow between agents, and f is the feedback about whether the task is successful after each agent performs their action. can be "A group of agents coordinate to finish a task.".
As mentioned, the computational graph g returned by the Trace Oracle T may have different graph structures. The length of the execution trace, e.g., in the debugging example above depends on how far the code executes. Similarly, the rollout length of in the RI. problem can be randomly determined. The formulation of the Trace Oracle abstracts the details of computational workflow, so problems from different domains can framed in the unified framework. This abstraction allows us to design the computational tool Trace for various applications.
D Trace Handles Error in Execution as Feedback
is worth mentioning that execution error can be directly used as feedback to optimize parameters in Trace. When execution error happens within a method decorated by bundle, Trace would adds special exception node to the global computational graph and throw an TraceExecut inneror to stop the computation. The computational graph ends at where the execution error happens. This exception node becomes the new output of the inputs to the decorated method (since the original method raises an error) and is the output of the truncated computational graph. Messages in TraceExecutionss Ennrrr can then be used as the feedback f in OPTO and propagated from the exception node to the parameters. be updated to avoid causing the same execution
By calling an OPTO optimizer, the parameters error. See the exception handling code in Fig. 2.
We find that this error handling mechanism has two convenient usages. First, this allows using Trace to automatically debug issues in the workflow due to incorrect parameter settings. Such errors can happen frequently especially when codes are parameters, as during optimization codes not satisfying syntax or downstream API requirements can happen. The second usage is to enforce constraints the workflow has to satisfy at different stags of computation. With Trace, if an intermediate computed result does not satisfy the constraint, we can simply throw an exception which states the desired constraint. This error signal would be caught by Trace and can then provide early feedback to efficiently improve the parameters, since the graph is truncated at the error.
E
Analysis of Trace
E.1 Proof of Complexity
Algorithm propagates the subgraph, represented by a priority queue (implemented as a min-heap). At time, it needs to maintain the subgraphs coming from W children separately. This leads to the space complexity of O(WN). This O(WN) space complexity leads to the extra N log N factor in the time complexity of MSP compared with propagto,,,,,,n,, which is the time needed for merging W subgraphs of size O(N).
F.2
Proof of Lower bounds
Consider an OPTO problem whose goal is to find a parameter matching k-digit binary number. The computation checks each digit against a reference number in an arbitrary order. The feedback either th check failed" or "All checks succeeded". Propagated feedback must communicate & bits
17

---

of information to interpret the feedback correctly; and the minimal subgraph conveys exactly that information. Updating the parameter using the minimal subgraph is trivial, whereas without it there are 2* possibilities to check.
F
When is OPTO Efficiently Solvable?
We show that OPTO covers wide range of complicated optimization problems. This shows that if OPTO can be efficiently solved, then many complex workflows can be efficiently optimized. However, the generality of OPTO also raises some fundamental questions, such as if OPTO is well defined and when OPTO can be efficiently solved. These questions stem from its generality of the context w and the output feedback f in OPTO, since e.g. they can be anything descriable texts. This flexibility makes the scope of OPTO go beyond standard mathematical optimization problems, where a setup has fixed context (e.g., "First-order optimization") and fixed type of output feedback f (a descent direction). Fully characterize the properties of OPTO, due to its generality, is beyond the scope of this paper and would require years of future research come. Nonetheless, here attempt to provide some preliminary answers and point out some research questions.
F.1 What is a solution?
Classical mathematical optimization problems have problem definition which itself the solution concept. For example, in minimization problem, it is clear we want to find the minimum of an objective function; even for problems as abstract and general an equilibrium problem, the problem setup clearly states the solution concept of finding point/set satisfying an equilibrium inequality [48]. One common pattern of these problems is that the solution concept something that can be described as conditions on feedback that the parameter should satisfy.
By contrast, in OPTO problem (O,w, w, T). by varying the context w, the desired parameter can change from one extreme to another. For example w may state "Follow the feedback" or "The need define the solution concept of OPTO differently, rather
feedback is adversarial.". Therefore, than just using the feedback. We need to also consider the context w appropriately. Below we make an attempt to give an axiom of OPTO for its solution be well defined.
Axiom (Verifiability). There is verification oracle (a human, machine learning model, polynomial-time algorithm) when given (0, w, f) can verify whether 0 is a solution or not.
Notice the verification oracle in Axiom not limited to just algorithms. This is intentional because we currently do not have algorithms that are intelligent enough to process the wide range of contexts and feedback that OPTO allows. Therefore, we include human judgement or the use of LLMs or other AI systems part of the definition, while acknowledging the impreciseness of the statement due to OPTO's soft computing nature. Lastly we note the verifiability only defined with respect to the context w and the output feedback f. not the execution trace y. That is, the verification of solution depends only on the output of computation.
F.2 Does a solution exist?
Under Axiom we can start ask the basic question of whether a solution to an OPTO problem exists or not. There are clearly problems where no solution exists (that is, no parameter in O can be verified by the verification oracle). For example if the feedback f is contradicting and yet the context is "Follow the feedback.", then there would be no solution that is satisfactory. On the other hand, if "Ignore the feedback", all parameters can be solutions. In the following, we assume solutions of OPTO under consider exist. This assumption would rule out problems, c.g., where the feedback is adversary to the context, and makes solving OPTO well-defined search problem.
Assumption For an OPTO problem (0, w. T), we assume there least a parameter such that it can be verified as solution by the verification oracle.
F33 Can OPTO be efficiently solved?
So far our discussion establishes OPTO as well-defined search problem, based on qualification on the context and the output feedback f. However search problems can be harhard. In other words,
18

---

we know that, without the execution trace, there are search problem instances modeled by some w and oracle giving f that cannot be efficiently solved. Take RL for tabular MDP an example of OPTO problem. Without the execution trace (i.e., not seeing the Markovian structure and trajectories), the problem has an exponential complexity (due to the size of the policy space) and we know by using the execution trace here, tabular RL can be solved approximately in polynomial time [49]. Another example is training neural networks. Without the execution trace, we have a complex black-box optimization with loss value, without gradients, whereas execution trace allows implementation of pack-propagation to compute the gradients at the parameters.
More broadly speaking, if we consider a "human" as an optimizer for OPTO, we see that (expert) engineers/researhhers when equpped with additional computational tools, can efficiently solve broad range of OPTO problems (such as by using the execution trace. From these observations, we conjecture using information in the execution trace is the key to unlock efficient OPTO. More precisely, we conjecture that OPTO is efficiently solvable when the context and the trace feedback need provide information to construct corrective search direction. For example, when the output feedback back is just a scalar loss, and yet the context + execution trace feedback does not provide enough information to compute descending direction then OPTO reduces back to black box problem. (See the problem instance in Appendix E.2). Nonetheless, identifying which subsets of OPTO are efficiently solvable a big open research question.
G A Additional Details of Trace and OptoPrime
G.1 Backward Step of Trace
The MSP extracts the minimal subgrpah of the full computational graph the workflow. Here we show visualization using the example in Fig. 4.
[arameterrNoe]]
10the parts
Exccution Flow Reedbacck Flow
-1.0
ACde::
bar
y add
2 nul nul(a, y)
ADefinitions:
[Node] A node...
[bar] A function...
[mul] This
multiply operator [add] This is an add operator.
1.0
[bar] This is a method that does negative scaling.
AInputs:
[add] add imputs..
b=1.8
3.0
a0thers:
a=2.0
y=3.0
z [mul] multipl... 6.0
abutput
2=6.0
aVariable
x=-1.8
Foodback
#Feedback:
Output should larger.
(a) This is an illustrative example of the graph con- structed by Trace and how feedback is backpropagated to the parameter
(b) We create succinct summary of the computa- tion graph using language that mimics program.
Figure A.I: Optimization Representation. For program of node (-1.0) z = a * and the optimization objective of max, z, Trace automatically constructs computation graph and represent the optimization problem as debugging report. Note that the real program and the traced execution graph different.
bar (x) ; y = a 1;
G.2 Prompts used in OptoPrime
OptoPrime an LLM-based optimizer. Its prompt is composed the following parts.
1. System Prompt: Representation Prompt (Fig. A.2) + ReAct+CoT Output Prompt (Fig. A.3)
19

---

2. User Prompt (Fig. A.4 or Fig. A.5)
where + denotes concatenation. We list the prompt templates of different components below.
You re tasked to solve haltinn/aggrrthmm problen. You will see the instruction, the code, documentation result.
function used code,
Specifically, problem will composed of the following parts: #Instruction: the instruction which describes the things you need to do or the question you should
#Code: the code defined in the problen. #Documentation
documentation of function be incomplete and just contain high-level description. You can use the values in #0thers to help infer how those functions work.
acode. The explanation might
@Variables: the input variables that you change.
#Constraints: the constraints or descriptians of the variables in #Variables. changeable. #0thers: the intermediate values created thraugh the code execution. code output.
#Inputs: the values of other inputs the code, which
10
11
#Outputs: result
12
#Feedback: the feedback about the code's execution result.
13
14
In aVariables, dInputs, #Outputs, and #0thers, the format is:
15
<data,type> <variable_nane
<value>
17
18
<type> is (code), means <value> is the source code of python code, which may include decstring and definitions.
19
Figure A.2: Representation Prompt that phrases the OPTO update as pscudo-algorithm question.
Output_format. output syntax:
following jsan fornat, satisfying json
"reasoning": <Your reasoning?,
"answer": <Your answer?,
"suggestion" {(
<variahle_12; Esuggested_valuee <variable_22; Saulessted_value_22 })
10
11
"reasoning" explein problen: aFeedback on #0utput means to #Variables considering how #Variables are used in #Code and other values #Documentation, #inputs, #0thers. 3. Reasoning about the suggested changes in #Variables (if needed) and the expected result.
#Instruction neans
13
14
If Ansttruction asks for
write it dawn in "answer".
15
you need suggest change values in "suggestion", Remenher can change only the values in #Variables, nat others. When <type> of variable is (code), you should write the new definition in the fornat of python code without name or the function signature.
aVariables,
down suggested
errors, and should not change the function
18
If no changes or answer are needed, just output TERMINATE.
Figure A.3: ReAct+CoT Output Prompt instructs LLMs should respond the format of (reasoning, answer. suggestion) and explains the output format.
20

---

you problem instance:
prrrual_probbem__nntteeee
Your response:
Figure A.4: User Prompt for OptoPrime without Memory
Now you see problem instance:
(actuel_problen_nneteme)))
Below are some variables and their feedbacks received in the past.
10
"variables"; (
(variablel_nane) name): (variablel_walue [variable2_name):: (variable2_valuel)
12
14
15
"feedback"; (feedback_1)
16
17
19
"variables":
20
(variahlel_nams) (variablet_walue2)
31
(variablez_name): (variablez.waluez)
22
24
"feedback"; (feedback_2)
26
27
28
29
Your response:
Figure A.5: User Prompt for OptoPrime with Memory
#Instruction
(instruction)
#Code
(code)
#Documentation
[dccueenaatin))
+Variables
11 {variables)
12
#Constraints
14 [constraints)
16 IInputs
17 (inputs)
18
19 #0thers
(others)
21
10tputs
23 (outputs)
Feedback:
26 (feedback)
Figure A.6: Prohlem Template used to fill the User Prompt. By default the Instruction (which is the context w of OPTO) is "You need to change the <value> of the variables in #Variables to improve the output in accordance #Feedhack."
21

---

H Comparison between Trace and TextGrad
Beyond the empirical results described in Section 5.5, there are several first principles reasons to prefer the Trace design to build future LLM-powered generative optimizers.
Joint vs. Individual Parameter Optimization: When there are multiple parameters to be opti- mized in workflow, Trace takes "joint optimization" view (constructing the minimal subgraph involving all the parameters described in the initialization of OptoPrime). This flexible design also allows alternative approach of "individual optimization" akin co-ordinate descent if desired, i.e., fix all but one parameter and individually optimize each parameter. Such an optimization heuristic be accomplished, e-g., by instantiating an OptoPrime instance per parameter and iterating through their updates; however, for many problems (like the toy example below). this strategy, which TextGrad employs, can be sub-optimal because consistent optimizer must additionally maintain state about any suggested updates to all other parameters when reasoning about an individual parameter update.
lebundle ()
def Function! (x):
return
@bundle()
Function2(y):
return y x
de xor_test (x, y)
return function1 (x) neq (fu(nctiion)()))
input1 = n3, trainable=True); imput2 node (4, trainable=True)
sor_test input?). backward (fenbbac=Fnn of inputs make the return True.
Listing 1: Individual Parameter Optimization is suboptimal in this example compared to Trace.
Heterogenous Parameters: Trace encapsulates all Python datatypes as node, including numbers, floats, strings, etc. In contrast, many generative optimization libraries like TextGrad restrict their "variables" (i.e. parameters and intermediate values) only be strings; consequently running numerical optimization experiments, or optimizing code alongwith prompts and hyper-pant prpee-pprameters can be challenging set up.
Modularity: Trace provides clean separation between the infrastructure (a platform akin to PyTorch) and generative optimizer (OptoPrime akin the Adam optimizer). Trace provides an object "TraceGraph" that describes the computational graph. This graph is independent of optimizers and LLMs. Such independent computation graph objects do not exist in TextGrad. We hope the Trace-provided interface can spur the development of many other generative optimizers, like how the OpenAI Gym API allowed the development of several Deep RL algorithms targeting a unified representation of RL problems.
Flexible Representation: The bundle functionality of Trace is more flexible in representing computations to an LLM than pre-defined templates used in other optimization frameworks. TexiGrad relies on special functions to chain variables together, such as using LFormattedLLMCall",,, while Trace supports any user-defined functions through decorators like bundle.
The strategy taken by OptoPrime can scale better to larger graphs (both in terms of етror accumulation across multiple LLM calls, and the costs of multiple LLM calls) compared to requiring one LLM call per computation node (and we see with bundle that not all computation nodes present equal difficulty for an optimizer). The difference in terms of computation costs between the two strategies can be large, especially when not all operators in the graph are as complex as querying L.M.. However there may be even better optimizers that blend OptoPrime and TextGrad so as to divide-andd-ooqueer huge computation graphs, we are excited about this avenue for future research.
Scalability:
22

---

I Experiment Details
L.1 Token Counts
We include token counts for the OPRO and OptoPrime prompts used across all our experiments, at the first iteration of optimization (note that OPRO's token usages grows linearly with iterations). We can see that indeed OptoPrime consumes significantly more tokens than OPRO. However, we observe consistently that even allowing 7-10x more iterations of OPRO OPRO performance plateaus to a worse level than OPTOPrime (c.g. Figure I: OPRO at Iter vs. OptoPrime Iter 2; Figure 5b: OPRO Iter 50 OptoPrime at Iter 5; Figure 6b: OPRO at Iter 30 vs. OptoPrime at Iter 10, etc.). OPRO is suboptimal not due to a token limit but instead a lack of information, which is captured and represented using Trace.
to equalize token costs, the
Domain
OPRO
OptoPrime
Numerical Opt
175
918
BigBench-Hard
400
1883
Traffic Opt
198
1679
MetaWorid
470
7101
Battleship
437
1305
Table A.1: Token counts.
1.2 Battleship
implement a simple battleship game board in Python. The exact code in the supplement. The game offers a string-based visualization of the board. It randomly places different types of ships 2-dimensional board with pre-specified width and height when it initializes. The agent does not see the ship location and has to select a coordinate on the board to hit next. One additional rule of this game is that the agent can go again if their previous coordinate selection (fire) is a hit, not counting as the finish of turn. In Figure 1, ran 10 trials, where in each trial, ran 20 iterations of training. We measure the reward as % of ship squares hit (over all squares occupied by ships). The reward plateaued at 60% because the game has chance element (heuristics and strategies only go so far strategy is only in effect if a hit happens. Otherwise, there is no information about where ships might be).
1.3 Numerical Optimization
Any classical numerical optimization problem can be framed as an OPTO problem. Consider (ar) and a target y", in a context finding the y" by changing 2; we know the most useful corrective feedback to change x the gradient Vą.. Similar to Trace, AutoDiff packages like PyTorch's AutoGrad have implemented dynamic graph construction with special classes like torch. Tensor. We want to validate whether it is possible to rely on binary text feedback, graph automatically constructed by Trace, and OptoPrime update x in the context of minimizing y y"|.
We constructed synthetic task environment where we can create a complex computation graph with arbitrarily complex connections between numerical variables. The focus of this environment is on the complexity of the graph, not on the complexity of the numerical operators. Therefore, we only use one-dimensional input and basic arithmetic operators to create a numerical optimization problem solvable by a first-order optimizer. This environment constructs a computational graph by sampling a number of times. At each time, it will either use previously computed variable sample new variable, and an operation will be sampled to combine them. The optimization task is, for a fixed number of steps, an optimizer needs to output æ that minimizes y.
We evaluate the following baseline methods. Basic Agent: a basic LLM agent that simply stores past information of (Ią-1, 3t-1) in context before choosing the next 24. OPRO Agent: basic LLM agent but we implement the state-of-the-art LLM optimizer OPRO [13], which updates the meta-prompt of the basic LLM agent. Torch Adam: the problem we construct is end-to-end differentiable. Therefore, we simply pass torch. Tensor (x) as input and use Adam optimizer to update. We
tune the learning rate slightly and found le-1 to work well. We compare two kinds of Trace-based
23

---

optimizers: Trace, where we allow OptoPrime to read in the entire computation graph before updating x, or Trace Masked, where we hide the computation graph.
We run 30 trials over different computation graphs and start all methods with the same initial x, y" We compute the absolute error, which is ly - y- I- On average, Trace is able match the best-in-class first-order gradient optimizer Adam [25]. It is not entirely surprising that all the other baselines are performing worse due to a lack of access to the computation graph. To our surprise, OPRO, by only accessing the history of input and output, as well as changing the meta-prompt, is able to eventually discover the correct solution. This confirms why there were early signs of success using LLMs for black-bor box optimization in a simple plug-and-play style. However, OPRO is not an efficient optimizer because it lacks access the Trace oracle. We show OPRO struggles even more when the computation graph gets more complex.
1.4
Traflic Control
We tested OptoPrime in a traffic control problem which is an instance of syper-parameter tuning. We used UXSim [26] to simulate traffic at a four-way intersection, where the tunable parameters are the duration of the green lights for each direction of traffic flow. The feedback is a scalar loss calculated by monitoring the flow of pseudo-random sequence of vehicles arriving at the intersection over period of 30 minutes. The loss computes an estimate of the delay experienced by all vehicles due to the intersection, as well as variability in this estimate for every link in the network; lower values are better. The goal of an optimizer is to identify values for all of the green light duration so as to minimize the loss using the fewest number of traffic simulations. If the green light duration for given traffic flow direction is set too low, then vehicles will queue up over time and experience delays, thereby lowering the score for the intersection. However, if the green light duration for a given direction is set too high, vehicles in other directions will queue up and experience delays, thereby lowering the score for the intersection. Hence an optimizer must find the right trade-off for temporally distributed and variable demands.
In Figure 5 we report the performance of SOTA heuristic from the traffic control literature. SCATS [27] (adapted to this toy setting) as well as two black-box optimization techniques: Gaussian Process Minimization (GP) [8] and Particle Swarm Optimization (PSO) [28]. All methods are initialized to evaluate the same starting parameter. GP and PSO further evaluate 5 random parameters; moreover, if they query a previously evaluated point, that query replaced by a randomly sampled parameter. GP constructs surrogate model to mimic the black-box traffic simulation function which maps from parameters to observed score. Then it minimizes a utility function (e.g. the lower confidence bound) using the surrogate model to pick the next parameter to evaluate. PSO on the other hand maintains particles in parameter space, each with a position and velocity. At each iteration of PSO, particles update their positions according to their previous positions and velocity, evaluate the function at the updated positions, and update the velocities of all particles using the observed values. Although GP and PSO are both black-box methods, GP can be thought to replace Trace oracle with instead a smooth differentiable surrogate function; whereas PSO is very different and maintains a candidate set of parameters (can be thought of as conceptually related to OptoPrime with memory).
GP appears be bad because even when converged, the exploration heuristic randomly samples parameters rather than pick the converged parameter. PSO appears bad because 10 iterations is insufficient for its convergence. Note that given enough number of iterations, black-box approaches will eventually perform well. Trace is quickly competitive with the SCATS heuristic, whereas OPRO not. Moreover, we find that memory is crucial for OptoPrime to perform well at this task. Finally, Trace consumes additional overhead compared to black-box methods; beyond the space and time complexity for running the traffic simulation, Trace additionally materializes the computation graph per iteration. Thus it can also be more expensive per LLM call compared to OPRO.
1.5 BigBench-Hard
Perhaps more surprisingly, there are many components that a workflow needs to learn Some of these components can be the prompt to generate output from an LLM, while other components can be code that needs to further process these outputs. In many workflows today, enabled by LangChain [29] and DSPy [22], only a small part of this workflow, predominantly, the input to an LLM API call, is optimized. These libraries optimize input to an LLM, and human engineers process that input
24

---

Iaport trace
Inport trace
class Predict (trcccttrageeMoouuee))
dataset losd_ dost_dataset (link, task)
def ...nit... (senn)
predict Predict()
seir porr-proptt_.. template trace. node
optinizer
Given "question", procuce the "answer', question:
trace.
in dataset:
10
trainablenTrue)
response for (eaparddex("qussttii question
11
ex['answer if feedback 'The answer correct'
12
def fael, question):
13
prerpt 5 -self trtte (cuestion)
12
clse: feedbock "The answer is incorrect'
14
response = self-call call. wwwllaallnnnnee Lin(user prompt) ansser extraex s_anserr (question, response)
13
15
Backward pass and update
16
return answer
optimizer.zen zero
17
optimizer zer. www.eeer..aaaaee backaord feedack))
18
Strace. bandl nttranaahee-Tru
zer mii.
def create_pronpt pronpt question):
forforullte
prompt with question return self. www.pprrteppppp.pp frr.aatee..orneeesssttto
(b) The optimizer class takes in any parameter re- gardless of whether is code or text. Although the actual optimization implementation can provide different treatments to many input types, the user interface stays consistent.
22
23
Strace. bundle(trainable-True
34
def extract_ansee (self, question, respense):
25
"xtrrtttt the answer out of liM response
26
answer response... split("answen ")E3). return answer
(a) We write workflow that prompts an LLM for a
question and extracts the answer.
Figure A.7: LLM-based Workflow Optimization Example.
and integrate it into other systems. Indeed, both libraries can enable robust and swift large-scale engineering efforts to build LLM--ased software. However, our goal is to develop self-adapting agents that modify their own behavior, should not ignore one of LLM's greatest strengths: code generation. Trace allows us to unify prompt optimization and code generation, which enables the creation of agents capable of fast learning.
In this example of an LLM-based workflow (Figure A.7), there are three parameters that are flagged as and extract_answer Note that two of them require the LLM to generate Python code, and one of them requires the LLM to modify text. Trace abstracted away the different data types and enabled direct update and optimization of them. Furthermore, a human engineer is often tasked with writing an error-free The output of an LLM can be highly stochastic and often change used to extract the response of an LLM has to be extremely robust and, therefore, arduous to create. Whenever major distribution shift happens in the LLM output, this code needs to be rewritten by human engineer, and it is hard for humans to anticipate all of LLM''s 's output patterns.
trainable for the optimizer: compt_template create_promp
time; the code that is
We set up the task of end-to-end workflow optimization. Unlike a typical LLM benchmark evaluation, where a lot of effort went into creating the perfect evaluate (answer, target) method so that all kinds of LLM outputs were post-processed. cleaned, and formatted to match the ground truth, choose a simple evaluation function (that extracts a segment or does exact string matching) and place the burden on the workflow itself to figure out how to create the right answer to satisfy the evaluation metric. We choose Big-Bench Hard [21] as our task because it has 23 subtasks and contains both language and algorithmic tasks.
We split each task dataset into training, validation, and test. For Trace and Trace-CoT, we use the first 15 examples for training, 5 examples for validation (picking the best learned workflow), and then evaluate the performance on test examples. We use template-based positive and negative feedback during training. The positive feedback is "The answer is correct! No need to change anything." The negative feedback is "The answer is wrong. We expect the output of your answer to be (ANSWER). Please modify the prompt and relevant parts of the program to help LLM produce the right answer." DSPy's prompt optimization method does not explicitly require a validation set, therefore, we just used all 20 examples for training. For both, we only optimize for I epoch. We either start with the vanilla boilerplate prompt template used by DSPy or we use the slightly sophisticated template used by DSPy's ' CoT module. Trace optimizes both DSPy's original design and outperforms their optimizer COPRO by 10% on algorithmic tasks.
Big-Bench Hard requires different answer outputs. Out of 23 tasks, 14 tasks require a multiple-choice answer with options provided in the question. of them require yes/no. task requires True/False, while 1 task requires valid/invalid. And the remaining tasks require answers that contain words or
25

---

Task Name
DSPy
+CoT
DSPy-PO
+CoT
Trace
+CoT
tracking shuffeed objects 7 objects
37.39
90.0
90.43
90.43
37.8
87.8
salient translation error detection
51.3
70.87
51.3
69.57
63.0
70.0
tracking shuffled objects 3 objects
39.13
94.35
97.39
93.91
38.7
96.5
geometric shapes
50.43
62.17
59.13
60.43
49.6
62.2
object counting
0.0
40.0
0.0
74.35
42.2
80.4
word sorting
0.0
0.0
0.0
0.43
84.3
74.3
logical deduction five objects
70.0
80.43
70.0
74.78
48.7
75.7
hyperbaton
74.78
86.52
74.78
88.26
78.3
91.7
sports understanding
0.0
0.0
0.0
0.0
79.6
45.7
logical deduction seven objects
68.7
64.78
68.7
64.78
45.7
69.6
multistep arithmetic
0.0
93.04
0.0
93.04
94.8
88.7
ruin names
84.35
87.83
84.35
87.83
87.8
90.0
causal judgement
7.78
70.66
74.25
70.66
70.1
54.5
logical deduction three objects
85.22
97.39
85.22
97.83
91.7
97.0
formal fallacies
1.74
81.3
62.17
81.3
73.5
67.8
smarks
86.08
87.34
86.08
87.97
81.6
87.3
boolean expressions
98.26
64,35
98.26
88.7
96.5
reasoning about colored objects
53.04
91.3
89.13
91.3
91.3
95.7
dyck languages
0.0
8.7
7.83
8.7
26.5
9.6
navigate
0.0
95.65
0.0
97.39
59.6
92.6
disambiguation qa temporal sequences web of lies
67.83
66.09
73.91
66.09
75.7
99.57
99.13
97.39
99.13
97.8
98.3
0.0
0.0
0.0
0.0
49.6
90.4
tracking shuffled 5 objects penguins table movie recommendation date understanding
37.83
96.09
37.83
96.09
58.3
88.7
69.84
92.86
97.62
92.86
81.7
91.3
83.48
76.09
83.48
76.09
81.3
75.7
69.13
85.65
69.13
85.65
70.4
85.7
Table A.2: Big Bench-Hare Task Result, O-shot performance. Some 0.0 here shown is because DSPy cannot find the clean/stripped output that matches what the automatic evaluation method expects, With additional human engineering, these numbers improve.
numbers. Even though DSPy's meta-promen optimization is trained each task individually, the output of LLM to the evaluation method is still not post-prose petcepocessdd, resulting in low performances of these tasks. However, Trace can optimize code and LLM prompt jointly to successfully deliver the response expected by an automatic evaluation method.
1.6 LLFBench Sora-World
We test the ability of Trace to optimize long-horizon workflows with complex dependencies. We experiment with using Trace to train controller (python code) for simulated Sawyer robot ma- nipulator. We use the Meta-World environment of LLF-Bench [23] as the simulator and consider three tasks reach, pick-place and push. LLF-Bench simulated benchmark with gym interface for testing an agent's ability to learn from language feedback. In these LLF-Bench Meta-World tasks, the observation is dictionary where each field denotes feature of the state and has vector value (e.g., the end-effector position, the goal position, the gripper status, etc.). The keys of the observation dictionary can differ for each task. The action space is 4-dimensional,, which controls the relative position of the end-effector and the state of the gripper. In each time step, the LLF-Bench Meta-War World simulator returns the observation dictionary and natural language feedback guide the robot (we use the 'a" mode of LLF-Bench with which the language feedback would contain information about the current performance, explanation of past successes and failures, and suggestions for the next step). An episode ends if the robot successfully solve the problem or because of time-out, For each task, LLF-Bench also provides task instruction explaining that the task is about controlling a Sawyer robot arm and the meaning of the action space (see [23]). We use that as the context of the OPTO problem. We consider episodic setting. For each experiment (a random seed), we randomly sample an initial configuration. Then for each iteration of optimization, reset the simulator that sampled
26

---

Success
reach
Trac hoteer
Masked
Berctions
Rerations
Reraboon
(a) Reach (Train)
(b) Pick Pick-place (Train).
(c) Push (Train).
pick place
Succese
GFRO
push
Nickic Tieer Macket
Iteratinns
Rerations
Iterations
(d) Reach (Test)
(e) Pick-place (Test)
(f) Push (Test)
Figure A.8: Learning the feedback control policy (code) for simulated Sawyer manipulator in LLF-Bench Metawrold. In each iteration (X-axis), one episode of rollout performed and then the policy updated. Mean and standard error of success rate over 10 seeds are shown.
initial configuration and run the robot policy for 10 steps until the episode termination due to success. We compute the sum of rewards and gives the output feedback f in texts in the format of "Success: <true/false> Return: <score>". Note that the initial condition for all iterations within an experiment is the same so that the optimization problem deterministic. policy's performance, for each experiment, we additionally run the learned policy starting from 10 held-out initial conditions, different from the fixed training initial condition. For each training algorithm discussed, we run it with 30 iterations, where each iteration consists of one episode rollout and one update.
evaluate the learned
To optimize the controller with Trace, we declare the control code as the parameter using the bundle decorator with trainable set True; the initial control code simply outputs zero vector [o, O, O, O). We decorate also the reset and the step function of the gym environment, so that the entire rollout of an episode can be traced end-to-end. In our implementation, a prototypical rollout would create graph with around 30 operations where the controller code parameter is used multiple times. This graph structure is similar to that of running recurrent neural network. For Trace, we experiment with OptoPrime with and without a memory of size 10. In addition to Trace, implement the stateoofthhean LLM optimizer OPRO [13] as a baseline. Compared with Trace, OPRO does not use the execution trace information but rely just memory of parameter and feedback pairs 12 To make the comparison with Trace fair, we append the feedback observation from LLF-Bench given at each time step to the final feedback received by OPRO; the other hand, Trace uses the simple final feedback of success and return and has to read the per-step feedback from the execution trace. To run OPRO in the OPTO setting, implementation only proposes single candidate in each iteration, which is then evaluated and provided with the output feedback. Since in [13] OPRO generates about 10 samples per iteration, so iteration in [13] is roughly equivalent to 10 iterations here.
The experimental results are summarized Fig. A.8, where we show the success rates at both the training initial condition well as the held-out testing initial conditions over 10 seeds. OptoPrime is clearly the top oop-eerforming optimizer, especially the version with memory. OPRO is able to solve Reach at the start but its performance degraded over iterations (this instability was observed in [13]) and gets similar performance as OptoPrime (without memory) in Push. To validate that the performance of OptoPrime is indeed due to using the execution trace, include an ablation where we mask out information in #Inputs, #Others, #Code, #Definition in the LLM context (see Fig. Ab),),
We set the problem horizon be 10 steps, find the expert policies implemented in LLF-Bench solve these problems within 10 steps.
12ehe original version of OPRO uses parameter-score pairs. Since we're interested in the more general setup of OPTO, extend it wwameterrffedbade pairs.
27

---

which lead to significant degrade in performance and stability. This ablation shows that additionally using the execution trace provides more informed search direction compared with just using just the output feedback, which agrees with our hypothesis.
This experiment features the most complex graph structures, and using Trace for optimization here is similar to pack-propagation over time. The experimental results here are quite impressive, showing that Trace is able to learn complex control logic in dozens of interactions, not only working on the training initial conditions but also on the held-out testing ones too. Nonetheless, we want to point out some limitations in the current experimental results. We find that the success rate of the learned policy varies largely across random seeds. Except for Reach (the simplest task), in seed, often either finds a policy close to 1.0 success rate 0.0 success rate. Therefore, the plots can roughly be interpreted as how long it takes to find a working policy. In addition, in these experiments, we find that providing task-related context necessary. We find the context needs be informative enough for humans to understand the problem 33 otherwise, the optimization can be solved efficiently with the time scale considered here. Nonetheless, this requirement reasonable, as there no free lunch.
The original instructions in the v2 environments of LLF--Bench does not contain task specific background, but only the task name. We find this task name alone, except for Reach, understand how to
vague even for humans observation dictionary is related to the problem, we added additionally one line to explain the task. Pick-place: "The goal of the task is to pick up puck and put it to a goal position.". Push: "The goal of the task is to push puck to goal position.". Without this extra information, none of the LLM--bsed optimizers works in the experiments.
28

---

Examples of the Optimized Parameters in the Experiments
Policy (trace Madule): def init(self, width, height): pass
def c_l_l__ (self. map):
return (enesseele
def select_coordinateetell,,, nap):
plan self. reason
10
self output (nap, plan)
12
13
return output
14
15
strace. www.buuddeeeeee.e. (trainable=True) def act(f map, plan):
16
17
map. select target coordinate Battleship gane. nap, denotes nisses, X denotes successes, and denotes unknown positions. ..*
19
20
return
21
bundle (trainable= def reason nap) +> str:
23
25
Given analyze the board in Battleship gane. In denotes misses, X denotes successes,
positions.
26
27
return [0,
29 def tra
30
policy Policy()
31
optinizer Funciionogeiiizze///Meaayyyy oaro parameters
32
board- BattleshipBoard0
33
34
feedback
36
board reset) node (wwardb init observation
37
38
6
39
epochs:
40
try:
41
output = belicyyseeet.ccorrnstt((oo
42
not traced
43
obs, reward. terninal, feedback = user_fb_for____ por_ppaciigg.shttbbrrr output data)
except TraceExecutionErrer
retry
45
output = e. exception_node
16
feedback output. reward, terminal False
47
48
if terminal:
50
buard reset_board()
(board. get_shots()) init
52
53
Update
wwtinizer.zeroffeeeeeee
55
optinizer. backward (output,, feedback) optinizer. (verbose=1rue))
57
58
Figure A.9: Battleship Agent and optimization code.
29

---

Iteration Success False
def reason nар) -> str:
...
Given nap, analyze the baard in gane. On nap, denotes misses, X denates successes,
and denotes unknown positions.
theck for first unknown position to try as patential nove for row_index,
in enunerate (nap):
for col_ index, val in enumerate(row):
if return [row_index, col_index]
10
11
Default to [6, 0J no unknown positions found return [e, 6]
12
13
Iteration
15
def reason(self, nар) +> str:
...
17 Given nap, analyze the board in gane. On denotes misses, X denates successes,
and denates unknown pasitions.
18
19
Improved strategic approach by checking surrounded by more which night indicate undetected
20
nax max_surrounding -1
best_spot [8,
22
for row_index, row in enumerate (map): for col_index, val enumerate(row): if val '.'
23
25
surrounding 9
in [-1, for dj in [-1, 6, 1]:
27
28
ni, nj row_index di, col_index dj
29
if 0 <= ni len (nap) and 9 <= nj < (enpna[[)) and nap[mi][ns] ==
surrounding if surrounding > nax_surrounding: nex_surrounding surrounding best_spot [row_index, col_index]
31
22
33
34
return spst_spot
35
36 ## Iteration 8 Success True
def reason пар) str:
38
naр analyze board gane. nap, misses, successes, and denates unknown positions.
40
...
41
nax_surrounding -1
42
best_spot [e, 0]
for row_index, in for cal col_index index, val in enumerste(row):
(nap):
44
46
surrounding =
17
for di in [-1, 8, 1]:
48
for dj [-1, 8, 1J:
49
ni, nj = row_ di, col col_ index + dj (nap) if map[ni][nj]] == !.::
nj (en(maa())):
51
surrounding
53
elif map[ni][ng] == 'x's
54
surrounding 1.5 increasing enphasis on proxinity
successful hits
55
surrounding > max_surrounding:
nox_surrounding surrounding best_spot [row_index, col_index]
57
58
return spot_spot
59
Figure A.10: Learned Battleship Code (reason). Functions with the same learned during different iterations or trials.
30

---

Iteration (Initialization); Success False
#e Iteration 1 Success : False
act(self, map, plan):
Given a nap, select a target coordinate a game. On map, denotes nisses, denotes successes,
denotes
positions.
Example inplementation, more sophisticated logic may required based an intended ganeplay mechanics
10
return [8, 1]
11
12 au Iteration 2 Success : True
@st(s.lf. map, plan):
14
15
target coordinate game. map, nisses, successes, and denotes unknown positions.
nap,
16
Use informatian from plan, which is output from code,, checking strategic position, not just first available
best_score
19
best_move = [0, 6, 8] # initiolly nave is considered for row_index, row in enunerate (nap):
20
21
for col_index, val in enumerate(row):
22
if val '.': surrruninngn for di in [-1, 8, 11: for [-1,
consider all unknown spots
à will hold number of 7.'s around the
24
26
ni, nj row_index di, col index + dj
27
(nap) and nj (map 1f nap[ni][nj] == '.':
28
29
hapes_ surrrunddnnn more '." araund, higher the chance of hit score > best_score:
31
32
best_sccre
33
best_nove = [row_index, col_index]
34
nap [ap[pan[e]]][laee
',' best -1:
35
return plan 1f по better found, and original place still unknown
96
return best best_move best_score -I else plan return the best move found, or stick with the plan if still valid
37
Iteration
39
def act(felf, map, plan):
Given 40
...
select target coordinate successes, and denotes unknown positians.
On denotes misses. X denotes
43
best_ -1
best_nove [0.
45
for row_index, row in enumerate (nap):
46
for col col_inge index, val (rumerate(row)>
if val '.' and [row_index, col_index] plan: hapes_surrounding = for di in [-1, for dj in [-1, 8, 1]:
48
50
ni, nj row_index di, col_index dj
52
If 0 <= ni (nap) and 8 <= nj (maa if nap[ni][ni] ".':
33
hopes_surrounddng
favoring positians with
unknowns surrounding
elif map[ni][nd] 'X':
56
hopes_surrounding ** increased incentive for noves near
spots
57
score = hopes_ surps_snrronniing
38
scare > best_score:
60
best_move [row_index, col_index]
best_ scerescore>
62
return best_nove
return plan
64
Figure A.II: Learned Battleship Code (act). Functions with the name learned during different
iterations or trials.
31

---

@trace_class
Dearriition=(Paaaatterrrr This is the Prompt class PLeciic(L1MC)llaabeee)) def (sclf) super().
self. denos [J
self. pttppt. dedent("
fields "question" ", produce fields "answer".
11
12
Follow the following fornat.
14
Question:
13
Answer:
16
17
18
Question: ()
19
Answer:
20
""*)
21
22
prampt_template trace.node prompt_templace trainable=True,
23
Template LLM...")
24
25
8trace bundle (traaiabeeeneee
def extract_answer (self, prempt_tenplate, question, respanse):
27
Need read answer
respanse, can contain additional thought, delibration
29
Use code to process response and find where the answer is. cawl_allnn""e_aeeaaa.allaelann the answer fron this text: " respense) again to refine the answer if necessary.
30
Can
22
Args:
33
prampt was question: Question has text describing the question but also "Gptions" response: LLM returned string response Process and return the evalustor wants to see.
prompt_template
query
get respanse
34
35
36
in the exact format that the
mindful the type of you need produce. It can be (A)/(B), number like 8, or a string, or Yes/No.
38
39
40
answer response... split ("lnse"Answer:: ().1) strip()
41
return answer
42
43
Strace. bundle (trainable==rue) def create_prompt (self, prompt_template, question):
45
46
function takes question and then to the prompt LLM answer,
47
Args:
48
prompt_ template: sone guidance/hinn Saltanccehintt//ee suggss//ugessions for LLM question: the question for the LLH
50
prompt_template. format (questionn
52
53
def (sll,, question):
54
55
question: text
56
57
We read in a question and produces response
59
psep_t self.create.re (ramptt prompt_template question)
60
response self. ( self ewt.eeettatt_aasee (self. wwwee..ppooppppeeeeeeeee tromat_temppate. question, respense) return answer
62
Figure A.12: Starting Code for BigBench. write similar style to DSPy's Predict module.
32

---

@trace_class
description ((Praattoone(rarrrneeeeeeod This Promot class PredictCol (LLHCallable):
def ...init..
super () 0i...iit......
self. demos []
self. prampi_template dedent("* Given the fields "question", produce the fields 'answer's
10
12
Follow the fallawing farnat.
13
14
Question: question
15
Reasoning: Let's think step by step in order to produce the answer. Me
Answer:
17
18
19
Question: ()
20
""*) )
22
self. prompt_template trace. (radee. (self.......... (ttss//llllss...eieeeeeeeeeeessp prompt_ trainable=True.
Template to the LLM...")
25
strace. bundle (trainable=
26
def extract_answer (self, prompt_template, question, respanse):
una
28
Need to read in the response, which can contain additional thought, delibration and answer
29
Use cade to process the respense and find where the answer is
30
colf.call_lln("Rettt the refine the answer if necessary.
respense) again
31
33
response: LLM returned string response
answer
evalustar wants to see.
35
mindful the type answer you need produce.
36
It can be (A)/(B), number like 8, string, or Yes/No. question: Question has text describing the question but also "Options"
37
39
answer response. response.splitt ("Answe:: strpp)
40
return answer
41
42
strace. bundle (trainable=True)
43
def create_prompt((ee (self. prompt_template. question):
44
question and then The prompt should instruct the LLN to reason, think. Args:
function
prompt LLM answer.
46
47
48
prompt_template: some guidance/ aaddance/hintes /tntt/hit////// /ttnts/suuggestionn for LLH question: the question for the LLM answer
19
51
return prompt_template format (question)
53
def forward question):
55
question: text
56
We read question and produces
58
39
user_prompt self. create_prompt prompt_template, question) response self.call sall_1lm ppomppp)))p)) answer self, extract_answer (sel promp.. templat,, question, response) return answer
60
62
63
Figure A.13: Starting Code for BigBench. We write it similar style to DSPy's Predict CoT (0-shot Chain-of- Thoughh module.
33

---

swaps Iteration initialization
def create_prompt(sell) prompt_template, question):
.*e
The function takes in question and then add the prompt for LLM to answer,
Args: prompt_tenplate:
for LLN
question: the question for the LLN to answer
return
11 #v Iteration >
12 def create_ prompt_templat question):
.**
14
The function takes in question and then add to the prompt for LLN to answer. prompt should
the carefully
occurring step step-
16
Args:
17
templatte sone huddancehiints/ssgeetion for LLM question: the question for the LLN to answer
18
20
Process this carefully: step-by-step." prompt_template prempt_cemplate. Format (question)
22
Figure A.14: Learned Predict module for BigBench. Functions with the same name learned during different iterations trials.
34

---

response is after 'Answer:' before any Iteration initiillizttion
def extract_answer (self, prompt_template. question, response):
Need to read in the response, which can contain additional thought, delibration and an answer.
Use code process the response and find where the answer is. Can use self.cull. cat. the answer from this text: response) again to refine the if necessary.
Args:
tompl_template: The pronpt that was used query LLM to get the response
10
question: Question has text describing the question but also "Options" response: LLM returned string response
11
12
Pracess it and return the answer in the exact format that the evaluator
13
Be nindful of the type of answer need to produce. can (A)/(B),
8, or string, , Yes/No.
15
16
answer response. (etsetstsssseer
return
18
19 Iteration
20 def extract_answer (self, response):
Need to read in the response, which contain additional thought, deliberation and 21
.*.
22
answer
Use code pracess the response and find where the is. Can use self.call_1n 1aalllll1""""tttti""" the answer from this text: response) again to refine if necessary.
24
the
25
Args:
26
response: LLM returned string response
27
Process it and return the answer in the exact fornat that the evaluator need to produce.
wants see.
28
Be mindful of the type of It can be (H), number like 8, or string, or Yes/No. has
29
question:
describing the question
"Options"
31
32
answer
33
segnents
34
for segment segnents:
if 'Answer:' segnent:
36
answer = segment segnent.split( ()rrst
37
refined_answer
half.caal_llnnReettte the refined answer fron this text: answer) return refined_answer
38
39
40 def extract_.nswerrsellf prompt_template, question, response);
41
Processes the LLM response and extracts the final
the required format.
Assuning 43
part
further commentary
15
extracted_pert. response. split(" Answer: ")[lJ.ppii((" ()(0).strip))
46
Find the section of the answer return it directly result = re. se.sschh(""(((A (")((((()') extracted_part)
47
result:
49
return result. (roup()
50
51
return 'No valid answer found"
52
def extract_answer (self, prompt_template, question, response):
54
Processes LLM response, extracting and formatting the final answer
56 Uses code to neticulously parse the response to locate the answer section.
37
Optionally refines answer by querying
again necessary.
58
59
Args:
response: string from expected format contains 'Answer:' followed the question: Description of the question being addressed, may include 'Options"
61
...
63
answer response... split( ('Attt(''swe::: (111. "D00]. strip(). lower (). split(', ()()] strip()
return answer)
65
Figure A.15: Learned Predict module for BigBench. Functions with the learned during different
iterations or trials.
35

---

#F Iteration initiallizatinn
prompt_temmp tepppptteelleeeeeeee Given the fields "question', produce the fields "answer'.
Follow the following format.
Question:
10 Answer:
12
13 Question: : ()
14 Answer:
15
16
17 AM Iteration >
18 prromp__templatee
..*
19 Given the fields "question', produce the fields 'answer
20
22
Follow the following format.
24
Question:
26 Answer: Please respond precisely in the fornat required.
27
28
29 Question: (
31
Figure A.16: Learned BigBench Prompt Template.
36

---

gripper_open Iteration (initialization); Success:
False
def contreller (obs):
feedback controller that computes action based on the observation.
Args:
obs: (dict) The observation from the environment. Each key is string (indicating type of observation) and the value is a list of floats. Output:
10
action: (list or ad.array) 4-dimensienal vector. ...
12 return [e, 6, 8, 8]
13
14 #e Iteration Success: False
15 def (ontr
16
17
feesback controller that computes the action based on the observation.
19
Args:
20
key string (indicating type of observation) and the value is a list of floats.
obs: (dict) observation from the environment.
21
Output:
action: (list nd. array) 4-dinensienal vector.
23
hand_pos obs hand_pos'1
25
goal_pos = obs goal_pss_
26
action [goal_pos hand_pos[0], goal_pos hand_pos[1]. goal_pos
hand_pos[2],
27
Adjusting the last element the action based on whether the hand is close enough to trigger grasping if abs (goal_pposst hand_pos[ol) 0.02 and abs (goal_poss [1] hand_pos(1]] 8.02 and (roall_ooo [2]]_ Assuming [3] triggers closing the gripper
28
29
action[3] return action
30
31
32 AM Interation 5; Success: False
def controller(abs))
34
feedback controller computes action based on observation.
36
37
Args:
38
abs: (dict) The observation from the environment.
39
Output:
action: (list nd. array) 4-dinensional vector.
41
42
hand_pos obs hand_pos'1
43
puck_pos obs pusk'pucc_pos'
44
gosl_pos obs obsl'goal_pp goal_pos'] obs(' "rrrgppr_dditance_.ppee '][8] 0.5 else
46
Adjusting strategy close gripper when in proxinity the puck
48
if gripper_open and ((ans [os[pp puck_pos (0]) 0.05) and (abs (hand_poss Puck_pos[1]] 0.85) (sbs puck_pos 0.05)): close the gripper
19
action [e, 8, 8, elif not gripper_apen:
30
direction_to_gaal [e.a9 hand_pos[0], 0.35 hand_pos[1], 0.12 hand_pos[2]] direction_to_goalle diraction_to_goal #] suggested pose puck grasped
52
action [direction_to_ppallll
53
else:
puck not carrying di direction_to_piie [puck_pos[[]] hand_pos[0], puck_pos[1] hand_pos[1], puck_pos[2] [2] hand_pos [2]]
Nove
55
56
action reiriiiio_cto_puccee tirrcttoo_to_nucc(t1), direction_to_puck(2]
57
Figure A.17: Learned Code LLFBench Meta- World Pick-Place (Part 1). Functions with the same name are learned during different iterations or trials.
37

---

Iteration Success: False
def controller(obs):
feedback controller that computes the action based on the abservation.
Args:
obs: (dict) The observation from the environment.
output:
11
action: (list or nd. nd.array) A 4-dinensional vector.
12
13
hand_pos = obs hand_pas")
14
puck_pos obs puck_pos')
15
gosl_pos abs goal_pos"
16
gripper_open
if obs obs['gripper gripper.distace.apart I[ol > 0.5 else
17
18
normalize Lambda x: [1 (nxx keymabs)), 1) for i In
19
20
close the gripper when close to the puck and the gripper is open
21
gripper_opes and and (abandd___po((2 puck_pos(2]) 0.85)):
puck_pos[e]] < 0.85) and (abs (hand__aaoox
puck_pos[1]]
22
return [0, 8, 0, 1] grasped. elif not gripper_open and ((abs (handd_pss_] [uck_pus(@]) 8.1) and (abmddan___os( puck_pos[1]) 6.1) and [2and_ puck_pas[2]] 8.1)):
towards goal position
normalized direction
24
25
direction_to_godt goal_pos [2] hand_pos[2]] (dirliiioiiit
[goal_pos[{]] hand_pos[0], goal_pos[1] hand_pos[1],
[o]
27
when the gripper is open and not close enough to the puck, move towards the puck
29
direction_te_pue [puck_pos[0]] hand_pos[@], puck_pos[1]] hand_pes[1], puck_pos[2] hand_pos return nermalize(ddreecioon.too
30
[0]
31
Iteration 13; Success:
33
def controller
...
35
feesback controller that computes the action based on the abservation.
36
37
Args:
38
obs: (dict) The observation from the environnent.
output:
40
action: (list or nd. ndrarray) 4-dinensional vector.
42
hand_pos = obs hand_pas")
43
posk_pos obs puck_pos')
44
gosl_pos obs "'al''ooa_psss
45
gripper_open
if obs poul''riiper_distie prriperrdiittamee.pprrrt 1[6] 0.5 else
47
normalize = Lambda x: [i max (naxs key=abs)). 1) for i in x] puck
48
Clase gripper if gripper_open and clase [usk_pos(1]) 8.85) and (abnd_add__ostll puckkpooooo_ (.85)):
gripper open
49
pocckpos((1]) 9.85) and (1bsdha
30
action [e. 8, 8, elif not gripper_apen and (hand______ puck_pos 8.1) and (abs [1] (2)nd puck_pos(2]) 0.1)):
Close the gripper
51
puck_pos[1]] 0.1)
52
U When the puck is grasped, adjust direction towards the geal with inproved precision ensure successful pickup
53
di docccilontto_goll normalize ([goal_poss))) hand_pos[o], goal_pos[1] hand_pos[1], gosl_pos hand_pos action direction_to_goss
[I] Keep the gripper closed
55
else: towards puck
carrying Eirrccioo_to_puee normalize ([puck_pos[0] hand_pos[0], puck_pos[1] hand_pos[1], puck_pos hand_pos[[]]]
57
58
action direction, to_k
[$] Open the gripper prepare for grasping
59
return action
Figure A.18: Learned Code for I.I. ..BeBhcch Meta-Worke Pick-Place (Part Functions with the same name learned during different iterations or trials.
38

---

"reasening": "The task involves controlling a robotic arn to perform a pick-place operation by interpreting observations and generating appropriate contral actions (4D vectors) to nove the arn towards the gaal. The objective is to adjust the contraller's 's code (__code)) based feedback #0thers suggests whether the novenents nade by the arn are leading it closer to or goal, along
each
of the
The feedback within
coordinates to [-0.07 0.68 8.12 o. }) The cantroller's code currently returns static action [0,
for next (e.8..
8, 9, 81, which does not adjust based on observations and hence does not efficiently the robotic
towards the goal. Given the repetitive ferdback suggesting go to
[-0.07 0.68 8.12 o. ], it appears that the controller needs to dynami conpute (hand_pos. puck_pos) and
based the difference between the geal_pes, along with possibly nanipulating the gripper state for pick and place operation.".
"answer";
"suggestion": (...)
9N Iteration
"reasoning": "The instruction involves creating controller for Sawyer robot arn salve a pick-and -lacce task by praviding sequential control actions based on observations of the environment despite the actions taken, the success criteria have not been net with the latest attempt achieving return
state of the robot. The feedback indicates that
provided for c_code defines controller function using the observed state to compute control actions aimed at grasping a puck
0.9574427394258822
noving it to goal position while adjusting the gripper its distance fron the puck. Feedback throughout the execution indicates that the controller adapts its strategy based on proxinity to the puck and whether it has been picked up. Feedback also suggests central to the task success. with corrections being nade to close the gripper when necessary.
the gripper opening
closing actions
controller's legic for deternining movements adjusting the gripper state appears reasonable, but the fredback implies that adjustments night required to ensure task completion. the conditions under which the gripper is closed adjusting the thresholds for puck pickup and delivery to the gosl.". "answer"; a*
puck or goal
could include fine-tuning
13
"suggestion": (
15 * Iteration 10
16
17
"reasening": "The task involves programming robot to pick puck and place it at a goal pasition by providing sequential 4-dimensional action vectors. The action represent
(open close) based robot's 's and puck's 's current pasitions. The feedback indicates that despite optinizing the controller logic it's not grasped, and head towards the goal once the puck is grasped, the actions have not resulted in task completion. Specifically, repeated suggestions to close the gripper (indicating remsined where move next suggest that the action selection logic night not be effectively responding
positions gripper
gripper when near the puck, move towards
puck
when it should have been closed) and guidance
properly engaging the right times. The latest code includes condition to normalize the direction of novement
night
gripper
for gripper pasitian, there seens to misunderstanding oversight in adjusting the robot's behavior based on the scenario successfully complete the pick-place task.". "answer";
19
"suggestion": (...)
21
Figure A.19; Example Responses of LLM for LLFBench Meta-World Pick-Place (Part I); Content in suggestion omitted for space (represented ...); please the code examples.
39

---

Iteration 12
"reasening":
task, controller function the robot's actions based on the abserved environment to achieve pick-and function correctly position, the goal position, and the gripper state. However, the feedback after each step in the decunentation indicates that although the robot making progress towards the goal, there is issue with the gripper action state not being switched to "clese" (1) at the crucial nament when the robat is close enough to the puck, Most fails
manipulates Sawyer -nd--lacce task. puck 's position,
actions
the
feedback comments suggest
right direction grasp the puck due to the gripper not closing. This inefficiency appears to arise from closing the gripper and possibly the normalized direction
is noving
the condi towards the goal after the puck has been grasped. The condition for closing the gripper (when close enough to the puck) seens to be net, but the action selected afterwards does nat effectively lead puck nanipulation optinized nanner, as indicated by the "Feedback" nessages suggesting novements that imply needing adjustments action selection.", "answer";
geal achievement in the mast
"suggestion"; (...)
Iteration 13 (Success)
LLM response:
11
"reasening": "The instruction requests inprovements based on feedback far Sawyer robot arn's controller function, aiming to accomplish pick-and-place task. The controller function accepts observations about the robot and the envirenment and computes an action to nove the robot accordingly. \nAnThe feedback provided indicates the task was successfully completed with suggestion that the final command should have been "[0.04, 0.88, 8.24, 1]', which probably indicates the precise position and gripper (closed) needed fine-tuning actions especially towards the end of the task to ensure the robot 's gripper is appropriately positioned and the gripper state is correct for completing the task. Since the feedback for the current code pasitively netes 'Success: True" with high return value, nо nodifications are recommended far "__code 1, An\nPreviaus feedback instances with lower iterative improvements to the controller logic, specifically related to grip control novenent precision. inputs accurately position the gripper and handle the puck. The current code evidently manages these aspects well by normalizing directions and managing the gripper 's open-close state effectively under varying circunstances (prexinity to puck or goal and whether the puck is already grasped). These adjustments have likely led to the successful "answer": "No changes are recommended as the feedback indicates successful task completion and suggests controller's current logic adequately requirements."
goal.
feedback suggests
need for
and recomnendations for change highlight the
sin
adjust
contreller's response
the described problem.",
12
the task's
13
"sugsestion": {
14
Figure A.20: Example Responses of LLMs for LLFBench Meta- Meta-World Pick-Place (Part 2); Content suggestion omitted space (represented as ...); please see the code examples.
40

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions (Trace, OPTO, OptoPrime) are clearly stated in the introduc- tion and detailed in Sections 2,3,4, and claimed improvements validated in Section 5. Meanwhile the aspirational goal of self-improving interactive agents is elaborated separately in the concluding Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
41

---

Answer: [Yes]
Justification: There are theorems about the computational complexity Trace and an information theoretic lower bound propagated feedback. Both theorems proved in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setups are outlined in Section 5 and detailed in Appendix I. Supplementary material additionally provides the exact code that was run produce all results, and only requires user to supply an OpenAl API key,
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
42

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All of the code to implement Trace and OptoPrime provided in the supplementary material. All of the experiments described in the paper are reproducible with the supplied code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: / / /nips. httpssccccpllll/p htpppbll/gguu wew.ddeeeooeeeeeeeeeeeeee iitomisssions ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setups outlined in Section 5 and detailed in Appendix I. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
All experiments include standard error bars from 2 10 replications.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
43

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification; All the experiments use the compute resources (LLM API for OptoPrime and machine to run Trace), which are listed at the start of Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips cctppc/c///hhiiicuuuiueeeeeeee
Answer: [Yes]
Justification: The contributions of this paper (a framework analogous to Py PoTch to general- ize the backpropagation algorithm) not have direct safety or security implications.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Sections 6 and 7 we discuss both the aspirational goals and their broader
impacts.
Guidelines:
44

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There are no models, data or APIs released along with the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets used the experiments (e.g. BigBench- are properly attributed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
45

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: There is code implementing Trace and OptoPrime that released along with the paper. The code repository contains licenses and several tutorial notebooks documenting each functionality. There datasets models in the release, hence datasheets and modelsheets are not applicable for this release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There crowdsourcing human subject studies conducted in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
46

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "JI042KgHNS", "reviewer": "Reviewer_YuBN", "summary": "The paper introduces Trace, a novel optimization framework that instances the concept of Optimization with Trace Oracle (OPTO). In Trace, the computational workflows is treated as dynamic graphs and rich information, including intermediate results, processing details and computational graph, are used as feedback for optimization instead of traditional gradients. The framework includes a general-purpose optimizer, OptPrime, to solve the OPTO problem. In the experiment section, Trace is shown to be comparable to the first-order gradient optimizer on a numerical optimization task, and outperform baseline LLM-based methods across a wide range of tasks including , hyper-parameter tuning, robot controller design, etc. Additionally , Trace offers a Python interface that can integrate seamlessly with PyTorch.", "questions": "1. Could Trace be adapted for use with other programming languages, and what would such an adaptation entail in terms of architectural changes?", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.I think using execution traces instead of gradients for optimization is quite innovative. This allows for the optimization of workflows that are non-differentiable or have dynamic computation graphs. \n\n2.This framework can be applied to a wide range of tasks, including robotics and AI systems.\n\n3.Trace has shown superior performance compared to other LLM-based optimizers and has demonstrated results comparable to those of traditional gradient-based optimization methods.\n\n4.The Python interface for Trace simplifies integration with existing codebases.", "weaknesses": "As someone outside of this field, I find this paper to be quite impressive.  I did not identify any specific weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and question. \n\nYes, in principle, Trace can indeed be adapted for use with other programming languages. The core design of Trace is based on the primitives node and @bundle, which define the nodes and operators, respectively, for the directed acyclic graph (DAG) abstraction of the traced computational process. Once the DAG is created in any programming environment, the algorithms used in Trace can be applied. \n\nIn our current implementation, we overloaded Python’s magic methods to seamlessly integrate with existing Python code. This approach may not be feasible in other programming languages due to the limitations of operator overloading, which could result in a less clean interface. Nonetheless, by building a set of operators using the idea of @bundle, we can create DAGs to abstract the computational process in other languages. One example demonstrating the feasibility of such an adaptation is the C++ versions of AutoDiff libraries like PyTorch (which are also based on DAGs).\n\nTherefore, we believe that the DAG-based design that Trace employs can be effectively adapted to other programming languages. However, we acknowledge that developing Trace libraries for other languages can require non-trivial engineering. We hope that the impressive results demonstrated in our paper will inspire future development to adapt Trace to a broader range of programming environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8umFIiP34Z", "reviewer": "Reviewer_aQXt", "summary": "This paper proposes an end-to-end optimization framework, Trace, for the automatic design and updating of artificial intelligence systems. Trace is based on Optimization with Trace Oracle (OPTO), treating the computational workflow of AI systems as a graph of neural networks, which can be updated via backpropagation. Additionally, this paper introduces OptoPrime, a general optimizer based on large language models (LLM), as a specific implementation of Trace. In the experimental section, the paper compares Trace with the state-of-the-art LLM optimizer OPRO across various tasks, including numerical optimization, traffic control, and robotic control. The results demonstrate the superior performance of Trace. The primary contribution of this work lies in modeling computational workflows as OPTO problems and designing Trace + OptoPrime to address these issues.", "questions": "1.Difference Between Graph Optimization and Individual Node Optimization: Graph optimization considers the entire computational workflow as an interconnected system, optimizing parameters in a holistic manner, which can lead to more coordinated and efficient results. In contrast, optimizing individual nodes treats each component in isolation, potentially missing out on interactions between nodes. However, the paper lacks experimental evidence to support the effectiveness of Trace in performing graph optimization over individual node optimization.\n2.Discrepancy Between Ablation Studies in Figure 5 and Figure 6: The conclusions drawn from the ablation studies in Figure 5 and Figure 6 differ, raising questions about consistency. \n3.Length of Trace's Prompts Compared to OPRO and Token Efficiency: The prompts used by Trace are longer than those used by OPRO. It is important to quantify this difference in length to assess its impact on token efficiency.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n1.Clarity of Writing: The structure of the article is clear, the language is fluent, and it is easy to understand.\n2.Novelty: This paper introduces a novel end-to-end optimization framework, Trace, which views the computational process as a graph and utilizes execution trace information to optimize parameters. Compared to traditional black-box optimization methods, this approach offers higher efficiency and greater interpretability.", "weaknesses": "Weaknesses:\n1.Limited Scalability: The OptoPrime optimizer mentioned in the paper has some scalability limitations, such as difficulty in handling parameters or nodes that cannot be represented textually, and challenges in dealing with computational workflows that contain a large number of nodes. \n2.The graph for Trace requires manual design, lacking automated methods.\n3.Limited Experimental Improvement: The experimental section shows limited improvement, with the differences between Trace and OPRO not being particularly significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions.\n\n\n# Limited Scalability \n\nWe agree that scalability is a limitation of OptoPrime, as discussed in Section 6 (Limitations) and Section 7 (Conclusion) where we note the current focus on textualizable problems. However, we wish to clarify that this limitation pertains specifically to OptoPrime and not to Trace itself. Trace is designed to scale to large graphs with minimal overhead (see line 242) and can handle non-textual nodes.\nOptoPrime's limitation arises from converting the propagated Trace graph into a single query that fits in the context window for a current-gen LLM. As its name suggests, OptoPrime is a first step in solving OPTO problems. We are inspired by the historical development of back-propagation, starting from optimizing networks with 10s of neurons (in the Rumelhart et al paper in 1986) to billions today. We anticipate better and scalable algorithms for OPTO developed in the future, such as using multi-agent workflows to scale to large graphs and VLMs to interpret multi-modal parameters and feedback.\n\n# Lacking an automated method  \n\nWe wish to clarify that Trace constructs the computational graph automatically (line 183) while the program executes (dynamically like in PyTorch), rather than requiring users to pre-define it or pre-compile it (statically like in Tensorflow 1). This means that the created graph changes automatically based on the workflow process, with different inputs potentially resulting in different graph structures. \nIn Section 3.1, we discuss how to abstract the workflow using @bundle operators. This abstraction is different from manually defining the graph, which may be the source of the confusion. The @bundle decorator is an optional feature that allows users to simplify the workflow’s text representation (e.g. to reduce input token costs), enabling LLMs to better understand and optimize the workflow. \n\n# Limited Experimental Improvement  \n\nWe respectfully disagree on the comment on the performance gap between OptoPrime and OPRO. In almost all experiments (Figure 1, 5b, 6a, 6b), OptoPrime is a significant improvement over OPRO, e.g. 2x-4x improvement in success or rewards. Only the experiment in Figure 6c shows the different algorithms’ performance is within the error margin.\n\n# Difference between Node and Graph Optimization \n\nConsider the following example that shows why optimizing over a graph is better than optimizing only individual nodes. \n```python\n@bundle()\ndef function1(x):\n    return x > 0\n@bundle()\ndef function2(y):\n    return y % 2 == 0\ndef xor_test(x, y):\n    return function1(x).neq(function2(y))\ninput1 = node(3, trainable=True); input2 = node(4, trainable=True)\nxor_test(input1, input2).backward(feedback=”Find a set of inputs to make the return True.”)\n```\nWhen we optimize an individual node, we only see that node input, the function output, and the feedback. We do not see the other inputs and how they can affect the outcome. When we optimize for the full graph however, we can see that each input only partially affects the outcome, and we need to jointly optimize both inputs to achieve a desired outcome.\n\n# Discrepancy between Fig 5 and Fig 6’s Conclusions \nIn all the experiments, OptoPrime with memory (denoted as Trace) performs better than OptoPrime without memory (denoted as Trace NoMem) and OptoPrime with memory but with the execution trace info removed from the prompt (denoted as Trace Masked). We noticed that the current writing is not clear about what each method means, which may cause confusion. Across the different ablations and Figure 5 and 6, we consistently see that memory improves performance, and masking the execution trace information hurts performance. We will better clarify the ablations in the revision. \n\n# Token Efficiency \nThank you for the excellent suggestion, we will include token counts for the OPRO and OptoPrime prompts in the paper. Here are the statistics for the prompts at the first iteration of optimization (note OPRO's token usages grows with iterations):\n\n| Domain    | OPRO | OptoPrime |\n| -------- | ------- | ------- |\n| Numerical Opt | 175 | 918 |\n| BigBench-Hard | NA | 1883 |\n| Traffic Opt | 198 | 1679 |\n| MetaWorld | 470 | 7101 |\n| Battleship | 437  |  1305  |\n\nWe can see that indeed OptoPrime consumes significantly more tokens than OPRO. However, we observe consistently that even allowing 7-10x more iterations of OPRO so as to equalize token costs, the OPRO performance plateaus to a worse level than OPTOPrime (e.g. Figure 1: OPRO at Iter 7 vs. OptoPrime at Iter 2; Figure 5b: OPRO at Iter 50 vs. OptoPrime at Iter 5; Figure 6b: OPRO at Iter 30 vs. OptoPrime at Iter 10, etc.). OPRO is suboptimal not due to a token limit but instead a lack of information, which is captured and represented using Trace.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCKuTW3vLw", "reviewer": "Reviewer_GPUE", "summary": "The paper proposes a problem specification (OPTO) and a solution (optimizer OptoPrime) as well as a software framework (Trace) for agentic program optimization. The paper demonstrates high performance of the bundle above in solving 5 benchmarks. The paper demonstrates the ability to optimize 3 types of parameters: numeric, text prompts, and python code. The paper proposes Trace as an analogue of imperative-style AutoDiff engines like PyTorch for agentic programs relying on (black box) LLM inferences. The evaluated benchmarks cover a reinforcement learning setting (Battleship, Traffic control and Robot manipulation), scalar regression (Numerical optimization) and question answering (classification and more in BigBenchHard).", "questions": "[1] On lines 79-80 you claim that “Remarkably, there is 80 no mention of Battleship nor details on how the functions reason and act should behave or adapt 81 in Fig. 2a”. However in Appendix H, Iteration 0 (Initialization) you provide relatively detailed instructions: “Given a map, analyze the board in a game. On map, O denotes misses, X denotes successes, and . denotes unknown positions.”. Since the Battleship game is a very well known one, ChatGPT could reproduce the code memorized from the public repositories. How do you assess the risk of this type of leak?\n\n[2] Where can an example of prompt optimization of an LLM agent as per line 767 be found?\n\n[3] I appreciate quoting “Complete” on Figure 2a as the example is not clear without `__call__()` and `select_coordinate()` that can only be found in the supplementary materials in exp/battleship_exp.py -> Policy2. I suggest replacing it with “An excerpt from”.\n\n[4] In Figure 3 it is not clear how can g1, g2 and g3 be different since the connectivity of the graph is defined by the optimized torch-style module, specifically a chain of 2 optimizable nodes Reason and Act in the example.\n\n[5] It would be valuable to know the USD cost of OpenAI API usage per experiment.\n\n[6] On the technical side, python version is not mentioned, datasets and ray are not on the requirements list. However I appreciate providing OAI_CONFIG_LIST_sample.", "limitations": "[1] The authors honestly admit that one of the main limitations of the current approach is LLM context length. Indeed, the scaling of the proposed algorithm that packs the entire trace into a single context for LLM inference inside OptoPrime.\n\n[2] Scaling of the proposed framework to more learnable parameters and more sophisticated agentic programs is yet to be demonstrated.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "[1] The authors do a great job in demonstrating the operational performance of Trace on 5 examples: Battleship, Numerical optimization, Traffic control, BigBenchHard, and Robot manipulation.\n\n[2] Great to see Trace compared with both traditional optimizers and the recent LLM-based (OPRO) in Figure 5b.\n\n[3] It is nice to see that the proposed method is designed to generalize to several types of payload (demonstrated 3: numeric values, text and code).\n\n[4] Experiments in “5.3 Unifying Prompts and Functions Optimization” neatly showcase joint optimization of prompts and code which is a clear novelty, even though one may argue that a text prompt and a text of a piece of code are both textual information. Good to see that the optimization works even without the history like Yang 2024 (OPRO) or population like Pryzant 2023 (ProTeGi).\n\n[5] The learning seems very sample-efficient with the reported scores achieved in just several single-sample iterations.\n\n[6] After running exp/run_prompt_bigbench.py I confirm the reproducibility of the results on BBH.", "weaknesses": "[1] The paper does not compare with “TEXTGRAD: Automatic Differentiation via Text” (arXiv:2406.07496v1) which is a concurrent work.\n\n[2] The narrative goes from describing a piece of software (Trace) to the problem specification (OPTO) and then to the algorithmic solution (OptoPrime). I would expect the problem specification and the proposed algorithm to be explained first while the software details are discussed later or even in the Appendix.\n\n[3] The experiment in 5.3 confirms that the proposed method works in principle, however due to the small scale of the optimized program (3 parameters), there is no clear signal that Trace will scale up for 10s or 100s of trainable parameters.\n\n[4] The definition of a trace (line 126) is not given. I am familiar with the verb “to trace” with respect to a symbolic program, however it is unclear what “a trace” means. From a random web page (https://www.ituonline.com/tech-definitions/what-is-an-execution-trace/):\n\nDefinition: Execution Trace\nAn execution trace is a record of the sequence of operations executed by a program during its run. This trace includes function calls, variable values, and the flow of control among other details.\n\nThis definition is not mathematically clear.\n\n[5] The work lacks the analysis of Figure A.12 where the learned prompt template does not make much sense.\n\n[6] Running the baseline exp/run_prompt_bigbench_dspy.py fails.\n\n[7] The paper does not demonstrate the optimization success in agentic programs with memory, i.e. when the optimized class like Predict below has internal state that changes from call to call:\n\n```\n@trace_class\nclass Predict(LLMCallable):\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions.\n# Comparison with TextGrad \nPlease note that we submitted Trace to NeurIPS on May 15 2024, and TextGrad was uploaded to arXiv and Github on June 11 2024. So a direct comparison was impossible.\n# Narrative\nWe considered the narrative you suggested and decided that we needed a very strong motivation for an entirely new approach to optimization (Section 1.3). We found that showing the Trace software in action first was the fastest way to demonstrate that this kind of optimization (OPTO) is feasible, practical and powerful.\n# Limitation on Scalability\nWe agree that scalability is a limitation of OptoPrime, as discussed in Sec 6 and Sec 7. We wish to clarify that this limitation pertains specifically to OptoPrime and not to Trace itself. Trace is designed to scale efficiently to large graphs with minimal overhead (see line 242) and can handle non-textual nodes.\nWe are inspired by the historical development of back-propagation, starting from optimizing networks with 10s of neurons (in the Rumelhart et al paper in 1986) to billions today. We anticipate better and scalable algorithms for OPTO developed in the future.\n# Definition \nWe will include a clear definition of an execution trace. The execution trace is defined as the sequence of operations and their execution results invoked when computing the output from a set of inputs. This execution trace can be expressed as a computational graph defined in Preliminary. We will clarify that the DAG g is the computational graph defined above. \n# Analyzing learned prompts \nWe will highlight the learned prompts in the revised Sec 5. One lesson from several automatic prompt engineering works is that human intuition is not a good guide for prompt engineering. Seemingly innocuous (and perhaps unreasonable) changes to a prompt template (like that in Figure A.12) can have large effects on an LLM’s behavior, thus motivating the need for their automatic optimization e.g. via Trace. \n# Bugs in Running DSpy baseline\nThank you for running our code. Unfortunately, the code uploaded to OpenReview included a stale version of the DSPy script. We apologize for the oversight. We provide a snippet of our correct setup in the one-page pdf. We will update the supplementary material.\n# Optimizing stateful agents\nOptimizing stateful agents is similar to optimizing recurrent neural networks; we need to explicitly represent the state as inputs to the learned functions (discussed in line 193). Our experiments on Meta-World and Battleship are examples of stateful problems. In these problems, the environment is stateful and that state is returned as input to the functions that Trace’s learning; these graphs are similar to that of optimizing an agent with an internal state. We hope these experiments are sufficient to address the reviewer’s concern.  \n# Battleship game\nWe independently found this error after the paper submission. The exact function definition used in the experiments is given in Appendix H, which is consistent with the code we submitted. You are right that Figure 2a over-simplifies important details about the function docstring and the current text in line 79 is misleading. We will change it to “Remarkably, there is no mention of Battleship environment APIs nor details on how the functions reason and act should behave or adapt in Fig. 2a”. \n\nWe agree that GPT4 likely has trained on code about the battleship game. But it does not know the API of the Battleship game (because we coded it up from scratch). If GPT4 did know how to solve the problem we presented, it could have solved it in the first iteration after one update, but that is not what we observed (Figure 1). This gap between the performance after one update and multiple updates indicate the sign and need of learning from feedback and interactions.\nWe will use “excerpt” in the revision to clarify the code snippets in Figure 2.\n# LLM Prompt Optimization Example\nAn example of LLM agent as per line 767 can be found in a new experiment that we conducted for the [virtualhome](http://virtual-home.org/) environment. This experiment is included in the one-page PDF for the rebuttal, including the code and figure. Virtualhome is a collaborative environment that requires two agents to work together to solve household tasks. Trace is asked to optimize and update a specific part of the prompt, which is the plan for future actions. Prior work (Guo et al., \"Embodied LLM Agents Learn to Cooperate in Organized Teams\", 2024) forces agents to have a round of conversation before they start the task. We show that Trace allows agents to have naturally emerging pro-social behaviors for some tasks (such as “putting plates into the dishwasher”), but not others (such as “reading a book”). \n# Graphs in Fig 3\nFigure 3 is an illustration of a general OPTO problem setup. In the Battleship example, you are correct that the graph is the same in every iteration. But, for a general optimization problem, the graph structure can be different, e.g. different parameters changing program flow during the forward pass, or simply because the execution is stochastic. Meta-World is an example where the graph structure can be different across iterations. The graph is a chain describing the multi-step interactions with the environment, and each episode ends either when the robot successfully solves the problem or when timeout happens. Therefore, iterations with successful episodes can have a shorter chain than those that fail and time out. \n# Token Cost\nThe cost depends on the graph size and the tokens required to describe the problem. Running OptoPrime with GPT-4-Turbo for the most complex experiment in the paper, MetaWorld, costs <$30 USD for one task (over 10 seeds, 30 iterations). Costs of other experiments are a fraction of this.\n# Technical Details\nWe currently require Python>=3.8, and we will update the setup.py to add the missing dependencies. Thank you for your feedback and running the code!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for reviewing this paper. This PDF contains figures and codes of the new virtual home experiments and a correction on the submitted code for running DSpy baseline, which is for addressing Reviewer GPUE's questions.  

Virtualhome is a collaborative, stateful environment that requires two LLM agents to work together to solve household tasks. Trace is asked to optimize and update a specific part of the prompt, which is the plan for future actions. Prior work (Guo et al., "Embodied LLM Agents Learn to Cooperate in Organized Teams", 2024) forces agents to have a round of conversation before they start the task. We show that Trace allows agents to have naturally emerging pro-social behaviors for some tasks (such as “putting plates into the dishwasher”), but not others (such as “reading a book”).

Meta-review: This paper introduces a novel end-to-end optimization framework, Trace, which views the computational process as a graph and utilizes execution trace information to optimize parameters. As a case study, this paper also developed a general-purpose optimizer, OptoPrime that can effectively solve OPTO problems,such as  first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, etc.,

The limitations of the works lie in the scalability such as the context length and graph size.  Despite some concerns about scalability and the need for clearer experimental results, the strengths of the paper—especially its novelty, versatility, and demonstrated performance—outweigh these weaknesses. Therefore, I recommend acceptance, with the suggestion that the authors consider revising the narrative structure and enhancing the clarity of experimental results in the final version.
Decision: Accept (poster)