Symbolic Learning Enables Self-Evolving Agents

## Abstract

Abstract The AI community has been exploring a pathway to artificial general intelligence (AGI) by developing “language agents”, which are complex large language models (LLMs) pipelines involving both prompting techniques and tool usage methods. While language agents have demonstrated impressive capabilities for many real-world tasks, a fundamental limitation of current language agents research is that they are model-centric, or engineering-centric. That’s to say, the progress on prompts, tools, and pipelines of language agents requires substantial manual engineering efforts from human experts rather than automatically learning from data. We believe the transition from model-centric, or engineering-centric, to data-centric, i.e., the ability of language agents to autonomously learn and evolve in environments, is the key for them to possibly achieve AGI. In this work, we introduce agent symbolic learning , a systematic framework that enables language agents to optimize themselves on their own in a data-centric way using symbolic optimizers . Specifically, we consider agents as symbolic networks where learnable weights are defined by prompts, tools, and the way they are stacked together. Agent symbolic learning is designed to optimize the symbolic network within language agents by mimicking two fundamental algorithms in connectionist learning: back-propagation and gradient descent. Instead of dealing with numeric weights, agent symbolic learning works with natural language simulacrums of weights, loss, and gradients. We conduct proof-of-concept experiments on both standard benchmarks and complex real-world tasks and show that agent symbolic learning enables language agents to update themselves after being created and deployed in the wild, resulting in “self-evolving agents”. We demonstrate the potential of the agent symbolic learning framework and open-source the entire framework to facilitate future research on data-centric agent learning.

## 1 Introduction

Recent advances in large language models (Radford et al., , ; Brown et al., ; Ouyang et al., ; OpenAI, ; Touvron et al., , ) open the possibility of building language agents that can autonomously solve complex tasks. The common practice for developing AI agents is to decompose complex tasks into LLM pipelines where prompts and tools are stacked together (Park et al., ; Hong et al., ; Zhou et al., ; Chen et al., ; Xie et al., ). In a sense, language agents can be viewed as AI systems that connect connectionism AI (i.e., the LLM backbone of agents) and symbolism AI (i.e., the pipeline of prompts and tools), which partially explains their effectiveness in real-world problem-solving scenarios.

However, the current state of language agents development is limited by the extensive engineering effort required to build and customize language agent systems for a specific task. Specifically, researchers and developers have to manually decompose complex tasks into subtasks, which we refer to as nodes, that are more tractable for LLMs and then carefully design prompts and tools, including API functions, knowledge bases, memories, etc., for specific nodes. The complexity of this process makes the current landscape of language agent research model-centric, or engineering-centric. This means it is almost impossible for researchers to manually tune or optimize language agents on datasets on which we can train neural nets in a data-centric way. This limits the robustness and versatility of manually coded language agents and requires substantial engineering effort to adapt language agents to new tasks or data distributions. We believe the transition from engineering-centric language agents development to data-centric learning is an important step in language agent research.

To this end, a number of recent efforts has been made on automatic optimization of language agents. For example, DSpy (Khattab et al., ) introduces a framework for algorithmically optimizing LLM prompts via bootstrapping or random searching in a combinatory space of different prompt components and GPTSwarm (Zhuge et al., ) further proposes to tackle the combinatorial optimization challenge raised in DSPy via an iterative optimization process. Agent-pro (Zhang et al., ) proposes a framework to optimize the prompts components corresponding to the agents’ internal policy in competitive environments. AgentOptimizer (Zhang et al., ) proposes a framework to optimize functions with carefully engineered prompts. While effective in some scenarios, these approaches only optimize separate modules in an agent system such as a prompt for a specific node. As a result, these optimization methods are prone to local optimum of isolated prompts, tools, and nodes that leads to compromised performance for the entire agent system. This resembles the early practice in training neural nets (Hinton and Salakhutdinov, ) where layers are separately optimized and it now seems trivial that optimizing neural nets as a whole leads to better performance. We believe that this is also the case in agent optimization and jointly optimization of all symbolic components within an agent is the key for optimizing agents.

Figure: Figure 1: Analogy between agent symbolic learning and neural nets connectionist learning.
Refer to caption: x1.png

In this work, we introduce a agent symbolic learning framework for training language agents. The agent symbolic learning framework is inspired by the connectionist learning procedure (Hinton, ) used for training neural nets. To be specific, we make an analogy between language agents and neural nets: the agent pipeline of an agent corresponds to the computational graph of a neural net, a node in the agent pipeline corresponds to a layer in the neural net, and the prompts and tools for a node correspond to the weights of a layer. In this way, we are able to implement the main components of connectionist learning, i.e., backward propagation and gradient-based weight update, in the context of agent training using language-based loss, gradients, and weights. Specifically, we implement loss function, back-propagation, and weight optimizer in the context of agent training with carefully designed prompt pipelines. For a training example, our framework first conducts the “forward pass” (agent execution) and stores the input, output, prompts, and tool usage in each node in a “trajectory”. We then use a prompt-based loss function to evaluate the outcome, resulting in a “language loss”. Afterward, we back-propagate the language loss from the last to the first node along the trajectory, resulting in textual analyses and reflections for the symbolic components within each node, we call them language gradients. Finally, we update all symbolic components in each node, as well as the computational graph consisting of the nodes and their connections, according to the language gradients with another carefully designed prompt. Our approach also naturally supports optimizing multi-agent systems by considering nodes as different agents or allowing multiple agents to take actions in one node.

The agent symbolic learning framework is an agent learning framework that mimics the standard connectionist learning procedure. In contrast to existing methods that either optimize single prompt or tool in a separate manner, the agent symbolic learning framework jointly optimizes all symbolic components within an agent system, including prompts, tools, and the pipeline that stacks them into an agent system. This top-down optimization scheme also enables the agent symbolic learning framework to optimize the agent system “holistically”, avoiding local optimum for each separated component. This makes it possible for language agents targeting complex real-world problems to effectively learn from data, opening up the possibility to transform the current state of language agent research from engineering-centric to data-centric. Moreover, since the language-based loss function does not require ground-truth when generating the language loss, our framework enables language agents to learn from experience and delibrately update all their symbolic components after being created and deployed in the wild, enabling “self-evolving agents”(^1^11Agents can also collect training data in the wild and update the LLM backbone via fine-tuning. In this way, all components in the agent can be updated. We leave this for future work.).

As a proof-of-concept, we conduct a series of experiments on both standard LLM benchmarks and complex agentic tasks. Our results demonstrate the effectiveness of the proposed agent symbolic learning framework to optimize and design prompts and tools, as well as update the overall agent pipeline by learning from training data. We open-source all codes and prompts in the agent symbolic learning framework to facilitate future research on data-centric agent learning.

## 2 Related Work

### 2.1 Language Models, Prompts, and Language Agents

Language model is a family of machine learning model that is trained to evaluate the probability of sequences of words or tokens. Large language models (LLMs) (Radford et al., , ; Brown et al., ; Ouyang et al., ; OpenAI, ; Touvron et al., , ) often refer to language models that adopt the autoregressive probability factorization scheme, parametrized by the Transformer architecture (Vaswani et al., ), consists of a large amount of parameters, and trained on large-scale corpus. With scaling of model size, training data, and computation, LLMs have demonstrated remarkable capabilities in generating human-like texts and understanding context.

Prompts, on the other hand, is the key for unleashing the capabilites of LLMs. Prompts are critical components in controlling the behavior and output of LLMs and serve as the interface between human and LLMs. The design of prompts significantly impacts the performance of language models and a number of progress have been made on prompt engineering, including in-context learning (Brown et al., ), chain-of-thought prompting (Nye et al., ; Wei et al., ), ReAct (Yao et al., ), self-refine (Madaan et al., ), self-consistency (Wang et al., ), recurrent prompting (Zhou et al., ), etc.

Language agents further extend the functionality of language models beyond simple prompting by allowing LLMs to use tools (Schick et al., ) and integrating LLMs into broader systems capable of executing multi-step tasks (Park et al., ; Hong et al., ; Zhou et al., ; Chen et al., ; Xie et al., ). By stacking prompts and tools into carefully designed pipeline, agents are versatile in various applications, from customer service automation to advanced data analysis.

### 2.2 From Automated Prompt Engineering to Agent Optimization

With the increasing popularity of prompt engineering in both academic and industry, a number of recent work investigated methods to automate the prompt engineering process. For example, Pryzant et al. () and Yang et al. () uses carefully designed prompts to unleash LLMs’ ability to do prompt engineering for themselves. On the other hand, Prasad et al. () and Guo et al. () employs different search algorithms such as genetic algorithms for prompt optimization.

Since prompts are critical components of agents, the success of automated prompt engineering opens up the possibility of automated agent optimization. Similar to the case in automated prompt engineering, methods for agent optimization can also be categorized into two categories: prompt-based and search-based. For example, Agent-pro (Zhang et al., ) and AgentOptimizer (Zhang et al., ) leverage carefully designed prompts to optimize either the prompts or the tools in a node of the agent pipeline. These methods work on isolated components within an agent. Another line of research explored search-based agent optimization algorithms.  Sordoni et al. () uses variational inference to optimize stacked LLMs. DSpy (Khattab et al., ) uses search algorithms to find the best prompts or nodes in a combinatory space. GPTSwarm (Zhuge et al., ) further improved the search algorithm for the combinatory optimization problem. These approaches have a few major limitations. First, the search algorithm mainly works when the metric can be defined numerically with equations that can be coded. However, most agentic tasks are real-world complex problems of which the success can not be defined by some equations, such as software development or creative writing. Second, these approaches update each component separately and therefore suffer from the local optimum of each node or component. These approaches also lack the functionality of adding nodes in the pipeline or implementing new tools. Our proposed agent symbolic learning framework, on the other hand, is the first agent learning method that optimize the agent system “holistically” and is able to optimize prompts, tools, nodes, as well as the way they are stacked into agents.

Furthermore, a number of recent efforts have been done on synthesizing data to fine-tune the LLM backbone of an agent (Chen et al., ; Qiao et al., ; Song et al., ). This line of research is orthogonal to our work and we believe they can be complementary to each other. ICE (Qian et al., ) is also a related work investigating inter-task transfer learning for language agents, which can be complementary with our method for building self-evolving agents.

## 3 Agent Symbolic Learning

Figure: Algorithm 1 Agent Symbolic Learning Framework

### 3.1 Problem Formulation

We first formulate the agent symbolic learning framework by drawing analogies to the components and procedures used in neural network training. We define the key components of the framework and explain the notations used throughout this section.

The agent symbolic learning framework, as illustrated in Figure 2, is inspired by the connectionist learning procedures used for training neural nets (Hinton, ). We first introduce the notations for key concepts by making analogies to that in the connectionist learning framework:

- •
Agent Pipeline $\mathcal{A}$: Similar to the computational graph in neural nets that represents the structure of layers and their connections, agent pipeline represents the sequence of nodes (or steps) through which the agent processes input data. A sequence of nodes $\{\mathcal{N}_{1},\mathcal{N}_{2},\ldots,\mathcal{N}_{n}\}$ that process the input data through various stages. Note that in some agent frameworks, the agent pipeline is input-dependent since the nodes are dynamically assigned during execution, which is similar to the case of dynamic neural nets.
- •
Node $\mathcal{N}$: An individual step within an agent pipeline. The role of a node in an agent is similar to a layer in a neural network. A node $\mathcal{N}_{n}$ receives Node Input $\mathcal{I}_{n}$, which are also in natural language form. In general, the input for a node consists of the output of the previous node and (optionally) inputs from the environment (e.g., human input). The node $\mathcal{N}_{n}$ processes the input $\mathcal{I}_{n}$ with an LLM using both prompts $\mathcal{P}_{n}$ and tools $\mathcal{T}_{n}$(^2^22$\mathcal{T}_{n}$ consists of the input and output for tool usage, and the implementation of the tool itself.). The output $\mathcal{O}_{n}$ is in natural language and passed to the next node.
- •
Trajectory $\tau$: Similar to the role of computational graph of neural nets, the trajectory stores all information during the forward pass, including the inputs, outputs, prompts, and tools usage for each node, and is responsible for gradient back-propagation.
- •
Language Loss $\mathcal{L}_{\text{lang}}$: Language loss in the agent symbolic learning framework is similar to the loss in neural networks since they both measure the discrepancy between the expected and actual outcomes. The main difference is that the language loss is in textual form and is produced by a natural language loss function implemented by a carefully designed prompt while conventional losses are float numbers computed with loss functions that are numerical equations.
- •
Language Gradient $\nabla_{\text{lang}}$: Similar to the role of gradients in connectionist learning, language gradients are textual analyses and reflections used for updating each component in the agent with respect to the language loss.

Figure: Figure 2: Illustration of the agent symbolic learning framework.
Refer to caption: x2.png

### 3.2 Agent Symbolic Learning Procedure

After defining the key components, we can summarize the workflow of the agent symbolic learning framework in Algorithm 1. In this section, we describe each step in the agent symbolic learning framework in detail.

##### Forward Pass

The forward pass is almost identical to standard agent execution. The main difference is that we store the input, prompts, tool usage, and the output to the trajectory, which is used for language gradient back-propagation. This is similar to deep learning frameworks such as PyTorch (Paszke et al., ) and TensorFlow (Abadi et al., ) that store the intermediate outputs and activation in the computation graph of the neural network.

##### Language Loss Computation

After the forward pass, we compute the language loss for a training example by feeding the trajectory into an LLM using a carefully designed prompt template $\mathcal{P}_{\text{loss}}$:

$$ $\mathcal{L}_{\text{lang}}=\text{LLM}(\mathcal{P}_{\text{loss}}(\tau))$ (1) $$

The key is the design for the prompt template, which is expected to holistically evaluate how the agent performs with respect to the input, environment, and task requirements. To this end, we carefully design a prompt template for language loss computation consisting of the following components: task description, input, trajectory, few-shot demonstrations, principles, and output format control. Among them, task description, input, and trajectory are data-dependent while the few-shot demonstrations, principles, and output format control are fixed for all tasks and training examples. The language loss consists of both natural language comments and a numerical score (also generated via prompting). We can optionally feed the ground-truth label for the input when generating the language loss. We call this scenario supervised agent learning. It can also generate language loss without ground-truth by evaluating the output and trajectory according to the task description. In this case, we can say that the agent is doing unsupervised agent learning, which enables language agents to self-evolving.
We present the detailed implementation of this prompt template in the Appendix.

##### Back-propagation of Language Gradients

In standard connectionist learning, the goal of gradient back-propagation is to calculate the impact of the weights with respect to the overall loss so that the optimizers can update the weights accordingly. Similarly, in our framework, we also design a “back-propagation” algorithm for language gradients. Specifically, we iterate from the last node to the first node and compute the gradient for each node with LLMs using a carefully designed prompt:

$$ $\nabla_{\text{lang}}^{n}=\text{LLM}(\mathcal{P}_{\text{gradient}}(\nabla_{ \text{lang}}^{n+1},\mathcal{I}_{n},\mathcal{O}_{n},\mathcal{P}_{n},\mathcal{T} _{n},\mathcal{L}_{\text{lang}}))$ (2) $$

The prompt template $\mathcal{P}_{\text{gradient}}$ is designed to instruct the LLM to generate language gradients that are analyses and reflections for the symbolic components within the node. Inspired by the idea of back-propagation, we give the language gradients of the node executed after the current node, as well as the information on the execution of the current node, which is stored in the trajectory. That’s to say, when doing analysis and reflection, the LLM not only needs to consider how the prompts and tools suit the subgoal of the current node but also has to consider how they affect the accomplishment of the subgoal of the next node. By chaining from top to bottom, the language gradients for all nodes are relevant and responsible for the overall success of the agent. This method effectively reduces the risk of optimizing toward the local optimum for each isolated prompt and tool, leading to the overall performance of agent systems.

##### Language Gradient-based Update

The final step in the framework is to update the prompts and tools in each node and optimize the overall agent pipeline with the help of language gradients. This is accomplished via “symbolic optimizers”. Symbolic optimizers are carefully designed prompt pipelines that can optimize the symbolic weights of an agent. We create three types of symbolic optimizers: PromptOptimizer, ToolOptimizer, and PipelineOptimizer. We present detailed implementation of these prompts in the Appendix.

PromptOptimizer: To facilitate prompt optimization, we split prompts into different components, including task description, few-shot examples, principles, and output format control. We then design separate prompts tailored for the optimization of each prompt component. All prompts share a detailed explanation and demonstration of how the LLM should focus on the language gradients when reasoning about how to edit the original prompt components.

ToolOptimizer: The ToolOptimizer is a pipeline of prompts that first instructs the LLM to decide the kind of operation it should use: whether the tools should be improved (by editing the tool description used for function calling), deleted, or new tools need to implement. Then the ToolOptimizer calls different prompts specifically designed for tool editing, deletion, and creation.

PipelineOptimizer:
The goal of the PipelineOptimizer is to optimizer the agent pipeline consisting of nodes and their connections. The prompt is designed to first introduce the agent programming language used to define the agent pipeline (we use the agent programming language introduced in Zhou et al. ()). Then the prompt describes the definition of a few atomic operations that the LLM can use to update the pipeline, including adding, deleting, and moving the nodes. It then instructs the LLM to first analyze how the pipeline could be improved and then implement the update using the atomic operations. Detailed descriptions of the agent programming language and the atomic operations used to update the agent pipeline are available in the Appendix.

Since all aforementioned optimizers operate in natural language space and some optimization operations need to be done in code space, we use a simple strategy that retries any illegal update up to three times and discards the update if the error persists. We also use a rollback strategy that re-runs the current example after optimization and rolls back to the original agent if the performance evaluated using the language-based loss function drops. Furthermore, we also include a “learning rate” component for each prompts in the optimizers which controls how aggressive the LLM should be when optimizing prompts, tools, and agent pipelines.

##### Batched Training

The aforementioned optimization scheme works with one training example at a time, which resembles stochastic gradient descent. Inspired by the fact that mini-batch stochastic gradient descent works better, or more stably, in practice, we also devise a batched training variant for symbolic optimizers. Specifically, we conduct forward pass, loss computation, and back-propagation for each example separately. Then we feed a batch of language gradients for the same node, and prompt the LLM to holistically consider all these language gradients when updating the agent.

## 4 Experiments

### 4.1 Settings

#### 4.1.1 Tasks

We conduct experiments on both standard LLM benchmarks and more complex agentic tasks. We describe the tasks, datasets, and evaluation metrics as follows:

**Table 1: Results on Standard LLM Benchmarks.**
| Methods | HotPotQA | MATH | HumanEval |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | GPT-3.5 | GPT-4 | GPT-3.5 | GPT-4 | GPT-3.5 | GPT-4 |
| GPTs | 24 / 38.8 | 33 / 44.3 | 23.2 | 53.1 | 59.2 | 71.7 |
| Agents | 27 / 37.5 | 39 / 49.8 | 23.8 | 56.0 | 59.5 | 85.0 |
| Agents w/ AutoPE | 29 / 39.8 | 38 / 50.3 | 22.5 | 57.2 | 63.5 | 82.3 |
| DSPy | 35 / 43.9 | 40 / 50.5 | 17.3 | 48.4 | 66.7 | 77.3 |
| Ours | 35 / 44.8 | 41 / 54.0 | 38.8 | 60.7 | 64.5 | 85.8 |
|  |  |  |  |  |  |  |

##### Standard Benchmarks

We conduct experiments on standard benchmarks for LLMs including HotpotQA (Yang et al., ), MATH (Hendrycks et al., ), and HumanEval (Chen et al., ). HotPotQA is a multi-hop QA task challenging for rich background knowledge. We use the “hard” split in the dataset since we find it to be more challenging for language agents. MATH is a collection of challenging competition mathematics problems. HumanEval is an evaluation set that requires LLMs or agents to synthesize programs from docstrings. As for evaluation metrics, we use F1 and exact match for HotPotQA, accuracy for MATH, and Pass@1 for HumanEval. Tools are disabled in these datasets to ensure the results comparison is meaningful with existing literature on these tasks.

##### Complex Agent Tasks

We consider creative writing and software development as two complex agentic tasks. For the creative writing task, we follow Yao et al. () and give 4 random sentences to the agents and ask them to write a coherent passage with 4 paragraphs that end in the 4 input sentences respectively. Such a task is
open-ended and exploratory, and challenges creative thinking as well as high-level planning. We use GPT-4 score to evaluate the passages following (Yao et al., ). The software development task, on the other hand, requires the agent system to develop an executable software given a simple product requirement document (PRD). We evaluate the compared agents according to the executability of the generated software, which is quantified by numerical scores ranging from $1$ to $4$, corresponding to increasing levels of execution capability. Specifically, a score of $1$ signifies execution failure, $2$ denotes successful code execution, $3$ represents conformance to the anticipated workflow, and $4$ indicates flawless alignment with expectations.

#### 4.1.2 Baselines

We compare our proposed method against the following baselines:

- •
GPTs: a simple baseline that use GPT and a carefully designed prompt;
- •
Agents: a language agent method implemented using the Agents (Zhou et al., ) framework(^3^33We have tested with other agent frameworks such as OpenAgents and AgentVerse and got similar results.) with a carefully designed prompts, tools, and pipeline;
- •
DSpy: a LLM pipeline optimization framework that can search the best combination of prompt components. It is not applicable for complex agent tasks where the evaluation metric can not be defined in equation and code;
- •
Agents + AutoPE: a variant where the prompt in each node of the agent pipeline is optimized by an LLM following the method described in Yang et al. (). It does not involve language gradient back-propagation and language gradient-based optimization.

We report results with both GPT-3.5 and GPT-4. We use the gpt-3.5-turbo-0125 endpoint for GPT-3.5 and the gpt-4-turbo-0409 endpoint for GPT-4. As for our approach, we start with the Agents baseline and then conduct agent symbolic learning on top of it.

### 4.2 Results

**Table 2: Results on software development.**
| Task | GPTs | Agents | Ours |
| --- | --- | --- | --- |
| Flappy bird | 2 | 2 | 3 |
| Tank battle game | 1 | 2 | 4 |
| 2048 game | 1 | 2 | 4 |
| Snake game | 2 | 3 | 4 |
| Brick breaker game | 2 | 3 | 4 |
| Average score | 1.6 | 2.4 | 3.8 |
|  |  |  |  |

##### Results on LLM Benchmarks

The results on standard LLM benchmarks are shown in Table 1. We can see that the proposed agent symbolic learning framework consistently improves over all compared methods. The performance improvement on MATH, a competition-level benchmark, is especially large. In contrast, conventional LLM-based prompt optimization method (Agents w/ AutoPE) and the search-based prompt optimization approach (DSPy) are not as stable: they results in good performance improvements in some cases but leads to significant performance degration in some other cases. This suggests that the agent symbolic learning framework is more robust and can optimize the overall performance of language agents more effectively.

##### Results on Complex Tasks

We present the results on software development and creative writing in Table 3 & 3, respectively. We can see that our approach significantly outperforms all compared baselines on both tasks with a even larger performance gap compared to that on conventional LLM benchmarks. Interestingly, our approach even outperforms tree-of-throught, a carefully designed prompt engineering and inference algorithm, on the creative writing task. We find that our approach successfully finds the plan, write, and revision pipeline and the prompts are very well optimized in each step. We also find that the agent symbolic learning framework recovers similar standard operation procedure developed in MetaGPT (Hong et al., ), an agent framework specifically designed for software development. This confirms the effectiveness of the proposed agent symbolic learning framework on real-world tasks where there is no ground truth and the overall performance cannot be calculated by equations or codes, as contrary to search-based algorithms such as DSPy.

### 4.3 Case Study & Analysis

Figure: Figure 3: An case study conducted on creative writing task.
Refer to caption: x3.png

We then show a case study for the optimization dynamics of the agent symbolic learning framework in Figure 3. We can see that our approach can effectively do prompt engineering and designing the agent pipeline in the way a human expert develops language agents.

Moreover, we find that the initialization of the agent system has non-negligible impacts on the final performance, just as the initialization of a neural nets is important for training. In general, we find that it is generally helpful to initialize the agent in the simplest way and let the symbolic optimizers to do the optimization. In contrast, the performance tends to become unstable if the initial agent system is over-engineered. A natural extension of this observation is that maybe we can do some kind of pre-training on large-scale and diverse tasks as a versatile initialization for general-purpose agents and then adapt it to specialized tasks with agent symbolic learning. We also find that the success of our approach is more significant and stable on complex real-world tasks compared to that on standard benchmarks where the performance is evaluated by accuracy or F1. This suggests that future research on agent learning should focus more on real-world tasks, and the agent research community should work on building a benchmark focusing on agent learning evaluation that consists of diverse complex agentic tasks and investigating robust approaches to measure progress.

## 5 Conclusion

This paper introduces agent symbolic learning, a framework for agent learning that jointly optimizes all symbolic components within an agent system. The agent symbolic learning framework draws inspiration from standard connectionist learning procedure to do symbolic learning. It uses language-based loss, gradients, and optimizers to optimize prompts, tools, and the agent pipeline with respect to the overall performance of the agent system. The proposed framework is among the first attempts to optimize agents that can solve complex real-world tasks using sophisticated pipelines. Our frameworks enables language agents to “learn from data” and perform “self-evolve” after being created and deployed in the wild. We conduct several proof-of-concept experiments and show that the agent symbolic learning framework can effectively optimize agents across different task complexity. We believe this transition from model-centric to data-centric agent research is a meaningful step towards approaching artificial general intelligence and open-source the codes and prompts for the agent symbolic learning framework to accelerate this transition.

## Appendix A Implementation Details

We adopt the agent programming language and framework introduced in Agents [Zhou et al., ], a language agent framework that enables developers to build language agents that stacks prompts and tools together into complex pipelines. The main advantage of the Agents framework is that it enables developers to use a config file to define the agent system, which makes it easier for the symbolic optimizers in the agent symbolic learning framework to perform update operations on the agent system.

## Appendix B Prompt Templates

**Table 4: Prompt Template for Language Loss Function**
| Prompt Template for Language Loss Function |
| --- |
| Loss with ground truth: |
| You are a fine-tuner of a large model. I will provide you with some output results from the model and the expected correct results. You need to evaluate these data and provide a score out of 10, please wrap the score using <score></score>. Additionally, please provide some suggestions for modifying the model’s output, using <suggestion></suggestion> to wrap your suggestions. |
| Here is the model’s output: |
| <result>result</result>; |
| The expected result is: |
| <ground_truth>ground_truth</ground_truth> |
| Please note: |
| 1. Ensure that the output is wrapped with <score></score> and <suggestion></suggestion> respectively. |
| 2. The output should be as consistent as possible with the expected result while being correct. For example, if the expected result is “BUST”, and the model’s output is “The women’s lifestyle magazine is ’BUST’ magazine.”, even though the answer is correct, you should advise the model to be more concise. |
| 3. The standard for a score of 10 is that the model’s output is exactly the same as the expected result in a case-insensitive manner, and without any unnecessary content. Even if the model’s output is semantically correct, if it includes superfluous content, points should be deducted. |
| Loss with ground truth and score: |
| You are a large language model fine-tuner. I will provide you with a model’s output and the expected correct result. You need to evaluate it and suggest modifications to the model’s output. Please use ‘<suggestion></suggestion>‘ to enclose your feedback. |
| Below is the model’s output: |
| <result>result</result> |
| The expected result is: |
| <ground_truth>ground_truth</ground_truth> |
| Here is the evaluation score for the model. Your goal is to optimize this score: |
| <score>score</score> |
| The relevant information about this score is as follows: |
| <evaluation_info>score_info</evaluation_info> |
| Note: |
| 1. Ensure that ‘<suggestion></suggestion>‘ exists and appears once. |
| 2. If the model’s output is satisfactory, you can output <suggestion>The output is satisfactory, no additional requirements</suggestion>. |
| 3. The output should be as close to the expected result as possible while ensuring correctness. For example, if the expected result is "BUST" and the model’s output is "The women’s lifestyle magazine is ’BUST’ magazine.", even though this answer is correct, you should remind the model to be concise. |

**Table 5: Prompt Template for Gradient Back-propagation**
| Prompt Template for Gradient Back-propagation |
| --- |
| Prompt-Level |
| You are now a prompt fine-tuner for a large language model. You are tasked with providing suggestions for optimizing the prompt template. |
| Please enclose your suggestions using <suggestion></suggestion>, for example, <suggestion>it could be made shorter</suggestion>. |
| The task is divided into multiple steps; I will provide you with the output from the previous step, the requirement proposed by the next step for the current output, the current output itself, and the prompt template. You need to suggest improvements for the current step’s prompt template. |
| - The prompt template that needs optimization is: <prompt_template>prompt_template</prompt_template> |
| - The output from the previous step is: <previous_output>previous_output</previous_output> |
| - The current output is: <output>response</output> |
| - The requirement proposed by the next step for the current output is: <requirement>suggestion</requirement> |
| In addition to suggesting modifications for the current prompt template, you also need to propose requirements for the output of the previous step. Please wrap these using <suggestion></suggestion>, for example: <suggestion>the analysis should include a comparison of original data</suggestion>. |
| Note: |
| 1. Ensure that the results are wrapped with <suggestion></suggestion> and <suggestion></suggestion>, and each tag appears only once. |
| 2. If you are the first node, you can state within <suggestion></suggestion> “This is the first node.” |
| 3. Please note that during your analysis, remember that this prompt template will be applied to multiple different datasets, so your suggestions should be general and not solely focused on the examples provided here. |
| 4. Please analyze step by step. |
| Node-Level |
| You are a large model fine-tuner. Now you need to try to optimize the information of a node. For a complex task, it has been divided into multiple nodes, each of which contains multiple roles that work together to complete the task of this node. Each role is backed by an LLM Agent, and you need to optimize the configuration information of one of the nodes. |
| Here are the relevant explanations for the Node configuration: |
| - The fields in the "controller" indicate the scheduling method of the model. If there is only one role, this item does not need to be optimized: |
| - "route_type" indicates the scheduling method, which has three values: "random" means random scheduling, "order" means sequential scheduling, and "llm" means scheduling determined by the LLM model. |
| - "route_system_prompt" and "route_last_prompt" are used when "route_type" is "llm" and are respectively the system prompt and last prompt given to the LLM model responsible for scheduling. |
| - "begin_role" is a string indicating the name of the starting role of this node. |
| - "roles" is a dictionary where the key is the role name, and the value is the prompt used by this role. |
| You need to decide how to optimize the configuration of this node. Specifically, you need to try to provide suggestions in the following aspects: |
| 1. Update the node description field. This field describes the function of the node and is also an important indicator to measure the performance of a node. |
| 2. Update the scheduling method of the role. Note that if there is only one role, no optimization is needed. |
| 3. Add a new role, and you need to clearly describe the function of this role. |
| 4. Delete a role, and you need to clearly describe the reason for deleting this role. |
| 5. Update a role, and you need to indicate how to update the description of this role. |
| Next, I will give you a Node configuration, and you need to provide optimization suggestions based on the current Node configuration. Please use <suggestion>[put your suggestion here]</suggestion> to enclose your suggestions. |
| ## Current Node Config |
| {node_config} |
| You need to first provide your analysis process, then give your optimized result. Please use <analyse></analyse> to enclose the analysis process. Please use <suggestion></suggestion> to enclose the optimization suggestions for the current node. Please use <suggestion></suggestion> to enclose the requirements for the previous node. |
| Note: The suggestions provided need to be in one or more of the five aspects mentioned above. |

**Table 6: Prompt Template for Optimizers**
| Prompt Template for Optimizers |
| --- |
| Prompt Optimizer: |
| You are now a prompt fine-tuner for a large language model. I will provide you with a prompt template along with its corresponding input and output information. |
| Please modify the prompt based on the provided data: |
| - The current prompt template is: prompt_template. |
| Here is some information about the model when using this template: |
| # Example index |
| - Output result: <output>response</output> |
| - Suggestion: <suggestion>suggestion</suggestion> |
| You need to analyze the content above and input the optimized prompt result. Please wrap your analysis in <analyse></analyse> and the new prompt in <new_prompt></new_prompt>. |
| Please note: |
| 1. When actually using the prompt template, the Python format() method is employed to fill variables into the prompt. Therefore, please ensure that the content enclosed in in both the new and old prompts remains the same, with no variables added or removed. |
| 2. Ensure that your new prompt template can be directly converted to a dictionary using the json.loads() method. Therefore, you need to be careful to use double quotes and escape characters properly. |
| 3. Ensure that <analyse></analyse> and <new_prompt></new_prompt> each appear only once. |
| 4. If you believe that the current prompt template performs sufficiently well, leave <new_prompt></new_prompt> empty. |
| Node Optimizer: |
| You are a large model fine-tuner. Now you need to try to optimize the information of a node. For a complex task, it has been divided into multiple nodes, each containing multiple roles that work together to complete the task of this node. Each role is backed by an LLM Agent, and you need to optimize the configuration information of one of the nodes. |
| Here are the relevant explanations for the Node configuration: |
| - The fields in the "controller" indicate the scheduling method of the model. If there is only one role, this item does not need to be optimized: |
| - "route_type" indicates the scheduling method, which has three values: "random" means random scheduling, "order" means sequential scheduling, and "llm" means scheduling determined by the LLM model. |
| - "route_system_prompt" and "route_last_prompt" are used when "route_type" is "llm" and are respectively the system prompt and last prompt given to the LLM model responsible for scheduling. |
| - "begin_role" is a string indicating the name of the starting role of this node. |
| - "roles" is a dictionary where the key is the role name, and the value is the prompt used by this role. |
| Next, I will give you a Node configuration and several modification suggestions. You need to modify the Node configuration based on the suggestions: |
| ## Current Node Config |
| {node_config} |
| ## Suggestions |
| {suggestions} |
| When providing the modification plan, you need to give the optimized result in the following format. It is a list, each element is a dict, and the dict contains an action field indicating the operation on the Node. |
| Your optimized result should be enclosed in <result></result>, that is, the content inside <result></result> should be a JSON-formatted list, which should be able to be directly loaded by json.loads(). |
| Note: |
| 1. If you think the current configuration is already excellent and does not need modification, you can directly output an empty list. |
| 2. The format of <result>[optimization method]</result> needs to strictly follow the given format, otherwise, it will be judged as incorrect. |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See above
soundness: 2
strengths: - The presentation of this paper is quite interesting.
- The paper includes tasks related to question answering, mathematics, coding and writing.
summary: The paper introduces a multi-agent framework called "agent symbolic learning," which aims to optimize all symbolic components within an agent system holistically. Drawing inspiration from traditional connectionist learning methods, this framework utilizes language-based loss functions, gradients, and optimizers to enhance prompts, tools, and workflows for agents, thereby avoiding local optimum and improving overall performance. It allows language agents to "learn from data" and "self-evolve" after deployment. Several proof-of-concept experiments demonstrate the framework's effectiveness across varying task complexities, suggesting a significant shift from model-centric to data-centric research in agent development, which authers believe is the key for optimizing agents.
weaknesses: - The improvements in the experimental results of this paper are not very significant.
- The paper is more like a multi-agent cooperation framework, describing operations such as reflection and discussion as forward and backward propagation, and combining them into a framework. The paper does not obtain better agents through training or similar methods, so describing it as "evolve" is a bit far-fetched.
- There is a lack of evaluation on complex sequential decision-making agent tasks, such as those in AgentBench [1], AgentTuning [2], AgentGym [3], and τ-bench [4], and the paper lacks comparison with these frameworks.
- The backbone models used are mainly from the GPT series, and it is necessary to include more models, especially open-source models.



[1] AgentBench: Evaluating LLMs as Agents

[2] AgentTuning: Enabling Generalized Agent Abilities for LLMs

[3] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

[4] τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to the different points above.

# References:

[1]: Yuksekgonul, Mert, et al. "TextGrad: Automatic" Differentiation" via Text." *arXiv preprint arXiv:2406.07496* (2024).

[2]: Banerjee, Agarwal, and Singla, ‘LLMs Will Always Hallucinate, and We Need to Live With This’.

[3]: Zhang, Muru, et al. "How language model hallucinations can snowball." *arXiv preprint arXiv:2305.13534* (2023).

[4]: Lanham et al., ‘Measuring Faithfulness in Chain-of-Thought Reasoning’.

[5]: Turpin et al., ‘Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting’.
soundness: 2
strengths: # Strengths:

## Quality:

SQ1: the paper is of good quality, with seemingly high reproducibility.

## Clarity:

SC1: the paper is easy to read and overall well-written.

SC4: I really appreciate the insights of section 4.3 and would like to encourage the authors to expand on their writing by being more explicit about what real-world tasks they have considered and that are not necessarily well-represented in mainstream benchmarks.

## Originality:

SO1: Apart from the comtemporary approach of [1], this paper is as original as it gets. And, if I understand correctly, it improves over [1] by adding (i) the possibility of autonomously modifying the computational graph of the language agent, and (ii) the possibility of autonously updating tools or implement new ones.

## Significance:

SS1: I think the main contribution of the paper has the potential to be greatly significance for the community, but I find the current state of the paper to inadvertently occlude some valuable insights about limitations for instance.
summary: # Problem:

The AI community explores AGI via the ‘language agent’ framework involving a complex workflow of LM prompting and tool usage. This framework can only be improved through through human experts’ manual engineering efforts (engineering/model-centric).

How to depart from this painstaking engineering-centric paradigm?

# Contributions:

The paper aims to switch to a data-centric paradigm where language agents have the ability to ‘autonomously learn and evolve in environments’ from the data they observe, all symbolic nodes being optimized jointly and autonomously rather than separately by domain experts. To this end, they propose the ‘agent symbolic learning’ framework where agents self-optimize using symbolic optimizers.

Symbolic optimizers are mimicking the connectionist learning algorithms of backpropagation and **gradient descent over the substrate of symbolic elements of the language agents, i.e. prompts, tools and the way they are stacked together** (as a graph of nodes), rather than over the substrate of numerical weights. They optimize a **text-based loss** that is the result of applying an LLM-as-a-judge framework onto evaluation of the whole forward pass/agent execution, and use backpropagation from the last to the first node along the computational graph of the agent execution.

**Each symbolic element is then optimized by asking an LLM to modify it based on the backpropagated feedback.**

The papers presents proof-of-concept experiments showing the value of their approach against static agent framework and other prompt/tool optimization methods (e.g. DSPy).
weaknesses: # Weaknesses:

## Quality:

WQ1: Section 3.1 offers an analogy between connectionist learning and agent symbolic learning, but it fails to mention and discuss where does the analogy breaks, such as for instance:

- in the fact that the loss function requires comparison to a groundtruth label which is not instantiated here in the agent symbolic learning ;
    
- in the fact that the lack of groundtruth labels makes the whole approach more akin to an unsupervised learning approach rather than a supervised learning approach (like connectionist learning) ;
    
- in the fact that the connectionist gradient is effective due to mathematical guarantees that ascertain that updating numerical weights in the gradient direction will lead to an optimization of the corresponding loss ; but there is no such guarantee when considering language gradient as they are only the result of an LLM output, which we know to be riddle with issues, such as hallucinations[2,3] and unfaithfulness[4,5] ;
    
- …
    

Adding this discussion would improve the quality and significance of the paper, by detailing to the reader possible limitations that could be addressed down the line, and it would greatly improve my appreciation of the paper towards proposing honest and thought-through research that does not build unnecessary hype.

WQ2: I would like to invite the authors to present results with open-source LMs, as it would be valuable for overall replicability purposes and also to understand better how the approach performs with different LMs.

WQ3: I would like to invite the authors to consider include an experiment that studies how the performance scale with the size of both the forward pass LM and the symbolic optimizer LM. I think it would provide great insights into the potential of the propose approach for the democratisation of LMs, for instance.

## Clarity:

WC1: In Section 3.2, in the paragraph explaining equation (1), ‘task requirements’ are mentioned without much details, and I believe it is the first time it is mentioned in the paper. I would like to invite the authors to define it as precisely as possible, possible in section 3.1 for problem formulation, as I now realise how important of an element it must be (Please clarify if I am mistaken?). I also suspect that it is an element that cannot be optimised in a data-centric way but is still dependent on manual engineering effort, along with the prompt $\mathcal{P}_\text{loss}$, right? If so, I mean to invite the authors to collate all of those elements that are not yet data-centric-ly optimisable within the propose agent symbolic learning framework and propose a discussion of how future works may try to address them?

WC2: In section 3.2, the language loss computation paragraph actually details supervised agent learning vs unsupervised agent learning scenarios. I think it would make better sense to introduce this in Problem Formulation Section 3.1, and clearly highlight in which scenario will the main paper present experiments. I was actually assuming until this paragraph that only the unsupervised agent learning scenario was performed given the mention of using LLM-as-a-judge framework. It might be valuable to nuance and explicitly detail what is meant by LLM-as-a-judge when it is used in Section 3.1 actually.

WC3: As much as possible, it would be really valuable to integrate the different prompt templates for loss and gradient, and gradient update computations into the main text of the paper, in a concise form, or at least point specifically at them and discuss some of their guiding principles further in depth, in order to give a more concrete understanding of their impact to the reader.

## Originality:

WO1: Optionally adding a comparison of the contemporary approach of [1] could improve the impact of the paper.

## Significance:

WS1: Following up on my concerns in SS1, I find that the paper does not sufficiently discusses the limitations of the proposed framework, which result in an impression of building up hype for a product rather than doing thought-through research. I would like to emphasise again that my appreciatetion of the work would be greatly improve if the authors could revise the paper by including a lot more insights about limitations and what does not work fully (yet) and/or how some critical elements may impact the performances of their proposed work. For instance, could it be possible to perform an ablation study on the quality of the task description or the quality of the different engineering-centric prompt templates for loss and gradient and gradient update computation, please?

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. What are the "components of the prompts" and "functions" on line 083 and line 085, respectively, referring to?
2. What exactly is the programming language introduced in Zhou et al. (2023b)? In what sense is it a programming language?
3. How does the "learning rate" defined for each of the prompts apply when optimizing prompts, tools, and agent workflows? 
4. Are Prompt Optimizer and ToolOptimizer part of the proposed framework? Why are they not described in Algorithm 1? Are they used in the experiments?  
5. What optimization method of DSpy is used in the experiments? 
6. Are the results presented on some hold-out test set or the set used for optimization? If applicable, how are the sets split?
soundness: 1
strengths: 1. This work is well-motivated.
summary: This paper proposes agent symbolic learning, a data-centric method for optimizing agentic systems presented in a sequential workflow. As part of the proposed methods, the analogy of back-propagation and gradient descent for optimizing neural networks is presented. Furthermore, optimizers that update tool uses and the agent workflow are introduced. The proposed method was tested on three benchmarks in different domains and showed improved performance compared to other baseline methods. The proposed method also outperforms other baseline methods on software development and creative writing based on excitability and GPT-4 scored metrics, respectively.
weaknesses: 1. The second paragraph of the introduction needs references.
2. The claim that "these optimization methods are prone to a local optimum of isolated prompts, tools, and nodes" in the paragraph from lines 079 to 092 is incorrect. GPTSwarm includes an edge optimization method, which optimizes the connection of nodes globally.
3. Moelding the nodes or components with a sequential structure, as defined in line 213, is not making a full analogy with computational graphs in neural nets, as claimed in line 211. A computational graph of neural nets can be arbitrary directed acyclic graphs.
4. It is better to have a citation for dynamic neural nets at the end of page 4.
5. There is an inconsistency of variable use. Variable $n$. is used for the number of nodes in line 213. However, it is used as an arbitrary index of the node in equation 2. The same inconsistency also applies to lines 12, 13, and 17 of algorithm 1.
6. It is unclear to me how the gradient of the last node is defined.
7. There are typos, such as the word "optimizer" in line 351 should be "optimize." 
8. Line 13 of algorithm 1 appends a gradient to the trajectory. This is not mentioned in the main text. Which strategy should I follow?
9. The abstract claims that the proposed methods outperform others on real-world tasks, which I do not think is the case.
10. It is better to give a reference about how OpenAI implements GPTs in line 423.
11. There is a lack of detail about how the initial agents are implemented.
12. Many experimental details are missing. For example, what are the learning rate and batch size used for experiments? Is the optimization done in a supervised or unsupervised setting? 
13. Many of the references are missing the publisher.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See the Weaknesses section above.
soundness: 2
strengths: **Originality:** The analogy to the neural network training paradigm is interesting.

**Quality:** The experimental results showcase the performance improvements achieved by the proposed approach.

**Clarity:** The framework is well-explained and the writing is easy to follow.

**Significance:** The problem of self-evolving agents considered is important.
summary: This paper proposes agent symbolic learning, a framework to enable automatic self-improvement of language agents in a data-driven way, in comparison to model-centric and engineering-centric optimization methods. The framework draws language-based analogs to back-propagation and gradient descent, optimizing agents via text-based weights, loss, and gradients. The key idea is to optimize the prompt, tool use, and workflow as a whole instead of separate components to avoid local minima. The effectiveness of the method is demonstrated through experiments on several benchmarks, showing improvements over baselines.
weaknesses: 1. **Reliability of Language Loss and Gradients**: The idea of using evaluator networks to improve performance is not new, especially for complex reasoning tasks. However, the reliance on LLMs for computing language loss and gradients raises concerns of reliability, especially when the authors want to "back-propagate" the loss. Deviations may accumulate and make the overall update unstable. Could the authors conduct experiments with complex tasks where the analogous symbolic network has extended node numbers and investigate how the language loss transits under such settings?
2. **Data-Centric Claims and Unsupervised Role of Data**: While the authors claim the framework to be data-centric and can be applied in an unsupervised way, the exact role of data under such setting is not thoroughly addressed. The appendix only provides templates for language loss function under the supervised setting. Could the authors provide the template for the unsupervised setting, and the corresponding experiment results?
3. **Convergence**: Convergence is a critical aspect of neural network training, yet the paper does not discuss when or how the analogous "training" process stabilizes for the framework. Could the authors elucidate specific metrics or criteria they use to determine convergence in their framework and potential challenges in reaching it?
4. **Expanded Baselines and Benchmarks**: While the framework shows promising results on selected tasks over several methods, additional baselines and benchmarks are needed to comprehensively evaluate the performance. Specifically, for the "standard LLM benchmark", the paper does not clearly state the benchmarking method for GPTs. Is it zero-shot or few-shot based? Given that the method uses few-shot examples, few-shot prompting for GPTs would be necessitated for a fair comparison. Also comparisons with other baselines, such as CoT/ToT at least, are important for validating the effectiveness of the method. A wider choice of tasks would also be favorable since creative writing and software development alone might not be sufficient for representing complex agent tasks.
5. **Ablation Studies for Component Analysis**: An ablation study on the components, such as the workflow optimizer and prompt optimizer, would provide more granular insights into each component’s contribution and help understand how the proposed method achieves its reported performance improvements.
6. **Open-Sourcing the Framework**: The authors mentioned that the framework would be open-sourced. I wonder if this could be done in the rebuttal period.

Meta-review: This work introduces "agent symbolic learning" which uses symbolic optimizers to enable language model based agents improve on their own in a data-centric way. Empirically, the authors present proof-of-concept experiments, showcasing the better performance of their framework on several tasks over baselines.

The reviewers pointed out that the paper is interesting, well motivated, and potentially impactful. However, there are a range of weaknesses, including the reliability of the method due to its significant reliance on LLMs, lack of baselines (e.g., "TextGrad: Automatic" Differentiation") and clarity.

Since the authors did not provide a rebuttal, the reviewers' concerns remain. Hence I recommend rejection.

Decision: Reject