How to Correctly Do Semantic Backpropagation on Language-based Agentic Systems

Authors: Wenyi Wang, Hisham Abdullah Alyahya, Dylan R. Ashley, Oleg Serikov, Dmitrii Khizbullin, Francesco Faccio, Jürgen Schmidhuber
Primary area: foundation or frontier models, including LLMs
Keywords: Agentic System, Large Language Model, Backpropagation, Computational Graph

Abstract:
Language-based agentic systems have shown great promise in recent years, transitioning from solving small-scale research problems to being deployed in challenging real-world tasks. However, optimizing these systems often requires substantial manual labor. Recent studies have demonstrated that these systems can be represented as computational graphs, enabling automatic optimization. Despite these advancements, most current efforts in Graph-based Agentic System Optimization (GASO) fail to properly assign feedback to the system’s components given feedback on the system’s output. To address this challenge, we formalize the concept of semantic backpropagation with semantic gradients—a generalization that aligns several key optimization techniques, including reverse-mode automatic differentiation and the more recent TextGrad by exploiting the relationship among nodes with a common successor. This serves as a method for computing directional information about how changes to each component of an agentic system might improve the system’s output. To use these gradients, we propose a method called semantic gradient descent which enables us to solve GASO effectively. Our results on both BIG-Bench Hard and GSM8K show that our approach outperforms existing state-of-the-art methods for solving GASO problems. A detailed ablation study on the LIAR dataset demonstrates the parsimonious nature of our method.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730382631337

thread_type:
official_review

<thread_object>
creator: reviewer_wczz

<content>
summary:
Recently, Language-based agentic systems have shown great potential in solving real-world tasks. However, these works usually relied on human-designed settings (e.g., instruction or prompt). How to enable an automatic optimization for agentic systems has been a challenge. In this paper, authors introduce a semantic backpropagation with semantic gradients, which explores how to change each component of an agentic system to improve outputs. Experimental results on BIG-Bench and GSM8K demonstrate that the proposed method can solve GASO problems.

strengths:
This paper formulates agentic system as a computational graph task and then introduces a semantic backpropagation to optimize each component of agentic system via semantic gradient.

weaknesses:
**Weaknesses**

1. Although authors claim the difference between TextGrad as TextGrad use textual gradients while the proposed method is semantic gradients, it seems the proposed method still heavily relied on language features and human design (e.g., Implementation 2).
2. Authors claim that in TextGrad ignores the dependency and heterogeneity, can you provide some real cases about the nodes with dependencies and why they are failed in TextGrad? Does it matter in optimizing agentic system? Can you highlight more insights about your paper when compared with TextGrad?
3. The proposed method still relied on an evaluation system to obtain feedback. So what happen if applying this method to some open scenarios. Besides, this paper only evaluates Big-Bench and GSM8k. Can you try more datasets to validate the generalization of the proposed method like GAIA. MMLU?

questions:
1. Can you provide some examples about semantic gradients when compared with TextGrad. From the provided code, the semantic gradient seems is still the natural language form.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673474644

thread_type:
official_review

<thread_object>
creator: reviewer_hxpc

<content>
summary:
To enable text optimization (omitting numerical gradients) of model outputs based on feedback, the authors propose Graph-based Agentic System Optimization (GASO). This framework formalizes the concept of semantic backpropagation with semantic gradients, generalizing several key optimization techniques, including reverse-mode automatic differentiation and TextGrad, by exploiting the relationships among nodes with a common successor. GASO demonstrates effectiveness in both the BIG-Bench Hard and GSM8K datasets.

References:

[1] TextGrad: Automatic "Differentiation" via Text.

strengths:
1. This work follows an important line of research focused on text optimization based on natural language gradients.

2. The experimental results demonstrate significant improvements over existing semantic optimization methods, such as TextGrad.

3. The proposed method is clear and straightforward to implement.

weaknesses:
1. The evaluation datasets are somewhat limited, comprising only BIG-Bench Hard and GSM8K. In contrast, TextGrad has been evaluated on additional datasets, including code generation, drug molecule optimization, and radiotherapy treatment plan optimization. It would be beneficial to see the potential of GASO applied to a wider range of applications.

2. Since GASO incorporates additional neighboring information in its computations, the cost associated with this approach is not clearly defined. It would be valuable to understand the trade-off between computational cost and performance.

questions:
See Weakness above.

1. Performances on other reasoning tasks, such as code generation.
2. The trade-off between performance and cost.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702834886

thread_type:
official_review

<thread_object>
creator: reviewer_hy82

<content>
summary:
This paper falls in the vein of research that's attempting to expand the concept of "gradient-based optimization" to language-based agentic systems. Specifically, several papers have attempted to take the concept of "autodiff" and apply it to graph-based agentic systems. From my understanding, this paper is most similar to TextGrad, but identifies some improvements in TextGrad that 1. make this paper more analogous to standard autodiff,  2. improves performance by taking advantage of more information during the backwards pass.

strengths:
Overall, I think this paper identifies a reasonable gap in the existing textual gradient-based approaches (e.g. TextGrad). Specifically, if I understand the paper correctly, the paper is saying that if you have a function `c = f(a, b)`, the autodiff rule for `da` given `dc` should also depend upon `b`, not just `a`, `c`, and `dc` (which is what TextGrad's formulation does). This follows naturally from autodiff's definition, where    say, the autodiff rule for `da` in `c = mm(a, b)` involves `b`.

The "intuitive" explanation given in the paper also makes sense to me.

This motivation for their results is also validated by their experimental results.

weaknesses:
I personally found the presentation of the paper somewhat confusing. In particular, I think a more concrete example walking through what exactly a semantic gradient looked like would make the paper much easier to understand, particularly if it were contrasted against the gradients from TextGrad. I see the examples of the evolved prompts in the appendix (which are useful!) but I'd be interested to see how the actual gradients look like, since that's the primary way in which this paper differs from prior work.

More broadly speaking, I also find myself being somewhat skeptical of the autodiff analogy, and to what extent semantic gradients meaningfully map to autodiff. For example, for mathematical autodiff a gradient directly corresponds to how that parameter effects the output loss (assuming an epsilon step). With semantic gradients, however, there's no such correspondence like this. 

I would also be interested in seeing how different graph structures lead to different performance. For example, as far as I can tell, all the graphs in this paper involve only one "layer".

I'd also like to see more benchmarks, as opposed to just GSM8k and BBH NLP/Algorithmic.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730775106197

thread_type:
official_review

<thread_object>
creator: reviewer_emlc

<content>
summary:
This work presents a framework to extend the text-based gradient method (ask the model to refine the prompt. This series of works often formulates this refinement as a gradient decent process of the discrete prompt) for handling the graph structure, such as using multiple prompts to form an agent for problem-solving.

strengths:
This work systematically discussed the "backpropagation" of the text-based gradient method and offered a framework to aggregate the gradients, making the text-based gradient methods close to the normal gradient descent.

weaknesses:
The paper writing could be improved. I think placing figures and examples from the appendix into the main document could largely improve the readability. The experiment setting could be more clearly stated in the main document. I found those in the appendix, such as the initial graph for BBH, GSM8K, and LIAR. However, the number of nodes and the diameter of the graph are small, raising concerns about the proposed method's generalization ability on more complicated graphs, such as the MCTS, tree of thoughts, or wizardLM series. 

Needs more in-depth analyses, such as the robustness of the prompt templates for the forward and backward functions. It's also better to test other models than GPT4 since the GPT4 has already achieved a very high score on GSM8K and BBH. Introducing other models could also be helpful in evaluating the backward function design. If I understand correctly, the backward prompt filled with real materials (instruction, question, response, and other statements) could be very long and complicated. My concern is that the GPT4 may not follow it well.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732497457052

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Revision 25 Nov 2024, 01:14 GMT

comment:
We would like to thank all the reviewers for their time and detailed feedback. We have done our utmost to integrate everything that has been said to improve the paper. This has taken some time to do and not been trivial. Most significantly, we have done the following:

1. Cleaned up the text in the paper to resolve grammatical errors and ambiguities.
2. Added BigCodeBench to the list of benchmarks we evaluate our method on.
3. Added ablation experiments for larger graph architectures. (Section A.1)
4. Added ablation experiments for the use of different models for the forward computations. (Section A.2)
5. Added input/output token statistics to determine how much the inclusion of neighboring information affects the cost of the method. (Section B.3)
6. Added a concrete example showcasing the significance of neighborhood information when generating the gradients. (Section C.3)
7. Added prompt robustness experiments for the forward and backward templates. (Section C.4)


Each reviewer will also find a tailored response below. Where logical, we have updated the paper in response to the feedback. Where possible, we have run an experiment to address each comment. We hope that the reviewers will find the improved paper adequate, but we of course remain open to further discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732822773329

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their time and insightful comments. We additionally thank HxPC for their response and their recommendation to accept the paper.

As the discussion period is drawing to a close, we would like to thank reviewers eMLC, hy82, and WCzZ for their comments that we have used to improve our submission. We would also like to gently ask them if our comprehensive responses and the corresponding changes we made to the paper are sufficient to have resolved their concerns? As always, we remain open to further discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new framework called Graph-based Agentic System Optimization (GASO) that extends the concept of gradient-based optimization to language-based agentic systems. It follows the direction of TextGrad, and proposes "semantic backpropagation" with "semantic gradients" to optimize individual components of an agentic system. The key claim is that by incorporating information from neighboring nodes during the backward pass, this method more closely mimics standard automatic differentiation and leads to improved performance. The experimental results on BIG-Bench Hard, GSM8K, and BigCodeBench demonstrate the effectiveness of GASO over baseline methods like TextGrad. A core finding is that considering the dependencies and heterogeneity of nodes within the agentic graph leads to better optimization.

Building upon similar concepts explored in TextGrad, this paper presents a systematic approach to addressing the "backpropagation" process for text-based gradient methods. The proposed framework provides a method for aggregating gradients in a graph-based agentic system, making it closer to traditional gradient descent. The experimental results demonstrate performance improvement over existing semantic optimization methods like TextGrad on the evaluated benchmarks.This paper also identifies a valid gap in existing textual gradient approaches by highlighting the importance of considering neighbor information during the backward pass, an aspect neglected by TextGrad.

The main weaknesses of the paper primarily concern the clarity of the presentation, particularly regarding the concept of "semantic gradients" and the distinction between GASO and TextGrad, which several reviewers found confusing, prompting requests for more concrete examples and visualizations. A further limitation was the initial scope of the evaluation datasets. While the addition of BigCodeBench during the rebuttal offered some value, the experimental results demonstrated only a marginal gain over the baseline (+0.2%), failing to alleviate the concern. Another key challenge identified is the conceptual analogy to traditional autodiff, as semantic gradients lack the direct and guaranteed loss reduction characteristic of numerical gradients. Finally, the computational overhead incurred by incorporating neighboring information remains a notable concern.

Overall, this paper lacks a clear illustration on the difference between GASO and TextGrad. And the focus of this paper is on agentic systems, but the benchmark (e.g., GSM8k) doesn't inherently require the capabilities typically associated with such systems, such as planning and environmental interaction. This raises the question of whether the reported performance gains truly reflect the method's effectiveness and the focus of this paper. Therefore, given the disconnect between the paper's focus on agentic systems and the non-agentic nature of the limited evaluation benchmarks, I lean towards rejection of this paper.