Gödel Agent: A Self-Referential Framework Helps for Recursively Self-Improvement

Authors: Xunjian Yin, Xinyi Wang, Liangming Pan, Xiaojun Wan, William Yang Wang
Primary area: applications to robotics, autonomy, planning
Keywords: Agent, Large Language Model, Reasoning, Self-Improvement

Abstract:
The rapid advancement of large language models (LLMs) has significantly enhanced the capabilities of AI-driven agents across various tasks. However, existing agentic systems, whether based on fixed pipeline algorithms or pre-defined meta-learning frameworks, cannot search the whole agent design space due to the restriction of human-designed components, and thus might miss the globally optimal agent design. In this paper, we introduce Gödel Agent, a self-evolving framework inspired by the Gödel machine, enabling agents to recursively improve themselves without relying on predefined routines or fixed optimization algorithms. Gödel Agent leverages LLMs to dynamically modify its own logic and behavior, guided solely by high-level objectives through prompting. Experimental results on mathematical reasoning and complex agent tasks demonstrate that implementation of Gödel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.

Decision: reject
Reviewer scores: 5, 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730129342594

thread_type:
official_review

<thread_object>
creator: reviewer_j1mu

<content>
summary:
The paper presents the Godel Agent, a prompt engineering framework inspired by the Godel machine, claiming to be a self-improvement framework. The authors evaluate Godel Agent in 4 tasks, and the game of 24.

strengths:
The research topic is interesting and important. The paper provides an extensive evaluation.

weaknesses:
1. There are several ambitious claims within the paper; for example, the abstract suggests that the method can explore the entire space to find the optimal solution. This is a bold statement and does not seem to be substantiated by the findings presented.
2. The central claim about the capability to create "different agents" from an initial agent mostly involves modifying the conditioning of the autoregressive order through prompt engineering.
3. Figure 1 shows some learnable components, yet the framework itself does not appear to incorporate learnable elements.
4. Line 50 claims "to eliminate the human design prior," which is challenging given that LLMs inherently contain human priors from being trained on human-generated text.
5. Discussions on self-awareness of LLMs are speculative and might be better left out of a scholarly paper focused on introducing a new method.
6. Section 3 is notably small and lacks clarity on the method’s implementation. It would benefit from additional details, particularly on how self-improvement is enacted within the Godel Agent and how evaluations are conducted.
7. Lines 215-233 imply that evaluations rely on the environments provided. If this involves using the test set for evaluation, it represents a significant limitation of the method.
8. How the solutions in Godel Agent are evaluated (referenced in line 6 of the pseudocode) needs clearer explanation.
9. It would be beneficial to compare the cost of the Godel Agent with other methods, such as the CoT, both in terms of time and financial resources.
The authors might want to reevaluate their claims and revise the paper to provide a clearer presentation of the proposed method.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476109353

thread_type:
official_review

<thread_object>
creator: reviewer_7jqx

<content>
summary:
The authors propose a self-evolving framework, Gödel Agent, inspired by the Gödel machine, to enable recursive self-improvements of agents, without relying on predefined routines or fixed optimization algorithms.
The authors conduct experiments to validate the proposed approach.

strengths:
1) A framework
The authors propose a self-evolving framework, Gödel Agent, inspired by the Gödel machine, to enable recursive self-improvements of agents, without relying on predefined routines or fixed optimization algorithms.

2) Experiments
The authors conduct experiments to validate the proposed approach.

weaknesses:
1. Why LLM-based agent is a valid approach? No LLM is perfect. All LLMs make mistakes. What is reliable information? What is ground truth? Then why should we rely on LLMs to make decisions? For software engineering, no LLM can achieve a pass@1 of 100%. To make it worse, the popular evaluation method with several tests, is not perfect, so the evaluation results are not fully reliable.

2.
How to make LLM-based agent a valid approach? LLMs may help generate plausible solutions. But there should be a mechanism to filter out or fix incorrect solutions.

The current submission does not provide such a mechanism.

How to improve an LLM-based agent? 
It seems this boils down to waiting for stronger LLMs, which is likely out of the scope of the paper, though.

3.
There are papers discussing / criticizing the reasoning / planning capacity of LLMs, which is the foundation of LLM-based agent (including those based on the Gödel machine), e.g.

LLMs can’t plan, but can help planning in LLM-modulo frameworks, in ICML, 2024.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv 2024

Without sound capabilities for reasoning / planning, how to build LLM-based agents, in particular, the Gödel Agent?

It appears that empirical improvements are not enough to answer the above questions.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730493019326

thread_type:
official_review

<thread_object>
creator: reviewer_xqbt

<content>
summary:
This paper proposes a new self-evolving framework, Godel Agent, which leverages LLMs to dynamically modify self-generated logic and behavior. Empirical experiments show that this framework enables agents to recursively improve themselves without predefined routines or fixed optimization algorithms.

strengths:
1. The workflow proposed in this paper is interesting; the core idea of recursive improvement is not implemented by improving the response itself step-by-step, but through updating the policy in an indirect way.

2. The paper is well written, and easy to follow.

3. The illustrations in figure 1 and examples given in Appendix C are clear and informative.

weaknesses:
1. The motivation for adding four additional tools is not well-justified; there is no evidence supporting that these are the correct tools to add or that they comprehensively cover all needs. It's counterintuitive because if we claim the model has the ability to refine its policy and develop new policies, why do we need to provide tool guidance? Additionally, it's notable in Appendix C that although there is variability in the best policy, this policy seems can be decomposed into a combination of the listed tools, which suggests this work is not fully distinguished from meta-learning optimized agents where humans manually define the high-level strategy, and the agent learn a strategy to combine or rank the high-level strategy.

2. The current framework heavily relies on the model's capability to generate and refine policies, as well as generate responses given a policy and query. However, this would not improve the model's ability to acquire new knowledge if such kind of off-line dataset available.

3. The current refinement requires an oracle to progress the performance (U in line 6 from Algorithm 1). For the current Table 1, is the number of oracle utility function calls the same across all methods, and is the number of model queries the same for different settings?

questions:
1. Based on Appendix C, the best policy found by Godel Agent for MMLU is the same as CoT-SC, why are the two numbers reported in Table 1 different? Similarly for the best policy for GPQA, why is it different from CoT? If this is due to extended few-shot examples or specific instructions, then it seems more like a prompt engineering problem, suggesting we're not using optimal prompts.

2. The reason why the method is evaluated on MGSM instead GSM8K is not well-justified, for my understanding, multi-lingual is not a target of the framework. Moreover, all current benchmark, even regular CoT has high accuracy, this together with limitation, I am questioning the generalizability of the framework on harder tasks, like MATH (https://github.com/hendrycks/math), it is reported that MATH data is contanmintated in some training dataset, so it would be safer to evaluate on MATH500 (https://huggingface.co/datasets/qq8933/MATH500), and omni-MATH (https://omni-math.github.io/). Additionally, the current evaluation would benefit from other agentic tasks, including code generation tasks, like SWEBench (https://www.swebench.com/).

3. For the related work, some recent recursive self-improvement work is not discussed
[1] Havrilla, Alex, et al. "Glore: When, where, and how to improve llm reasoning via global and local refinements." arXiv preprint arXiv:2402.10963 (2024).
[2] Hosseini, Arian, et al. "V-star: Training verifiers for self-taught reasoners." arXiv preprint arXiv:2402.06457 (2024).
[3] Qu, Yuxiao, et al. "Recursive introspection: Teaching language model agents how to self-improve." arXiv preprint arXiv:2407.18219 (2024).
[4] Kumar, Aviral, et al. "Training language models to self-correct via reinforcement learning." arXiv preprint arXiv:2409.12917 (2024).

4. How to safeguard the current workflow if the model is allowed to modify the policy and interaction code directly, and the policy will be used to generate further responses. How can we prevent runtime memory issues and ensure that both the code implementation and the final generation are not harmful?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696003092

thread_type:
official_review

<thread_object>
creator: reviewer_waxh

<content>
summary:
* A big limitation of LLM agents is that there are handcrafted, self-improving agents are a promising direction to make this more autonomous
* self referential framework that enables agents to recursively improve themselves without predefined routines
* Agent can alter its own code and runtime memroy

The paper introduces Gödel Agent, a self-referential framework that enables agents to recursively improve themselves without predefined routines or fixed optimization algorithms.
Inspired by the Gödel machine, Gödel Agent allows agents to modify their own code and logic using large language models (LLMs), guided only by high-level objectives.
The agent uses techniques like monkey patching to read and alter its runtime memory, achieving self-awareness and self-modification.
Experiments across coding, science, and math tasks show that Gödel Agent outperforms manually crafted agents in performance, efficiency, and generalizability.

strengths:
* Seems like a promising and open-ended approach for removing humans in the loop of agent pipeline building
* ADAS Meta Agent is a strong similar baseline and Godel outperforms it
* Demonstrated effective self-improvement across multiple domains including reasoning

weaknesses:
* Experiments seem limited in scope—mostly on controlled tasks; might not scale to complex real-world applications or embodied tasks
* The final policies that are returned by the method don't seem very complicated or different from a basic human designed template. The agent designs discovered in the ADAS paper seem much more complex and creative. What explains this difference? 
* Given the limited evaluations, the room for self-improvement seems limited, can we really get much better than the base model for mathematical reasoning with fancy agent pipelines and prompts? A better showcase of the method would be on openended or embodied tasks vs text based reasoning
* What’s the upper limit of self improvement, does it saturate?
   *Whats the improvement in the 6 iterations
* More detailed comparison to ADAS is needed in related work (Automated Design of Agentic Systems)
   * What is Meta Agent Search and why does this Godel agent perform better?
      * This needs to be clearer in the paper
* The method is somewhat vague and not clear what’s going on, and what this has to do with self reference/recursion. Is this different than just tasking an LLM with modifying its own agent code?

questions:
How does Gödel Agent ensure safety and prevent harmful behaviors during self-modification? How do you put constraints on it such that it doesn't blow all of your GPT credits or hog all the GPUs on your system?

What mechanisms are in place to handle errors or prevent the agent from degrading its performance over time?

Can this approach scale to more complex tasks that require long-term planning or interaction with unpredictable environments?

How does the agent's performance depend on the underlying LLM's capabilities? Can it surpass those limitations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730737547609

thread_type:
official_review

<thread_object>
creator: reviewer_alkv

<content>
summary:
The paper proposes a self-improving language model agent ("Godel agent") and evaluates it on a variety of benchmarks. The results suggest that the self-improving agent achieves better performance than hand-designed agent scaffolds or meta-learned agents, while also being cheaper to run than the latter.

strengths:
* The proposed method is conceptually simple and clear and provides and intriguing possibility of building self-improving agents with current LLM technology.
* The experiments include multiple benchmarks, a number of baselines and generally suggest strong results.
  * The authors run ablations of different components of their agent and find all of them improve performance.
* The qualitative analysis of the results is insightful and provides a good understanding of the strategies the self-improving agent implements.

weaknesses:
**Results**
I'm concerned about the presentation of the experiments and the **fairness of the empirical evaluation**. My key concerns are that the Godel agent uses significantly more inference compute than any of the other methods and that the description of the evaluation protocol is not sufficient to determine if the evalaution is fair.

Here is my understanding of the evaluation setup (please correct me if I'm wrong):
* The Godel agent is run for N iterations on a validation set of samples. In each iteration is produces a new policy and a new learning function.
* After N iteration the policy is evaluated on held-out test problems and this result is reported.

This process gives the Godel agent significantly more inference compute than any of the hand-designed agent systems, which makes the comparison unfair. The only fair comparison in Table 1 is to Meta Agent Search which seems to typically be comparable in performance. Based on these results, I am not convinced the the Godel agent actually improves performance in a fair comparison.

This concern could be addressed by letting the baseline be a best-of-N method that applies N somewhat random perturbations to the policy and chooses the best of them. For example, this could be implemented by using GPT-4 to generate N different prompt variations and choose the best according to performance on the validation data.

Relatedly, according to Appendix B the Godel agent uses GPT-4 for the learning algorithm but all methods only use GPT-3.5 for the policy. This makes the comparison additionally skewed in favor of the Godel agent which has access to GPT-4 in contrast to the other methods. A fair comparison would use GPT-4 for both the learning algorithm and the policy.


**Presentation**

I also have some concerns about the presentation and framing of self-improving agents in the paper. The paper often uses the term "self-awareness" without qualification. I'm concerned that anthropomorphization leads to a less scientific discussions and I would recommend the authors try to rewrite the paper to remove phrases like "Our Godel Agent achieves self-awareness".

Moreover, potential risks from self-improving AI are not appropriately discussed. There is a significant literature on risks from self-improving AIs and this is a key concern in the AGI safety community. I think this literature should be acknowledged in the paper and there should be a more complete discussion of broader impacts of this technique than currently the case.

questions:
**Clarification questions**: Please answer the following questions to help me better evaluate the paper.

* Does the learning agent ever see the data the final policy will be evaluated on?
* For how many iterations do you run the Godel agent and how did you determine this hyperparameter?
* In general, how did you tune prompt variations and hyperparameters for this agent? How did you prevent overfitting to the test data?
* Could you please clarify the action space of the agent and how the agent learns about the possible interactions? Is it only given the prompt in Appendix A or is it also given few-shot samples for example?
* How does the agent execute the policy? Is this happening automatically each iteration, or does the agent manually chose to exit the policy? Is this different during iteration and during evaluation?

**Concerns**: Addressing the following concerns would make me reconsider my score.
* Clarify the evaluation protocol.
* Justify the choice of having baselines that use less inference compute. Provide a best-of-n baseline or argue why this is not a good comparison.
* Clarify the comparison to Meta Agent Search and wheather your method produces qualitatively better solutions.
* Justify the choice to use GPT-3.5 for the policy models. If possible, provide some results using GPT-4 for the policies.

**Overall assessment**
I'm intrigued by the premise of the paper, but quite concerned about the evaluation and reproducibility. If the authors can address my concerns and answer my questions, I'm willing to reconsider my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732290646506

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-up: request for engagement during discussion period

comment:
Dear Reviewers,

With only a few days remaining in the discussion period, we would greatly appreciate your engagement to ensure a constructive dialogue. In our revision, we’ve worked hard to address your feedback, making significant improvements to the paper. Detailed responses to reviewer-specific points in the each separate comment below.

We are eager to hear your thoughts on these updates and hope you’ll have a chance to review our responses. We value your time and effort in shaping this submission.

Thank you again for your thoughtful reviews and for considering our responses.

Best regards,

The Authors of Submission 13979
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an ambitious goal of creating a fully self-improving agent that can modify its own programmatic code. However, there are a few concerns that make this work not yet ready for publication. Firstly, the evaluation is not fair on a compute basis, since there is no control on the number of generation calls (mentioned by Reviewer ALKV). Secondly, the reported results for the Gödel agent are not better than ADAS, as the error bars are all overlapping, with the exception of MGSM. Deeper analysis revealing how the incremental program modifications and final programs produced by Gödel agent differ from and contribute to its success with respect to baselines like ADAS would make the contribution here much clearer.