Title: Learning Interestingness in Automated Mathematical Theory Formation
Venue: NeurIPS 2025
Accepted: yes
Keywords: automated mathematics

AI for math

interestingness

reinforcement learning

scientific discovery

theory discovery
Abstract: We take two key steps in automating the open-ended discovery of new mathematical theories, a grand challenge in artificial intelligence. First, we introduce Fermat, a reinforcement learning (RL) environment that models concept discovery and theorem-proving using a set of symbolic actions, opening up a range of RL problems relevant to theory discovery. Second, we explore a specific problem through Fermat: automatically scoring the interestingness of mathematical objects. We investigate evolutionary algorithms for synthesizing nontrivial interestingness measures. In particular, we introduce an LLM-based evolutionary algorithm that features function abstraction, leading to notable improvements in discovering elementary number theory and finite fields over hard-coded baselines. We open-source the \fermat environment at github.com/trishullab/Fermat.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FERMAT, a reinforcement learning (RL) framework for automated mathematical theory formation, and EvoAbstract, an LLM-driven evolutionary algorithm to learn \"interestingness\" measures for guiding theory discovery. FERMAT models theory formation as a Markov Decision Process (MDP), enabling systematic exploration of mathematical concepts, conjectures, and theorems. EvoAbstract synthesizes interestingness metrics by integrating function abstraction, outperforming hard-coded baselines (e.g., HR measures) in discovering 180 ground-truth number theory entities. Key contributions include: (1) formalizing theory formation as an MDP with symbolic actions; (2) an abstraction-learning mechanism for modular program synthesis; (3) empirical validation showing EvoAbstract discovers 20.25 (succ_zero_eq) and 39.74 (arithmetic_base) ground-truth entities, surpassing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. FERMAT’s MDP formalization unifies theory formation steps (definition production, proof attempts) under a standard RL framework .\n2. EvoAbstract’s abstraction learning (e.g., reusable subroutines like calculate_example_balance) improves modularity and performance by 50% over FunSearch .\n3. Rigorous comparisons across baselines (Random, HR, GPT-4o) and ablation studies (FunSearch) validate component contributions .\nPractical Impact: FERMAT’s REPL interface and open-source code enable reproducibility and real-world use in educational theory exploration .\n\n**Weaknesses:**\n\n1. FERMAT does not leverage dependent type systems (e.g., Lean), limiting expressiveness for complex mathematics .\n2. EvoAbstract relies on GPT-4o, introducing bias and high computational cost for reproduction .\n3. Evaluations focus on elementary number theory, lacking validation in algebra or analysis .\n\nQUESTIONS:\n1. Can FERMAT/EvoAbstract be extended to algebra or analysis? A case study on ring theory concepts (e.g., ideals) would validate generalizability.\n2. Are plans to automate production rule discovery (beyond fixed rules in Appendix A.1)? Demonstrating rule evolution would enhance autonomy .\n3. How might integrating Lean’s dependent types improve entity representation? An ablation on type-aware vs. type-agnostic interestingness would clarify impact .\n4. Can EvoAbstract use lightweight neural networks instead of GPT-4o? Reducing LLM dependency would improve accessibility .\n5. How does FERMAT handle knowledge graph redundancy as theories expand? Metrics on graph efficiency (e.g., cycle detection) are needed .\n\nLIMITATIONS:\nThe authors acknowledge technical limitations (scalability, domain scope) but underaddress societal impacts. Suggestions:\n\n(1). Discuss risks of misusing FERMAT to generate flawed mathematical theories in safety-critical applications (e.g., cryptography verification).\n(2). Address biases in the curated ground-truth set, which may favor Western mathematical traditions over others.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Fermat, a RL environment that formally models the formation of mathematical definitions, conjectures, and proofs of conjectures, and EvoAbstract, an evolutionary approach for finding intrinsic reward functions that encourage exploration in Fermat that leads to the proving of certain known definitions, conjectures, and theorems. Fermat works over states modeled through graphs that represent definitions, conjectures, and theorems, and policies update these states by taking a variety of actions (new definitions, conjectures, and proof attempts, backed up by an automated prover). EvoAbstract uses an evolutionary approach that includes defining new intrinsic reward metrics while learning and using an abstraction library. The paper presents experiments that start with two different initial theories, comparing performance with a range of hardcoded intrinsic reward and a prior evolutionary search algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+Uniqueness: to my knowledge, this sort of “automated theory-building” with definition and conjecture formation, in addition to theorem proving, in which an interestingness score is learned, has not been done before.\n\n+Significance: Mathematical theorem proving is of broad interest, as is conjecture formation. Recently, intrinsic reward has been used to guide conjecture formation. This work should hence be of interest to both the theorem proving and open-ended learning communities. The experimental results show considerable improvement over baseline approaches — in particular, the introduction of abstraction learning and seeing the performance boost over FunSearch is really helpful.\n\n+Quality: the Fermat environment provides an interesting, interpretable environment for theory-building.\n\nWeaknesses\n\n-Clarity: while much of the paper was fairly clear, aspects took quite a bit of time to understand. Minor example: you describe the computational interpretation as being a function to evaluate whether an input satisfies a given symbolic definition, which makes me think that it should return a bool, but the example simply implements an operation, returning a number. More serious example: I found 2.2 to be really opaque. In particular, you describe factoring a policy space into two spaces, a space of policy templates and a space of intrinsic reward functions. It was really unclear, at that point, what the class of policy templates might be. My understanding is that you actually just have a simple fixed algorithm for getting a policy from an intrinsic reward (sampling and simulating, using the interestingness of the outcome to choose, Alg 2), but I could only understand this by being directed to Alg 2 which is referenced in section 5.\n\n-Significance: the experimental comparisons, aside from that with FunSearch, seem like a foregone conclusion. These all have hardcoded intrinsic reward functions, whereas FunSearch and EvoAbstract get to optimize over generations. This might be surprising if the intrinsic reward functions were evolved from scratch, but they get to lean on GPT-4o, and the GPT-4o baseline demonstrates that this is a fairly strong starting point. It would be interesting to see how well evolutionary search works if it has a weaker starting-point, or if evolving intrinsic reward is good for theorem-proving (as opposed to directly evolving policies, if possible), or if other forms of open-ended learning (e.g. adapting Minimo to a fine-tuned LLM setting) might be better.\n\nI think it’d also be helpful to relate this work to others on evolving intrinsic motivation [1, 2] and meta-RL [3].\n\n[1] Singh, Satinder, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. \"Intrinsically motivated reinforcement learning: An evolutionary perspective.\" *IEEE Transactions on Autonomous Mental Development* 2, no. 2 (2010): 70-82.\n\n[2] Alet, Ferran, Martin F. Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling. \"Meta-learning curiosity algorithms.\" *arXiv preprint arXiv:2003.05325* (2020). \n\n[3] Norman, Ben, and Jeff Clune. \"First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs.\" *Advances in Neural Information Processing Systems* 37 (2024): 27490-27528.\n\nQUESTIONS:\nCovered in significance concerns above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents two key contributions towards automating mathematical theory formation: FERMAT​​: A novel RL environment that models mathematical discovery as an MDP, with symbolic actions for concept discovery and theorem proving. EvoAbstract​​: An LLM-driven evolutionary algorithm that learns "interestingness" measures to guide theory exploration.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The MDP formalization of theory formation is rigorous and opens new directions for RL research in mathematical discovery.

2. EvoAbstract's abstraction learning mechanism is a clever extension of evolutionary algorithms, enabling compositionality in program synthesis.

Weaknesses：
1. Evaluated only on elementary number theory; generalization to other mathematical domains remains unproven.

2. Dependence heavily on Z3

QUESTIONS:
1. Can you rely on stronger mathematical formalization systems such as Lean4?
2. What exactly are the external rewards? How are they designed?
3. Given that existing models possess a lot of prior knowledge, is there knowledge leakage when using the domain of number theory? Are there better domains for evaluation?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a reinforcement learning environment called Fermat which is subsequently used to learn “interestingness” of mathematical objects. The problem was modeled as intrinsic reward optimization where the goal is to discover the best intrinsic reward function I that can guide action selections to maximize the long-term (sparse) task specific reward. The intrinsic reward functions are implemented as python programs that can be generated by LLMs. More specifically, the authors prompts one LLM () to propose candidates as if it’s doing evolution search, and use another LLM to identify useful subroutines from evolved programs, which are provided as additional context to the first LLM.  The authors show that the learned interestingness measures can better guide the discovery of ground-truth mathematical entities in the theory of arithmetic and its simple fragment.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The overall design of the system is interesting. In particular, instead of blatantly asking the LLM to criticize itself, the integration of an evolution-like algorithm to update the programs proposed by the LLM provides more factual groundings for candidates selection. In addition to that, the abstraction learning also makes sense because validated sub-routines help the (first) LLM avoid going wild, thereby guiding the search towards more promising regions of the program space.\n- The careful formalization makes the problem setting clear. The production rules with examples really helped with understanding how things are modeled as an MDP and how the system progresses. The overall presentation is very good.\n- The qualitative analysis (in addition to the quantitative one) provides explanatory insights. Since what’s generated by the proposed approach are explicit python programs, the exact behavior of them can be observed and interpreted. The authors demonstrated that some subroutines do look similar to human-engineered measures proposed by previous works.\n\nWeakness\n- I am somewhat doubtful about the branding of “interestingness”. My understanding is that the system is essentially just learning the intrinsic reward functions (given the sparse extrinsic one), but this function does not necessarily express interestingness. A transition step can be given a high reward because it’s a tedious but necessary construction towards the target entity. As to the real interestingness, I think you have already “explicitly” defined it using the set of entities \\mathcal{E} (which induces the extrinsic reward)?\n- Minor: learning an intrinsic reward function is fine, but if re-constructing well-known interesting mathematical entities is the ultimate goal, maybe other regular techniques in RL that can deal with sparse rewards are also worth trying.\n\nQUESTIONS:\n1. Clarification needed: page 19, algorithm 2, the interesting measure I takes an entity and the graph G, but by definition on lines 80 and 94 it should take states and actions. Is there a mix of notations here?\n2. In the discussion you mentioned that the templates limited search space, and I agree it is a good idea to do so to limit the computational resources required. I’m just curious, do you have preliminary results when you don’t provide a template at all? What’s the behavior of the LLMs in that case?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors introduce a new reinforcement environment called FERMAT for mathematical discovery. In particular, the mathematical theory is modeled as a Markov Decision Process (MDP), where the state is a graph representing mathematical entities (vertices) and dependencies between the entities (edges). The key challenge of the paper is how to automatically score the \"interestingness\". To do this, the authors develop an approach called EvoAbstract, which is LLM-based, with the aim of learning the intrinsic value of a mathematical object. The evaluation is done using the elementary number theory, using the Z3 solver as the prover.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths ###\n\n- The paper addresses an interesting and relevant problem of mathematics, working in a more open-ended process\n\n- The paper is fairly well-written and clearly motivated, although the central idea concept of interestingness is less clear.\n\n- The approach of defining the mathematical problem as an MDP and then using reinforcement learning to discover theorems and conjectures is an interesting research direction.\n\n\n### Weaknesses ###\n\n- The theory that is used as evaluation is quite limited, and the work does not explore or evaluate the use of richer theories. This is also one of the weaker parts of the paper; by using a prover such as Z3, there are limitations of what can be achieved, compared to if a higher-order logic (e.g., HOL4) or dependently type theory (e.g., Lean or Rocq) is used.\n\n- I find the description of the central concept of \"interestingness\" to be rather vague. As a consequence, the performance comparison with manual interestingness (as described in Section 5) is not obvious. I would suggest that a deeper discussion about interestingness is added to the paper, which includes examples of different levels of interestingness. \n\n- Some figures need clearer explanations. For instance, Figure 2 (which is important) can be described in a clearer way. One way of doing it is to add numbers to the figure and then describe the different parts step-by-step, referring to these numbers in the figure in the caption.\n\nQUESTIONS:\n- To what extent can this approach be translated and used with another prover, e.g., if Rocq, Lean, or HOL4 is used instead? That is, how tightly integrated is the approach to Z3?\n\n- In Section 5.1, it says  \"However, agents guided by the best discovered measures can still prescribe high interestingness to entities of little value.\" So, what does this mean? If a system prescribes high interestingness value, and you (as a human) still think it is little value, why is a certain interestingness score good? \n\n- To what extent is the \"interestingness\"  biased on the LLM's knowledge of interesting theorems (i.e., theorems that have been part of the training data for the LLM)? To what extent can we then discover new theorems of interest that have not already been discovered?\n\n- It is not completely clear how the actions for proving works. t says on line 68 \"attempts to prove or discover it\". But how is the action of proving or disproving a theorem then captured in the MDP? Please clarify this, if possible.\n\nLIMITATIONS:\nLimitations are briefly discussed in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}