GeoDRL: A Self-Learning Framework for Geometry Problem Solving using Reinforcement Learning in Deductive Reasoning
Year: 2023

Abstract:
Ensuring both interpretability and correctness is a great challenge in automated geometry problem solving (GPS), and the scarcity of labeled data hinders learning mathematical reasoning from samples. Therefore, we present GeoDRL, a self-learning geometry problem solving framework that integrates logic graph deduction and Deep Reinforcement Learning (DRL) to optimize geometry reasoning as a Markov Decision Process. GeoDRL employs a Graph Neural Network on a Geometry Logic Graph, updating the problem state using a symbolic system. Incorporating DRL into deductive reasoning enables GeoDRL to achieve unsupervised self-learning while maintaining correctness. GeoDRL, through unsupervised learning, exhibits enhanced accuracy in the Geometry3K dataset, improving by 11.1% over previous SOTA methods, and simultaneously boosts efficiency and interpretability.

heading: 1 Introduction
text: Automated geometry problem solving (GPS) has been a long-standing task in mathematical reasoning research, drawing significant attention from researchers (Wen-Tsun, 1986; Chou et al., 1996; Seo et al., 2015; Sachan and Xing, 2017; Lu et al., 2021, 2022b). As depicted in Figure 1, GPS can be described as follows: given a geometry problem description which typically consists of a diagram and text, the objective is to produce a flow of solution steps leading to the final answer. Solving a geometry problem consists of two key steps: parse and reason, where parsing involves extracting specific interpretation from the question diagram and text and reasoning involves using the interpretation to develop a solution.
There has been a long-standing debate between symbolic and probabilistic approaches in mathematical reasoning. In geometry problem solving,
∗ Corresponding Author
the symbolic approaches (Seo et al., 2015; Sachan and Xing, 2017; Lu et al., 2021) first parse the question diagram and text into logical expressions in a formal symbolic language, and then utilize a predefined set of geometry theorems to solve the problem. These approaches are known for the mathematical rigor, as they construct a symbolic system for logical interpretation and theorem application. In contrast, the probabilistic approaches (Chen et al., 2021, 2022; Zhang et al., 2023) view GPS as a special case of text generation with multi-modal input. These data-driven approaches learn mathematical language and solving strategies from problem-solution samples to generate human-readable solutions without prior mathematical knowledge.
Nevertheless, there are notable limitations to both categories. The symbolic approaches struggle with selecting the appropriate theorem (Gelernter, 1959; Nevins, 1975). The heuristic search strategies tend to involve redundant steps (Lu et al., 2021), which harms the readability of the solution. On the other hand, the probabilistic approaches rely heavily on manually annotated data, which is often scarce and expensive. Furthermore, the solutions generated by deep neural network are doubtful in mathematical rigor and correctness (Chen et al., 2021, 2022). Even an advanced language model like GPT-4 is prone to make arithmetic mistakes
13468
(Bubeck et al., 2023), which is unacceptable in mathematical problem solving.
Accordingly we propose a self-learning GEOS framework, GeoDRL, to addresses these issues. Our framework consists of two phases:
1. Parsing the question diagram and text into a Geometry Logic Graph (GLG). GLG is a heterogenous attributed graph encoding geometric information, where nodes represent geometric primitives (point, line, etc.), edges represent relationships between primitives (perpendicular, parallel, etc.), and attributes represent properties of primitives (length of line segment, measure of angle).
2. Integrating Deep Reinforcement Learning (DRL) into the reasoning procedure. Here, GLG serves as the state and a predefined geometry theorem set serves as the action space. We implement a Graph Neural Network (GNN) to learn the graph representation of GLG to estimate the Q-value of the state. At each step, DRL agent selects a geometry theorem as an action for the current state according to the Q-value. The symbolic system performs it, updates the state, and provides a reward until the problem is solved.
By introducing reinforcement learning, we construct a self-learning framework that enables unsupervised learning of problem-solving strategies, alleviating the scarcity of labeled data. Furthermore, our framework integrates neural network and symbolic system in the deduction, resulting in interpretable step-by-step solutions while maintaining mathematical correctness.
Our contributions are summarized as follows:
• We propose a self-learning geometry problem solving framework, GeoDRL, which integrates Deep Reinforcement Learning into deductive geometry reasoning. This enables training without annotated solution data.
• We integrate decision-making of neural network and symbol-manipulation of symbolic system in our framework, which enables interpretable step-by-step deduction while maintaining mathematical correctness.
• We structure geometry information as Geometry Logic Graph(GLG), which preserves geometry primitives, attributes and relations.
GLG demonstrates its effectiveness in geometry interpretation modeling by outperforming sequential logical expressions.

heading: 2 Related Work

heading: 2.1 Geometry Problem Solving
text: Early work focus on geometry theorem proving (Gelernter et al., 1960; Wen-Tsun, 1986; Chou et al., 1996), which is to demonstrate the truth of a geometric proposition through a logical deduction. These early efforts utilized symbolic computer systems to perform logical deductions and symbol manipulation. More recently, systems for solving geometry problems, such as GEOS (Seo et al., 2015) and Inter-GPS (Lu et al., 2021), have been developed. These systems take geometry diagram and text as input and provide a problem solution, which involves parsing multi-modal information, utilizing theorem knowledge, and conducting quantitative reasoning. In recent years, there has been a trend of training language model on large-scale corpora to learn mathematical reasoning. Examples in the geometry domain include GeoQA (Chen et al., 2021) and UniGeo (Chen et al., 2022), which view GPS as a special case of text generation and use language models to generate solutions. While this approach is highly generalizable, it also carries the risk of lacking mathematical rigor.

heading: 2.2 Deep Reinforcement Learning
text: Deep Reinforcement Learning (DRL) is a machine learning approach that aims to learn a long-term strategy to maximize the reward signal in an optimization problem. In recent years, there have been numerous studies that apply DRL to solve mathematical problems, including math word problems (Wang et al., 2018; Lu et al., 2022a), arithmetic expression calculation (Chen et al., 2018), math theorem proving (Kaliszyk et al., 2018), symbolic reasoning (Poesia et al., 2021) and so on. But there are very few explorations that view geometry problem solving as a RL task. In this paper, we model geometry reasoning as a Markov Decision Process on a logic graph and use Deep Q-Network (DQN) (Mnih et al., 2013) to choose appropriate theorems for deductive reasoning.

heading: 3 Problem Statement and Preliminaries

heading: 3.1 Problem Definition
text: In this paper, we focus on Plane Geometry Problem Solving. We define the task as follows: given a
plane geometry problem P = {D,T} including diagram D and text T , the objective is to find a deduction sequence τ that satisfies the goal g of P . The basic terms in our framework are defined below.
Definition 3.1 (Primitive) A primitive is a basic geometry element that constitutes the diagram, such as point, line, circle, polygon, etc.
Definition 3.2 (Attribute) An attribute is a property of the primitive, usually expressed as a numerical value, such as the length of a line segment, the measure of an angle, and the radius of a circle.
Definition 3.3 (Predicate) A predicate is a geometry relation between primitives, such as point lies on line, line 1 is perpendicular to line 2, etc.
Definition 3.4 (Interpretation) An interpretation is a complete description of all primitives, attributes, and predicates in a geometry problem.
Definition 3.5 (Theorem) A theorem is a priori geometric knowledge applied in interpretation to obtain new primitives, attributes or relations.
Table 1 shows an example of the interpretation of a geometry problem. We further define solution steps as follows: in a n-steps deduction sequence τ = {p1, p2, ..., pn}, pt is the t-th step in τ , which is expressed as pt = It−1
kt→ It, where k ∈ KB is a theorem in geometry theorem set KB and I is the geometry interpretation.
Based on the above definitions, the flow of GeoDRL is described as follows:
1. The parser implements a function Parse(·) to extract the initial geometry interpretation I0 and the goal g from the question diagram D and text T , which is formulated as I0, g = Parse(D,T ).
2. The reasoner implements a function Reason(·) to produce a deduction sequence τ , given the initial interpretation I0, the goal
g and a geometry theorem set KB, which is formulated as τ = Reason(I0, g,KB)
A remaining issue is the selection of the optimal solution τ∗ from multiple candidate sequences C = {τ1, τ2, ...}. We prefer to choose the shortest solution, and if there are multiple solutions of the same length, choose the one with a lower time. It is formulated as τ∗ = argminτ∈C(|τ |, T ime(τ))

heading: 3.2 Reinforcement Learning Modeling
text: In 3.1, we described geometry problem solving, where the reasoning is modeled as solving Markov Decision Process (MDP) (Puterman, 2014). The MDP for geometry reasoning consists of all the potential geometry interpretation states and the transition probabilities between them. Hence, we adopt Q-Learning, a DRL algorithm to choose actions based on the Q-value of state. In this framework, geometry interpretations is the state space S = {si|i = 1, 2, ...}, geometry theorem set is the action space A = {ai|i = 1, 2, ...}. DRL agent is a deep neural network that produces value Q(s, a) for selecting action a in state s. Each time step environment performs an action according to Q(s, a), updates to the new state s′, and provides a reward r. The goal of the algorithm is to make the agent learn a policy π : S → A to maximize the accumulated reward.

heading: 4 Approach
text: This section introduces our proposed geometry problem solving framework, GeoDRL. The overall architecture is illustrated in Figure 2. Here we follow the flow described in 3.1 and introduce the parser and reasoner separately.

heading: 4.1 Parser
text: The parser is aimed to extract primitives and attributes from the question diagram and text, build relations between primitives, and parse them into the interpretation. For structured modeling of geometry information, we propose Geometry Logic Graph (GLG) as the representation of geometry interpretation in our framework. GLG is a heterogenous attributed graph G = (V,E,Σ, L) where (1) V is the set of vertices denoting primitive types; (2) E ⊆ V ×V is the set of edges and e = (u, v) ∈ E is an edge from vertex u to vertex v, denoting geometry relation between these two primitives; (3) Σ is the domain of attributes and L is the function that assigns attributes to vertices. l(v) is the attribute
of v. For example, if v denotes a line segment AB, then l(v) denotes the length of AB. GLG is a directed graph, where each edge e = (u, v) between different types of primitive has its inverse edge e−1 = (v, u). Furthermore, to represent the interpretation I and the goal g within one graph, we add the goal g as a special attribute of targeted primitives in the GLG. Figure 3 shows an example of GLG.
Here we discuss the advantages of GLG compared with symbolic logical expressions in previous works. First, GLG is structured as a graph incorporating geometry interpretation into its nodes and edges, which avoids redundant symbolic grammar. Second, in symbolic logical expression, a primitive is identified by its notation, resulting in the feature learned by the neural network being specific to its notation. It is counter-intuitive, since changing the notation of a point from A to X won’t affect the
problem substantially. In GLG, the feature of a primitive is only determined by its type, attribute and relations with others, instead of its notation.
Specifically, we parse the diagram and text separately. For the diagram, we use PGDPNet (Zhang et al., 2022) as the parser. PGDPNet is an end-toend model for extracting primitives and predicates from geometry diagrams, achieving state-of-the-art performance on plane geometry diagram parsing. PGDPNet takes a diagram image as input, generates the instances of points, lines and circles, and the logical expressions indicating relationships between primitives. For the text, we borrow the rulebased text parser in Inter-GPS (Lu et al., 2021). The semantic parser parses text into symbolic logical expressions, usually including the goal. We combine the logical expressions from the above two branches and structure them into a GLG, consisting of an initial interpretation I0 and the goal g.

heading: 4.2 Reasoner
text: Given an initial interpretation I0, a goal g, and a geometry theorem set KB, the reasoner is aimed to produce a deduction sequence τ where It satisfies the goal g. We model it as a Markov Decision Process and adopt DRL to solve the MDP. We implement an environment, i.e., a symbolic geometric system. The system parses the geometry interpretation and goal, performs actions, updates states and gives rewards. We design a DRL agent to implement the DQN algorithm (Mnih et al., 2013), using GNN to approximate the q-value function.
DRL agent selects an action for the current state according to the Q-value, and interacts with the environment to obtain the reward and the next state.

heading: 4.2.1 Environment
text: The environment is depicted as a symbolic geometric system. The system resolves GLG from 4.1 and stores the geometry interpretation and goal. It applies a geometry theorem, updates the interpretation and generates a reward according to the action instruction given by the agent. In an update process s→ s′, our reward function is set as follows:
Reward = { Solve(s′, g)− αe− tσ s ̸= s′ −1.0 s = s′
(1)
Solve(s, g) = { 1.0 s ⊢ g 0.0 otherwise
(2)
Where t denotes the time spent in the process. The system provides a positive reward 1.0 only when the next state s′ satisfies the goal g, that is, the problem is solved. We add a time penalty factor −αe− tσ to encourage the agent to choose more efficient actions. If the state is not changed by applying the theorem, the reward is set to −1.0 to avoid this action. In one word, the reward function encourages agent to choose a shorter and faster path with less invalid actions.

heading: 4.2.2 Action Space
text: Action space is defined as a geometry theorem set KB, where each theorem k is defined as a conditional statement p ⇒ q with a premise p and a conclusion q. At time step t, when applying theorem k on interpretation It−1, if It−1 satisfies the premise p of k, then It−1 is updated according to the conclusion q:
It−1 ⊢ p⇒ It ← It−1 ∧ q (3) If It−1 does not satisfy the premise p, it will not be updated. Moreover, even if p is satisfied, It may remain unchanged. It often happens when a theorem is used repeatedly in succession. We call both cases invalid theorems, which leads to r = −1.0.
Specifically, we define 24 theorems in this work, covering various classes such as angles, circles, triangles, and polygons. In addition, we define auxiliary-line as a class to deal with some difficult geometry problems.

heading: 4.2.3 GNN Architecture
text: We utilize a GNN to learn the graph representation of GLG to estimate the Q-value of the state. This is accomplished through the implementation of a Heterogeneous Attributed Graph Transformer built upon a transformer-based architecture. Given a GLG G = (V,E,Σ, L), the input of our model is a sequence of nodes with a special token [GRAPH] as the start. [GRAPH] aggregates the node features to capture the global graph representation. Each node includes node type vi and node attribute l(vi), which are projected to node embedding hi:
hi = Etype(vi) + Eattr(l(vi)) (4)
Node type is the type of primitive such as point, line and circle. Node attribute is commonly expressed as a number or math expression with variables. In practice, recording the specific value of every attribute makes little sense. For example, AB=3.2cm, BC=3.5cm. In this case, we only need to know that the lengths of AB and BC are two unequal numerical values. Therefore, we use several replacement words to replace the less frequent attribute values.
The model encodes the structural information of GLG by modifying the self-attention mechanism in transformer. Edges are incorporated into selfattention by adding bias term matrix B, formulated as:
Attention(Q,K, V ) = softmax( QKT√
d +B)V
(5)
Bi,j =    Eedge(ei,j) (vi, vj) ∈ E 0 i = j
−∞ otherwise (6)
When there exists an edge ei,j from vi to vj , the edge embedding of ei,j is added into the position (i, j) of attention score as a bias term. Otherwise, we block the attention channel by setting the score to -∞. Global graph token [GRAPH] attends to every node to aggregate all the features as the global graph feature. As a result, we implement a one-hop graph attention network. Finally, the graph feature vector is sent to an FFN with the output dimension of action space size |A| to produce the Q-value.

heading: 4.2.4 Agent Operation
text: DRL agent operates by interacting with the environment, i.e., the symbolic system. Algorithm
Algorithm 1 DRL Agent Operation 1: s, g ← env.init() 2: M ← {} 3: Solved← False 4: while Solved = False do 5: Q(s, ai)← GNN(s), i = 0, 1, 2, ...|A| 6: a← epsilon− greedy(Q(s, ai), ϵ) 7: s′, r ← env.step(s, a) 8: if s′ ⊢ g then 9: Solved← True
10: end if 11: M .append((s, a, s′, r)) 12: if M.amount() > N then 13: GNN.update() 14: end if 15: s← s′; 16: end while
1 describes DRL agent operation. First, the system initializes an interpretation with GLG obtained from 4.1 as the initial state s0. Then it enters a while loop, where each time step t GNN takes GLG of st−1 as input and calculates Q(st−1, ai) for every action ai. DRL agent adopts ϵ-greedy exploration strategy to choose action at. The symbolic system performs action at, updates state st, and returns a reward rt. Meanwhile, the system checks if st ⊢ g, and if so, assigns a flag Solved indicating jumping out of the loop. The data samples {(st−1, at, st, rt)|t = 1, 2, ...} are accumulated and stored into an experience memory. After it contains a certain amount of samples, we randomly sample one batch each time from it to train the Q-Net using the Bellman-Equation:
Q(s, a) = max a′
(γQ(s′, a′) + r(s, a)) (7)
where the training objective is to minimize the SmoothL1 loss (Girshick, 2015) of Q(s, a) and maxa′(γQ(s
′, a′)+r(s, a)). In the inference stage, DRL agent chooses the action a that maximizes Q(s, a) for the current state s. The symbolic system performs the action and updates the state until the problem is solved. Finally, we gather each step pt and obtain the deduction sequence τ .
Considering the interpretability of the solution, τ needs to be further processed to deliver a humanreadable answer. There might be redundant updated parts in interpretations. For example, there are several pairs of similar triangles at step 2 in
Figure 1, but only△ACD ∼ △CBD leads to the goal. To address it, we record the In ∩ g and backtrack the updates that lead to it. Eventually, we get the final answer.

heading: 5 Experiments
text: In this section, we conduct experiments on public geometry problem datasets and compare the performance of GeoDRL against the previous methods. Furthermore, we analyze the contribution of different schemes by ablation study and discuss the superiority and limitations of our model with typical cases.

heading: 5.1 Implementation and Setup

heading: 5.1.1 Dataset and Parameter Settings
text: We mainly conduct experiments on Geometry3K (Lu et al., 2021), a public geometric dataset with 3000 SAT-style plane geometry problems. PGDPNet is employed as the parser with the public available model weight1. Our symbolic system is builtupon Inter-GPS (Lu et al., 2021), with an improved logical parser and expanded geometry theorem set. We implement a Graph Transformer with 12-layer, 12 attention heads, a maximum of 256 nodes, and a dimensionality of 768, as the Q-Net. For fairness, transformer-based baselines share the same parameter settings and random seeds. Each experiment for GeoDRL is repeated three times and obtained the mean scores. For more parameter settings and experiment details, please refer to Appendix A.

heading: 5.1.2 Training Method
text: To accelerate the training process by avoiding extensive initial exploration, we adopt Imitation Learning (IL) method to make our agent learn an initial policy. Specifically, we utilize the heuristic strategy in Lu et al. (2021) to search for deduction sequences on a few training data samples (500 problems) and train a teacher model on them. We then use the teacher model to interact with the environment and generate samples to Experience Memory following Algorithm 1. A randomly-initialed student model is then pre-trained on these samples for 2000 steps to learn an initial policy from the experiences of the teacher model. Following this, we proceed to train the student model as a DRL agent on the train set.
1https://github.com/mingliangzhang2018/PGDP

heading: 5.1.3 Inference
text: In the inference stage, we adopt the Beam Search algorithm to choose the final solution τ∗. The score of a candidate sequence τ is calculated as:
p(st|st−1, at) = eQ(st−1,at)∑
a′∈A e Q(st−1,a′)
(8)
score(τ) = ∑
t
log(p(st|st−1, at)) (9)
Where p(st|st−1, at) is the transition probability from st−1 to st by taking action at, which is calculated by applying softmax function on the Qvalue. With the beam size of k, we finally obtain k candidate sequences {τ1, τ2, ..., τk} with the top k scores and select the optimal τ∗ as the result. In this study, we implement GeoDRL with the default beam size of 5.

heading: 5.2 Evaluation
text: We evaluate GeoDRL on the Geometry3K dataset with several baselines: three probabilistic approaches including FiLM (Perez et al., 2018), FiLM-BERT (Devlin et al., 2019), FiLM-BART (Lewis et al., 2020), and a symbolic approach InterGPS (Lu et al., 2021), which is the state-of-theart method on Geometry3K. Table 2 shows the results. GeoDRL significantly outperforms the other methods in overall accuracy and most problem
types. Compared to the previous state-of-the-art method Inter-GPS (Lu et al., 2021), GeoDRL obtains 11.1% and 10.9% enhancements in overall accuracy with or without the ground truth parsing results. Notablely, GeoDRL(GT) achieves comparable accuracy scores to the human expert, which demonstrates the superiority of our reasoner.
We evaluate the efficiency of GeoDRL and InterGPS. For fairness, both methods use ground truth parsing results of Geometry3K to avoid the impact of different parsers. As shown in Table 3, compared to Inter-GPS, GeoDRL demonstrates superior performance, achieving not only higher accuracy scores - up to 89.4% with beam size 5 versus 77.5% for Inter-GPS, but also taking less time - a minimum average of 2.67 seconds against InterGPS’s 6.35 seconds. Additionally, GeoDRL solves each problem in fewer steps, averaging 2.27 steps with beam size 1, in contrast to Inter-GPS’s 6.68 steps. Considering that our symbolic system is derived from Inter-GPS, comparing the time and steps of these two methods makes sense. Due to the heuristic algorithm in choosing theorems, InterGPS is found to generate solutions with redundant steps, leading to a longer solving time and more steps. Overall, GeoDRL is notably superior to the previous state-of-the-art Inter-GPS in terms of effectiveness and efficiency.

heading: 5.3 Ablation Study
text: We conduct an ablation study on Geometry3K to figure out the contributions of different schemes in our framework. Results are shown in Table 4. Compared to traditional symbolic expressions, GLG significantly improves the accuracy of single-step theorem prediction from 67.1% to 84.7% when the beam size is set to 1. To explore the effectiveness of unsupervised-learning, we generate solutions with random search and train our model on it. Compared to supervised learning, the self-learning RL strat-
egy significantly improves prediction efficiency: the average time is reduced by 24.6% (0.87s) and 29.6% (2.53s) for beam sizes of 1 and 5, respectively, and also an improvement in accuracy. This indicates that our designed RL strategy with time penalty contributes to learning more efficient solutions. With the heuristic search strategy from Inter-GPS (Lu et al., 2021), we obtain poor-quality solutions with numerous redundant steps, averaging as high as 9.58 steps. Additionally, increasing the beam size leads to an improvement in accuracy, but also increases the average solving time.

heading: 5.4 Discussion
text: Here we discuss the interpretability of our solution. Compared to probabilistic approaches, GeoDRL has explicit parsing and reasoning phases. The parser generates human-readable intermediate results and a visualizable graph. The reasoner provides a step-by-step deduction procedure with mathematical justifications. Unlike symbolic approaches, GeoDRL generates an explicit probability score for each theorem to search for a highprobability solution, providing explanations for every step.
By utilizing the beam search algorithm, based on state transition probabilities, GeoDRL is able to produce multiple solutions for a single geometry problem. Figure 4 shows an example. Providing various solutions is highly beneficial, particularly in online-education applications for students of varying skill levels, where we can tailor the solutions to the student’s learning progress and knowledge base.
In some cases, the pipeline of parsing and reasoning can have negative impacts on problem-solving. The most common one is the shaded area problem, as shown in Figure 5. The key to solve such problems is to express the shaded area as the arithmetic results of regular shapes’ areas, which requires joint efforts of parser and reasoner on the specific diagrams.

heading: 6 Conclusion and Future Work
text: In this paper, we propose GeoDRL, a self-learning geometry problem solving framework by integrating Deep Reinforcement Learning into deductive geometry reasoning, which enables unsupervised learning of problem-solving strategies. We structure geometry information as Geometry Logic Graph to glue discrete geometry literals together. The combination of neural network and symbolic system allows efficient solution while maintaining correctness. Experiments demonstrate that GeoDRL outperforms state-of-the-art approaches in
terms of accuracy, efficiency and interpretability. Our work presents a promising idea of integrating reinforcement learning into deductive reasoning for geometry problem solving. We believe that this framework is applicable to a majority of mathematical reasoning tasks. In the future, we intend to expand our framework to broader mathematical domains to become a general-purpose mathematical reasoning framework.
Limitations
1. Error accumulation: incorrect parsing results will affect the reasoner. In our experiments on Geometry3K dataset, incorrect parsing results lead to a substantial 21.0% performance drop.
2. The reasoner relies on manually predefined theorems, which limits its adaptability.
3. The random exploration in the RL training process leads to uncertainty in the rate of convergence.
Ethics Statement
In this research, We have made every effort to ensure that our study adheres to the ACM Ethical Principles. All data used in this study have been publicly available. This work may inspire the following research of mathematical reasoning. The potential risk of this work is that students may use the system as an auto-problem-solving tool to cheat on exams or assignments.

heading: Acknowledgements
text: This work was supported by National Key R&D Program of China (No. 2021ZD0113301) (No. 2021ZD0113301).

heading: A Experiment Details
text: The number of parameters of Graph Transformer model is around 44,000,000. The model is trained on the train set of Geometry3K dataset for 50000 steps using the AdamW optimizer (Loshchilov and Hutter, 2019), with a batch size of 32 and a learning rate of 3e-5. It costs around 90 minutes to train 1000 steps on 4 Nvidia Titan X GPUs. In reinforcement training, the parameters are set as ϵ = 0.1, γ = 0.5 and α = 1.0, σ = 60 for the reward function. In the inference stage, the time limit to solve one problem is set to 300s.
B Implementation Details
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5.4 and Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
3 A4. Have you used AI writing assistants when working on this paper? ChatGPT. Rephrasing and Polishing. All sections.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 4
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The public dataset used in our study is already checked and anonymized by previous work "Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning" and "Solving geometry problems: Combining text and diagram interpretation".
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.1 Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.1 Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we intend to expand our framework to broader mathematical domains to become a general-purpose mathematical reasoning framework. Limitations
1. Error accumulation: incorrect parsing results will affect the reasoner. In our experiments on Geometry3K dataset, incorrect parsing results lead to a substantial 21. 0% performance drop. 2. The reasoner relies on manually predefined theorems, which limits its adaptability. 3. The random exploration in the RL training process leads to uncertainty in the rate of convergence