Identifying Axiomatic Mathematical Transformation Steps using Tree-Structured Pointer Networks
Authors: Sebastian Wankerl
Jan Pfister
Andrzej Dulny
Gerhard Götz
Andreas Hotho
Venue: Accepted by TMLR ()

Abstract: The classification of mathematical relations has become a new area of research in deep
learning. A major focus lies on determining mathematical equivalence. While previous
work has simply approached the task as a binary classification without providing further
insight into the underlying decision, we aim to iteratively find a sequence of necessary steps
to transform a mathematical expression into an arbitrary equivalent form. Each step in
this sequence is specified by an axiom together with its position of application. We denote
this task as Stepwise Equation Transformation Identification (SETI) task. To solve the
task efficiently, we further propose TreePointerNet, a novel architecture which exploits the
inherent tree structure of mathematical equations and consists of three key building blocks:
(i) a transformer model tailored to work on hierarchically tree-structured equations, making
use of (ii) a copy-pointer mechanism to extract the exact location of a transformation in the
tree and finally (iii) custom embeddings that map distinguishable occurrences of the same
token type to a common embedding. In addition, we introduce new datasets of equations for
the SETI task. We benchmark our model against various baselines and perform an ablation
study to quantify the influence of our custom embeddings and the copy-pointer component.
Furthermore, we test the robustness of our model on data of unseen complexity. Our results
clearly show that incorporating the hierarchical structure, embeddings and copy-pointer into
a single model is highly beneficial for solving the SETI task

of research in deep Published in Transactions on Machine Learning Research (01/2025)
Identifying Axiomatic Mathematical Transformation Steps using Tree-Structured Pointer Networks
Sebastian Wankerl
Center for Artificial Intelligence and Data Science (CAIDAS) Julius-Maziin Maziss--aiimmi-as---- Universität Würzburg, Germany
Boden- Würltemberg Cooperative State University Mosbach, Germany
Jan Pfister
Center for Artificial Intelligence and Data Science (CAIDAS)
Julius-Marinn Maziis-Mzziiiinns---a Universitää Würzburg, Germany
Andrzej Dulny
Center for Artificial Intelligence and Data Science (CAIDAS) Muliu--Mciimaan--nnniniee Universsttt. Würzbbrg, Germang
Gerhard Götz
gertarl.gess@@oocccc..h
Baden- Würllembery Cooperative Stale University Musbach, Germany
Andreas Hotho
Center for Artificial Intelligence and Data Science (CAIDAS) Mulius-Mazimiiee Universität Würzburg, Germany
Reviewed on OpenReview: https: // pttp//vee..rr.eeeeemmee/ee net/ forum? id= gLQ801ewup
Abstract
The classification of mathematical relations has become learning. A major focus lies on determining mathematical equivalence. While previous work has simply approached the task as binary classification without providing further insight into the underlying decision, we aim to iteratively find sequence of necessary steps to transform a mathematical expression into an arbitrary equivalent form. Each step in this sequence is specified by an axiom logether with its position of application. We denote this task as Stepwise Equation Transformation Identification (SETI) task. To solve the task efficiently, inherent tree structure of mathematical equations and consists of three key building blocks: (i) transformer model tailored to work on hierarchically tree-structured equations, making use of (ii) copy-pointer mechanism to extract the exact location of a transformation in the tree and finally (iii) custom embeddings that map distinguishable occurrences the same token type to common embedding. In addition, we introduce new datasets of equations for the SETT task. We benchmark our model against varions baselines and perform an ablation study to quantify the influence of our custom embeddings and the copy-pointer component. Furthermore, test the robustness of our model on data of unseen complexity. Our results clearly show that incorporating the hierarchical structure, embeddings and copy-pointer into a single model is highly beneficial for solving the SETI task.
further propose Preetointe,,,t, novel architecture which exploits the

---

Published in Transactions on Machine Learning Research (01/2025)
1 Introduction
Deep learning has surpassed traditional machinc-learning methods in multiple domains such as computer vision, natural-languaag processing, speech recognition and many more (Wang el al. 2020 González-Carvaip
& Merchhá 2021: O'Mahony et al. 2020 Also the field of symbolic mathematics has seen deep
learning models applied to various objectives (Luu et al.
2023), such as recognizing equivalent mathematical
Mali et al. 2021 Wankerl et al. 2023] or performing symbolic compulations for, e-g., integration (Lample & Charton 2019), linear algebra (Charton, 2022) and solving recurrent equations [D'Ascoli et al. 2022). In addition, neural networks proved to be useful
expressions (Wankerl al. 2021 Arabshahi et al. 2019
for generating step-by-step solutions to word problems (i.c. mathematical problems where the problem is
partly formulated using natural language) and mathematical proofs [Azerbayev et al. 2024 Yu et al. 2024
Hendryeks et al., 2021).
The above-mentioned tasks of recognizing two equivalent mathematical expressions and of generating step- by-step solutions provide an inspiration for our work. Precisely, we want to find a sequence of verifiable axiomatic steps that is needed to transform mathematical expression into another given equivalent form. We denote this task as Stepwise Equation Transformation Identification (SETI) task.
1.1 Introduction to the SETI Task
As an introductory example to the SETI task, consider the following pair of equivalent expressions: x (y-x) and z²y. For humans, it is easy to that the right expression can be derived from the left by first applying the commutative law on the multiplication in the parentheses, yielding x ex-xxy), then applying the associative law, yielding (x- -)- and finally rewriting x x as x², yielding .².y. y.-
A sequence of such steps describes the transformation from one expression into the other. We observe that the SETI task can be solved iteratively, by only predicting the next step towards transforming one equation into another. This method (be it a human, a rule-based system or a neural network) is then applied on the input equation and its intermediate predictions until the target equation is reached. Thus, for a given pair of mathematical expressions we want to predict which axiom applied at which position of the expression tree yields a suitable step towards transforming the left- hand equation into the right-hand one.
Figure 1 provides a visualization of this setting. Note that the nodes of the left subtree (orange) are enumerated by consecutive index. However, its only purpose is to make the nodes distinguishable for any method to extract the position at which the axiom was applied. It does not denote a different meaning regarding the mathematical interpretation, i.e. both 2g and xı signify the same variable x, but at different position.
Given a large enough equation and set of axioms, the mumber of applicable axioms at each step rapidly increases. Empirically, deep learning proved to be beneficial for a wide range of problems with large search spaces (Silver et al. 2016 Vinyals et al. 2015). Hence, we think that noural networks can be a usefuul hcuristic to select a set reasonable transformation steps. However, there are multiple hurdles when applying existing models to our task.
1.2 Limitations of Previous Research
So far, previous research has treated equivalence as a mere classification task without generating intermediate steps (Wankerl al., 2021; Arabshahi et al.] 2019: Mali et al.] 2022 Wankerl et al., 2023). However, the classification task does not yield an explanation for the equivalence and depending on the complexity of the expression and the knowledge of the user, it might not be easy to see. Moreover, in remotely related settings like doep-learning-bane based step-by-step solutions to word problems, only the final answer is checked, but not
the predicted intermediate steps (Azerbayev et al.) 2024 Yu et al. 2024).
In this work, we want to provide an approach for overcoming these limitations. Besides, the latter task is generally approached using large language models (LLMs) which are trained for multiple purposes and consequently consist of billions of trainable parameters, making them computationally demanding for both fine-tuning and inference.
2

---

Published in Transactions on Machine Learning Research (01/2025)
Inpat
Construct Expression Tree
Prodict Transformation for One Stop
Model
Outpur
x- (x) yx
Tree Www.PonnerNeee Net.
Classifiattin Thekk identily suitable
Pointing mnk: identify mode to apply the
Tre-SStruttred Attendion
Pointer Geverntor
Netwwar's
P-P - a-a
Apply Preiction the
x- (x-y) exlry
TresPointer Net
(f - (a (
Figure 1: Overview of our SETI task: Given imput equation, the left side should be transformed into the right side by repeated application of TreePointerNet. At each step, the net predicts a suilable axiom and the root node of the subtree to apply the axiom. Note that for disambiguation of the mathematical tokens every node in the left subtree is combined with its incremental count to make it unique.
1.3 Contributions
To solve the SETT task more efficiently than LLMs, we introduce a new neural network architecture, Tree- PointerNet, designed specifically to solve both parts of our SETI task. It combines a tree-structured trans- former encoder (Nguyen et al., 2020) with a pointer augmented decoder (Vinyals al.j (2015) and a custom many-to-one embedding layer that captures the semantic of the tokens independent. of the position while making them still distinguishable for the pointer decoder.
Furthermore, we create and release new dataset, consisting of equations and axiomatic steps needed to show their equivalence. This is necessary since to the best of our knowledge no well-suited dataset for the SETI task cxists in litcrature. Our dataset contains a wide range of mathematical functions, such as polynomials, logarithms, exponentiation, and trigonometric functions. addition, it contains equations of varying complexity measured by the depth of the parse tree and the number of axiomatic steps needed show the equivalence.
Our contributions can be summarized as follows: (i) We introduce the SETI task in deep learning on symbolic mathematics aiming at explaining the relation of two equivalent mathematical expressions using axiomatic steps (ii) We design & novel neural network architecture to solve the proposed task efficiently and compare to various neural networks known from literature (iii) We create a well-suited dataset for this task with varying complexity. The code for our experiments available https: thub. wwtp:otooooo/c...e.ee Uniwe// seematic-stees-treeppe
2 TreePointerNet Architecture
Our goal to identify mathematical transformations as depicted fig. [ and described in section 1.1 To inherently and efficiently capture the structure of mathematical expressions, we base our TreePointerNet on the troc-structured transformer model by Nguyen al. (2020) (further denoted as Trenfrrnsfomerr). It consists of an encoder which in contrast. to a standard transformer (Vaswani al.] (017)) working on flat, explicitly encode the hierarchy of mathematical
iee sequential input, allows to input tree. Thus, operators and the link between operators and operands. Morcover, it is also equipped with a decoder that we can use to predict the applied axiom.
To identify the particular root node of the subtree that should be transformed, we propose a sree-copy-pointee mochanism. Hence, the model can be trained directly extract any arbitrary node occurring the input tree. A model without a copy-pointer component would be restricted predicting nodes observed during training. We merge the seee-copy-pointer distribution with the decoder output of the Tree Transformer using
3

---

Published in Transactions on Machine Learning Research (01/2025)
gating layer. It learns to switch between copying nodes from the imput tree and predicting axioms at each decoding step.
The first step of the model is to project the input tokens to an embedding. Generally, an individual embedding is learned for every distinct token. However, we represent each occurrence of a mathematical token as a token on its own. Consequently, this would lead different and independently learned embeddings, although they actually represent the same mathematical entity. Thus, we incorporate custom embedding layer which allows identical mathematical tokens to share one embedding over all their occurrences while still being unabigguuusl identifiable.
In the rest of this section, we describe our TreePointerNet architecture in detail as we designed it for our lask. A visualization of the model is given in appendix A.1
2.1 Background: Tree Transformer
Nguyen et al. (2020) propose an extension of the transformer that incorporating tree-structured input into the attention mechanism. Prociscly, their architecture receives a tree input (encoder) and outputs sequence of tokens (decoder). In this section, give a short overview of the architecture. For more details, we refer to the original work.
x xx),,,) }, b non-terminal nodes N = ... and a Given a tree T specified as a set of leaves E = ==f ! relation R(x), mapping each node æ to the set of all nodes that are successors of in T, including æ itself.
The tree is first decomposed into two tensors corresponding to the emboddings of its a ordered leaves [I... la] [l) = L E Raxd, and b non-terminal nodes [m....nb]] N Raxd, where denotes the size of the embedding.
Following this, new tensor S E R (b+ ()xaxd is constructed from & and N as
1/
if
F1,,F(C,,
n4-1 else af R(xM)))
(1)
0 otherwise.
Each column S.y contains the hidden representations for all nodes on the path from the root of the tree to the j-th leaf 2f. Row S, contains representations of the i-th node zN in every column if it is on the path from the root node to the j-th lcaf xff, otherwise zero.
As a next step, the bottom-up cumulative average S.j ({S)) 1 (So. +... + S.,)) is calculated, assigning to Sij the average over all values below itself. Finally, branch-level embeddings n. representing each subtree rooted in non-terminal node xyY are obtained, resulting in tensor N = [ mn. E R Each n, is calculated as the weighted average of the i-th row over all non-zero columns using V as
n; = V(S w); = 1j S.s L CDR(EN)
(2)
Above described process called hierarchical accumulation. Furthermore, since N is a tensor consisting of all possible subtree embeddings, able capture the global tree structure but not the local neighborhood of nodes and their relative positioning. To alleviate this limitation, hierarchical positional embeddings E are added to each non-terminal node z:V. They can be interpreted embeddings on the scquential input of a transformer. They capture the absolute position of each non- terminal node x through a concatenation of two separate embeddings: The first embedding captures the number of nodes in the subtree spanned by NYY. The second embedding captures the number of nodes on the same level as IN lying to the left it.
generalization of the positional
Analogously the transformer, the model consists of an encoder and a decoder component (see fig. left and middle). Instead of the self-attention on sequential input, the encoder calculates the attention wetween
4

---

Published in Transactions on Machine Learning Research (01/2025)
all possible pairings of the leaves' L and non-terminal nodes' N representations. Then, the decoder calculates the cross-attention between both. Given decoder-side query Q E Rixd and the leaf and node embeddings L and N, the affinity scores Aqc E gxx and AQN E 12tb are computed as follows:
Agc (QW))
(3)
AQN (QWa)(WW<)//va
(4)
where Wq and WK denote the trainable key and query weights.
Then, to obtain the value representation of the leaves L, they are multiplied with the weights LWr. The non-terminal nodes are encoded as described above: N'' = V(u(F(CWv, NWW,, R) + E), Lue) where Lue is the learnable weight for the sum in eq. 2 Thereby the embedding of the terminating leaf for each path influences its weight in the sum.
The final cross-attention then given as Ag = softmax([Aqq' Agcl) Agd [Wv]] The subsequent FFN and Add&:Norm layers work analogously to those in regular transformer model (Vaswani al. 2017).
2.2 Pointer Network for Copying Parts of the Tree
To identify the position where an axiom was applied, we construct a model that incorporates a pointing mechanism. Pointer networks have the benefit that parts of the imput can be directly copied to the output. This not possible using a regular transformer decoder architecture like it used the Tree Transformer which is limited to outputting tokens it has seen during training. It beneficial for our task since it allows the model to output the node where the transformation should be applied directly, instead of having to reconstruct it from its vocabulary. Given the nature of our input data and our task, the pointer component must be equally able to point to the leaf as well as the non-terminal nodes.
In the standard transformer architecture, the queries of the attention heads of the encoder-decoder attention layers are passed through from the previous decoder layer, whereas the keys and values are obtained from the output of the encoder. Consequently, these layers capture the importance of every token in the input with regard to each token in the output sequence (Vaswani et al. 2017 Enarvi et al.) (020).. Hence, the thereby emerging distribution can be interpreted pointers the elements of the input, i.c. those that receive a high attention are good candidates for transfer to the output.
In the Tree TrrefTranformer, the importance of each element of the tree with regard to the currently decoded token in the output sequence is read out from the cross-attention layer. It sufficient for us to obtain the alignment scores"] between the currently decoded element on the target side and the nodes in the input tree. To enable this, we concatenate the affinity scores Aqe and AQv as defined in eqs. (3) and (4) to new tensor A [Aqc; AQvl E (+x((=+)), Hence, A contains the alignment. for each element of the tree at each decoding step t.
The pointer distribution Proine (x) is then given by the sum over the alignment scores of cach occurrence of each symbol in the input. Hence, we calculate (x) such that we iterate over both the input leaves zf and non-terminal nodes xN while summing up their respective aligument scores from as
Pt Proine (x))
Σ :EE
[Aqc: AQNI
4 :|
Σ
The pointer distribution is calculated in addition to the regular distribution over the full vocabulary from the Tree Transformer's normal output. To subsequently decide if the model should copy from the input or generate new tokens, a generation probability Pyen E [o. 1] (Seo et al. 2017 is learned. In practice, to calculate Pyen for time step I. a galing layer is used which is defined as
peen a (W yr] + b),
(6)
Defined as the dot product between decoder-queries and keyoder-kess.
5

---

Published in Transactions on Machine Learning Research (01/202)
X
X
Figure 2: Visualization of our many-to-one embeddings ME. The label indicate the token of the node and the colors indicate the embedding assigned to a node. Tt can be seen that different tokens like Tn, I1, and æ share the same embedding. Thus, they do not have to be learned independently by the network but yet are distinguishable tokens.
where is the embedded decoder input, yt the decoder output at time step & and W learnable weights with bias b (compare the right part of fig. gh.
The final output distribution is then given by employing Pgen as a soft switch between the input attention distribution and the vocabulary distribution from the decoder output. Mathematically, for each token w at time step it defined as
pt P'(w) - -p'en pe Peccab (w) + (1 (1-D (oonne(
(7)
Thus, a value close to one for Pyen prefers the token predicted by the TreeTransformer decoder while a value close to zero leads to copying node from the input tree.
2.3 Many-To-One Embedding Layer
We need to be able to represent arbitrarily many occurrences of an operator, variable, or constant, for example Ti, 22, F3, in distinguishable way. However, we would embed them using standard embedding layer, this would require the model to learn multiple representations of semantically identical tokens independent of each other. Since they share their meaning, want them to also share their embedding. To this end, introduce many-to-one embeddings. They enable include every occurrence as individual symbol in the vocabulary while mapping all identical node lypes to the same shared embedding as explained in the following.
Let V denote a vocabulary of size [ representing a set of tokens to. 22....,2pp and denote an input sequence consisting of lokens 81 ... 8n- In the traditional setting, an embedding layer maps all occurrences of each element s; E corresponding to the same token By E V to the same dense embedding ey E Rd where d denotes the embedding dimension.
This approach assumes all tokens E V to have unique meaning and a large overlap of tokens between the vocabulary and the input data. However, this assumption is not justified in our setting for two reasons. If all occurrences of the same symbol would be mapped to the same token, e.g. up = '*' for the multiplication, U1 = '+' for the addition, the pointer could not be used to unambiguously output the position in the tree. If
instead all occurrences would be added as individual embeddings-cet e.g. up '*o' for the first multiplication, Vi ='," for the second and on the network would need to learn representations for all occurrences of the same symbol independently.
In addition, our model should be able to generalize to trees with larger depth than used during training. For this it has to be able to properly deal with trees where the number of occurrences of a symbol is higher than seen during training. Therefore is mandatory that our input embedding supports this setting and is able to generalize to this increased symbol count as well.
Consoquently we define our many-to-one embeddings. Let M (+.*,.... sin, cos.... 0,1...., x, y. 2} de- scribe the set of mathematical entities use for constructing our input and let N be counter associated
6

---

Published in Transactions on Machine Learning Research (01/2025)
Sample Axiom
Sample Expressions
Insert into Axiom
Figure 3: Generation of the initial set of equations. First an axiom is chosen. Its variables are then substituted with sampled expressions. The colors of the nodes represent the alignment between the variables of the axiom and the sampled expression.
with each token describing its index in the imput (c.f. fig, (left subtree)). Then, each token in our input can be considered as a tuple v = (m,, ci), m. E M, a E N. Our many-to-one embodding layer ME has to fulfill the following properties: Given two tokens va (Mai Ca); V/ = (ma. (6), Mas Vb V, we want MF((2)) = MF())) e ma = mp. Hence
VieN ME(UK) ME((ME: i)) W h(mm))
with h h(m)) being the one-hot vector associated with my E M and W ggl]xdd is trainable weight matrix. A Visualization of the many-to-one embeddings is given in fig.
3 Data Set Generation
To generale data for our SETT (ask, we exlend the generator introduced by Wankerl al. (2023). They generate pairs of mathematical expressions with several different types of relations, such as pairs of equal expressions, pairs where one expression is the derivative of the other, or expressions which have a constant offset. For our task, we only use and modify the part of the generator which creates pairs of equal expressions (equations). Every equation can be built of up to three free variables It U: E R++ the integer constants -4...4, 4, the real constants π and e, the binary operators +, -, ", and the functions In, sin, cos, tan.
To obtain the equations, the generator works in two steps. It starts with set of 112 axioms covering mathematical subjects like polynomials, exponentiation, logarithms, and trigonometry. An axiom can be considered as elementary mathematical rewrite rule. Exemplary axioms are + 0 - a, + B B + (a+ + () + + a (8 +7), or In(a- B) + In(a)) + In(3). Here, the arrow indicates the direction of rewriting, e.g. a + can be rewritten as a and In(a (3) can be rewritten as In(a) + In))..
Generate Start Equations As a first step, the generator creates a set increasingly complex expressions by iteratively substituting already obtained expressions into the free variables of randomly sampled axioms. For example, assuming that the generator already created the expressions (x+x) and (x- (r-v), it can substitute them into the commutative law (a a)) yielding (xr +x) (x v) = (x w) (xr + I)). This process is depicted in fig.
Generate Axiomatic Transformations Although the previous step is helpful to create a basic set of mathematical expressions, the data cannot be used for our SETT task itself since the transformations are not of an axiomatic nature. Therefore, as a next step, the expressions are modified according to some randomly selected matching axiom. In this step, the generator can substitute variables with parts of the expressions. For example, the generator can match the associative law a - (3- + (a (3-3)- -3 y with the expression (x x)) (x y) yielding ((x + x) x) y- Then, the modified expression and its original form are added a sample to the dataset: (x)+ 1 y (r+ (x-y). This process is visualized in fig, (upper part).
While the generator by Wankerl et al. 2023) applics this step on a random side of each equation for an undefined number of times, we instead keep track of the number substitutions already performed and
2The full list of axioms is given in appendix

---

Published in Transactions on Machine Learning Research (01/2025)
LHS of Sampled Equation
Unify with Axiom
Transform Expresion
Build Equation
Step
Axiom
Step 2
Axiom
Figure 4: Visualization of generating an equation with a distance of two steps. First, the Ihs sampled equation is extracted, unified with random matching axiom and transformed. The original expression and its transformed form then build new equation. By repeating this step, a second expression is generated. The rhs of the initial equation is kept and combined with the second expression to construct that is two axiomatic sleps apart. The colors of the nodes visualize the aligument between the expression and the variables of the axiom. The dashed red node corresponds to the root node of the modified subtree.
equation
modify the left side of each expression only. For example, we record that ((x + +x) . y x) (x y) is derived in one step. Subscquently, we can apply + a on the expression, yiclding an equation that is transformed in two sleps: (((x + x)) + 0) x) y (x xx) (x y). If axiom matches the expression at multiple positions, random node is selected as the root node for applying the axiom. This process is visualized in fig. (lower part).
The generator as described so far might generate redundant samples where an axiomatic transformation is immediately reversed in the next step. Above example might be reversed to (x++)) 2) y (x (z) (x) y) by applying the reverse axiom from before (a + - a). To prevent the generator from creating such loops, we forbid the usage of axioms which would reverse the previously generated expression, Furthermore,
ensure more uniform distribution of the axioms, count how many times each matching axiom has been used in the pairs generated far, and use the inverse of that count probabilities for sampling.
Check Validity of Transformations The mere application of axioms described so far can create mathe- matically invalid expressions. For example, ( (m/)))> could be generated, but since In(1) =0 0, this expression is undefined. To avoid the inclusion of such samples in the datasct, all expressions are chocked using Sympy (Meurer et al., 2017). If variables appear in the expression, they are assumed to be positive real numbers. All expressions that Sympy evaluates to NaN or infinity are discarded and thus do not appear in the dataset.
Make all Tokens Uniquely Identifiable To make the occurrences of the mathematical tokens inside an expression uniquely identifiable, we enumerate the tokens the left side of each equation by adding a counter to each type of token. Above example is therefore represented (((x) +132) +o00) 20) '30 (x++r) (x.y).
8

---

Published in Transactions on Machine Learning Research (01/2025)
Build Samples for the SETI task Every sample can be described as 3-tuple (Ihs; rhs,, dss as,Pi). Here, Ths; rhs; denotes mathematical equation and dg denotes axiom that, when applied at position pi: describes a step for transforming lhs; towards rhs, The position p; matches the root node of the subtree inside the parse tree of lhs; that should be modified when applying the axiom a; (c.f. fig. D However, every sample describes just one possible step and not the whole chain of transformations between the left. and the right side. Above example thus corresponds to the following tuple: ((((2) +1 22) 0o) 1 zo) yo = (x (x) (x y), a ++0 a, +o).
We create about 8,5 million samples for training, 400,000 samples for validation and 14,000 samples for testing. In the training set, we set the maximum distance between two expressions to five, meaning that the left side of all equations can be transformed into the corresponding right side in at most five steps. Moreover, in our training set we only include equations whose parse trees have a maximum depth of 7. Detailed statistics of the data can be found in appendix B
4 Experiments
In this section, we describe our experimental setup including baselines for comparison, how the target sequence is structured, and our used hyperparameters.
4.1 Baselines
We employ three additional baselines: a transformer (Vaswani et al., 2017 a pointer network by Fnarvi et al. (2020) to which we refer as SeqPointer and a recurrent seq2seq model that makes use of an LSTM [Hochreiter & Schmidhuber 1997] encoder and decoder with Bahdanau attention (Bahdanau et al. 2015). We add the last model to allow ancillary comparison with a recurrent neural network architecture besides all the other transformer- based models. For brevity, we refer to this model as LSTM.
All three models receive the prefix representation of each input equation. We experiment with different tokenization strategies. As a first variant,, we represent the mathematical symbol and its index as one token. For example, the equation In(1)/1 = 0 is tokenized to [=, /_0, 1n_0, 1_0, 1_1, 0].
Although this tokenization is equal to the tokenization for TreePointerNet, it has the disadvantage that the tokens corresponding to high indices appear much less frequent in the dataset. To alleviate this problem, we also experimented with a second tokenization strategy, namely representing the index as a separate token. Above example would then be represented as [=, /, 0, In, O, 1, 0, 1, 1, O]. However, in our experiments it yielded inconclusive results compared to the first tokenization. Since no tokenization is superior compared to the other, we stick to the results obtained with the first tokenization for our experiments because it cquals the tokenization for TrecPointerNet. We present the results obtained with the second tokenization in appendix F
The prefix notation has the advantage that it is more compact than the human-readable infix notation since it does not require parentheses. It has also been used by various previous research employing sequential
models for symbolic mathematics (Lample & Charton 2019 D' A'coolli et al. 2022 Wankerl et. al., 2023
We also experimented with LLMs as baseline for the SETI task. However, our experiments it was not solvable using the tested models and hence we do not include it as an additional baseline in our experiments. Details about our experiments with LLMs can be found in appendix F
4.2 Modeling the Target Sequence
All target sequences for all models consists of two tokens: class token for the applied axiom and the root node of the modified subtree in the expression tree. As commonly done when training seq2seq models, we explicitly terminate each sequence with an snd-ee--equence token. An exemplary target sequence might hence be: ax _x_ y_2, EOS.
9

---

Published in Transactions on Machine Learning Research (01/2025)
4.3 Evaluation Strategy
The input to the models is an equation that either given as a parse tree (c.f. lig. 1 or as a sequence as described in section 1.1 The output of the models are two tokens (axiom and position) as described in section 1.2 To find sequence of axiomatic steps for transforming the left side of an equation into the right side, we repeat below procedure for up to = 5 step. We use greedy decoding, ic. in every step, chose the most likely pair of axiom and position.
Let 0; denote the predicted axiom and PJ denote the predicted position within equation lhs; = rhs, We use a custom rule-based algorithm to verify if å can be applied at node p.. Precisely, we check if p. exists within Ihs; and if spans a subtree where â matches. If this not the case, we terminate the process and count the equation as unsolved. Otherwise, we construct lhs by applying the axiom in lhs, Then, we check if lhs] and rhi; are syntactically identical. If this is the case we count the equation as solved. Otherwise, we feed it into the neural network again, obtaining the next transformation step. If we did not obtain syntactically equal equation after n steps, it also counts as unsolved.
4.4 Ablation Study and Generalization Ability
We perform an ablation study on our TreePointerNet. Precisely, we remove the pointer-copy mechanism and the many-to-one embedding, respectively, quantify their benefit for the task at hand. Thus, we retrain the model without one of these components while leaving all other hyperparameters unchanged rule out their potential influence on the obtained results.
In addition, we test the model on deeper trees (depth 8-12) than seen during training (depth up to 7). Furthermore, we test on equations that require more steps to transform (6 9 steps) than those seen during training (1-3 steps). In doing so, we explicitly test how well our model generalizes to trees which lic outside the training distribution and represent mathematical expressions of higher compositionality.
4.5 Implementation and Training
All our experiments are implemented using the Fairseq (Ott et al. 2019) toolkit. We train all models using the Adam optimizer (Kingma & Ba 2015) on batches of 16 samples for up to 100 epochs or until the loss stagnates or deteriorates over a period of 10 epochs. We ran a hyper-parameter search using the Optuna framework (Akiba et al.) 2019), exploring 100 configurations each for our model and the baselines. Our search space included the number of layers in the encoder and decoder, the number of attention heads used per layer, the number of attention heads used for pointing (where applicable), the size of embeddings and the hidden representations in the encoder and decoder, the dropout rates and the learning rate of the optimizer. The exact parameters and scarch spaces are given in appendix C
Due to the hyperparameter optimization, our NeeePointer has only 2,649,901 trainable parameters. Thus, this model requires a much lower number of parameters for achieving optimal results compared to SeqPointer and transformer (22,766 222,768,646 6444 and LSTM (6,496,256). Hence, it is particularly efficient with regard to the number of parameters, as very well tailored to our SETT task. On a modern multi-core CPU, evaluating TreePointerNet averages to 21 equations per second.
5 Results and Analysis
In this section we present and discuss our obtained results. Figures 5 to S show the average accuracy (# standard deviation over five model runs, each time initialized with different random soed), i.c. the fraction of correctly transformed equations grouped by the number of required transformation steps. The detailed numeric results given in appendix D
10

---

Published in Transactions on Machine Learning Research (01/2025)
00%
80%
Percerbage of Seived Exuation
80%
70%
50%
Indefintarret (ours)
60%
40%
Percentage
50%
Teeeoe (ours)
20%
SooPaineer
40%
Trensoomer LSTM
Figure 5: Results for all Models on Equations of up to 5 Required Transformation Steps
Figure 6: Ablation Study on TreePPinter.Net
5.1 Model Performance
As can be scen in fig. 5 TreePointerNet outperforms all basclines by large extent, achieving an overall accuracy of 74.09%, whereas the SeqPointer only achieves 59.47% and the transformer even weaker with 59.28%. The LSTM achieves an accuracy 59.47%. All models prove to be stable (stdev below 2%).
Considering the equations where only one transformation step is required, it is noticeable that the per- formance of all models very similar. Here, TreePointerNet accuracy of 88.43%, and SeqPointer (88.41%). Only the LSTM (85.49%) performs slightly worse. It is therefore standing to reason that, given adequate number of trainable parameters, all tested neural archi- tectures can comparably recognize elementary axiomatic differences between two mathematical expressions.
par with transformer, both reaching
When analyzing the equations requiring multiple transformation steps, a clear decline in the accuracy of all models can be observed with each additional transformation step. Yet, the TreePointerNet outperforms the baselines to large extend since it looses much less accuracy than all baselines. For example, when going from one to two transformation steps, the best baseline (SeqPointer) already looses 17.58 percentage points, whereas TroePointerNet only looses 6.41 percentage points. Thus, TreePointerNet preserves accuracy of 82.02% on theses equations, whereas SegPointer reaches only 70.83%. This trend continues. On the equalions requiring five steps, Poeeeeinte achieves a total accuracy of 59.05 .9.0%% which is only 29.38 percentage points below the accuracy on equations of one step. However, the other models loose on average 50.51 percentage points, with the LSTM being the most robust model achieving an accuracy 37.4%.
Predicting transformation steps for equations which are more than one step apart require the networks to identify more complex mathematical patterns than the mere axiomatic differences that are equally well recognized by all models. Understandably, models which capture those patterns with less certainty quickly loose more overall accuracy with each step, since each step corresponds to an independent classification.
Summarizing above results, our TreePointerNet model clearly outperforms the baselines and proves to be stable and efficient. At the same time it is noticeably more robust to changes in the number of required transformation steps where can generalize with significantly smaller performance loss. For illustrative purposes, we present few exemplary equations as they were transformed by TreePointerNet in appendix G
5.2 Ablation Study
We further analyze the influence of the many-to-one embedding and the copy-pointer on the TreePointeNNet's performance by removing one of these components a time. The results are presented fig, 6
Removing the many-to-one embeddings, the model's performance drops to an average accuracy of 57. 57.31% making it comparable to the performance of the baseline models. In particular, the model reveals the same
11

---

Published in Transactions on Machine Learning Research (01/2025)
60%
50%
35%
Percerbyye of Savvdd Equatton
50%
50%
80%
Ieaefinderfee (eurs)
GegPainter
Transfomar
60%
LSTH
35%
20%
SooPainerr
TeePo (ours)
mhendoomr
30%
LSTM
10%
Figure 7: Robustness Study for all Models on Equations of Deeper Parse Trees
Figure 8: Robustness Study for all Models on Equations of up to 9 Transformation Steps
decline in accuracy on the equations with more transformation steps. This supports our motivation from section 2.3 for introducing the many-to-one embeddings.
First, without these embeddings the model now has to learn single representation for each occurrence of mathematical token instead of sharing them. This likely decreases the performance as there is less training data available to learn the mathematical meaning of each symbol. Second, several symbols are potentially mapped to an untrained embedding since they did not occur in the training data. Hence, the model can not understand their semantics which lcads to a stronger decline in prediction performance.
Removing the pointer component causes the performance of the model to drop to an average accuracy of 1.53%. This behavior can be understood when considering the many-to-one embeddings. Since all occur- rences of the same symbol are mapped to the same embodding, the generative part of the network can longer disting between them. Hence the model can not identify the position where the axiom should be applied.
The pointer component which directly extracts them from the alignment scores between the output tokens and the nodes (cf. section 2.2 necessary benefit from using the many-to-one embeddings. When removing both components at the same time, the average accuracy drops to 57.14%, marginally worse than when only removing the many-to-one embeddings. We conclude that having pointer component is slightly beneficial for our task, even without our many-to-one embeddings.
5.3
Robustness Data of Higher Complexity
We perform two further experiments to evaluate the robustness of our model data of unseen complexity. First, increase the complexity of the equations themselves. In line with previous research (Arabshahi
et al. 2019 Mali et al. 2021 Wankerl et al.] 2023). we use the depth of the parse tree as the measure of complexity. The results are presented in fig. 7 All models clearly loose accuracy on this data, but on average TreePointerNet (42.4%%) still outperforms all baselines (27.18% 29.63%). Beyond that, this experiment reveals a similar overall trend as can be observed on the shallow trees in that the performance of all models declines with the number of transformation steps. Yet, in contrary outperforms the baselines on the equations of one transformation step by least 7.99 percentage points. Thus, NreePointerNet identifies the mathematical patterns in data of unseen complexity more robustly than the standard models.
the latter, TrecPointerNet already
Second, we evaluate the models on equations which require more transformation steps than seen during training. The results are shown in fig. g and seamlessly connect to the results presented in fig. 5 Therefore, we observe contimuation of the previous trend, with TreoPointerNet still outperforming the basclines. On our tested maximum of nine required transformation steps, TreePointerNet still reaches an accuracy of 49.149 14% which is at least 20.4 percentage points above the baselines.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Accuracy by Depth Equation Tree and Level of Application of Axiom
Level of Transformation Root
Depth of Tree
Average
1
2
4
5
6
97.0%
90.9%
94.0%
98.3%
96.4%
92.2%
95.6%
98.4%
91.8%
91.8%
90.5%
93.1%
98.6%
91.0%
82.7%
84.1%
84.3%
88.1%
99.5%
88.4%
81.5%
76.5%
78.1%
77.7%
83.6%
Table 2: Most Froquently Misclassified Axioms and their Counterparts
Axiom
Confused with
(a 3) + (a²). (87)
(a-B)) .7 (B.
cos(a) coS( az)
0.000
a a.9- a/B
(a.7)/B (B/8))
3] + a ( 77)
(a. 77 (B..BB.. (B.7)
a 1-8+a+(-1)
cos( - a) + +-1 +
a 0.3-3.a
a +
5.4 Interpreting the Model's Predictions
To make the results of model more illustrative, we want to discuss prominent patterns in our model's predictions. Precisely, want to identify axioms which are commonly confused and analyze the influence of the depth of the whole equation tree as well as the level of the root node where the axiom has to be applicd. However, due to the urt of the SETT lask, unambiguous labels only exist. for the subset of equations that can be derived in one step. Thus, the analysis in this section is performed on this subset.
We start with evaluating the performance of TrecPointerNet with regard to the depth of the tree and the level of the root node of the applied transformation. The results are presented table We find an overall decline in the accuracy the deeper the input tree. On shallow trees with depth of 3-5, the transformations are identified with accuracy of at least 93.1%. The accuracy drops to 88.1% trees of depth 6 and 83.6% on trees of depth 7. However, this result seems natural since deeper trees generally correspond to longer and more complex equations making the correct transformation harder to identify.
Considering the level of the root node the transformation within the equation, by lendency we observe the accuracy to decline the deeper the level of the root node within the tree. This true independent of the overall depth of the tree. Descriptively, the deeper the level of the transformation the closer it to the leaf nodes. Thus, transformation applied close to the leaf nodes corresponds to more local change within the equation. While transformations which occur directly under the root node (level 1) are recognized with an accuracy of at least 97%, those which are on the deepest possible level are only recognized with an accuracy between 92.2% (tree of depth 3) and 77.7% (trees of depth 7). So overall the deeper, i.e., more local and hence smaller, the change is, the worse is recognized by the model.
Finally, we analyze the model's ability recognize the applicd axiom. We found that from the 112 axioms we use in our data set, the vast. majority is classified correctly in nearly all cases. Overall, there are only six axioms where the model has larger difficulties and which could only be identified correctly in 10%-90% of all cases. They are listed in table together with the axiom they are most likely confused with in descending order, ie., the topmost axiom is the one which is most likely misclassified. Keep in mind that only the left side of the axioms are matched with the expression tree.
The misclassified axioms reveal the prominent patterns that they always match very similar trees like the axioms they are confused with. The axioms cos(a) - cos(-1 a) that confused with 0 a and -1-cc0((a) cos(a) are structurally identical apart from the
(3) that is confused with cos(TT a) -1 -1-
13

---

Published in Transactions on Machine Learning Research (01/2025)
cos function (corresponding to unary node). All other axioms correspond to structurally identical trees which only differ in the operators or constants. For example a- - 9 and a+ - 3 only differ in the binary operator node (multiplication vs addition). Similarly, (a- + (a7)) and (a- a a a only differ in the rightmost binary operator node (power vs multiplication). We therefore conclude that TreePointerNet has minor difficulties when it comes to distinguishing structurally very similar trees and trees which essentially only differ in the specific values of the nodes.
6 Related Work
In this work, we introduce the SETT task. To the best of our knowledge, there exists no closcly related work that addresses this kind of problem. Yet, there is loosely connected work we want to discuss in this section.
Pointer Networks Pointer networks describe a class of neural network architectures which learn to predict the conditional probability over positions or indices of the input sequence instead of tokens from e-g. fixed vocabulary. They were originally introduced to approximate NP-hard geomctrical problems by Vinyals al. (015)). The approach was adopted for various tasks where the extraction of parts of the input is required helpful, e.g. summarization (Gu et al.)
2016 Miao & Blunsom 2016 See et al. 2017 Enarvi et al. 2020
Gulcchre et al.] 2016), sentiment analysis Yaan et al. 2021 Pfister ct al. 2022 relation extraction Nayak
& Ng 2020) and signal analysis (Moussa el. al.
2023
Most above works introduce pure pointer networks. Technically, these networks use an attention distri- bution to identify relevant sections their input. Therefore, they can be used to copy or indicate parts of the input but they cannot generate new tokens. Thus, they are not useful for our research as they cannot generate token for indicating the applied axiom.
However, there also exist pointer-generator networks (See et al. 2017 Enarvi et al. 2020 These models are able to copy tokens from the input sequence to the output sequence, along with generating new tokens. They are implemented by augmenting the decoder of a regular seq2seq model with a pointer network and a gating layer that switches between the two components. While SSee et al. (2017) implement their network based on an LSTM encoder-decoder architecture, Enarvi et al. (2020) augments transformer encoder-decoder architecture with a pointer network. Our research builds on this idea and introduces a model that is not limited to pointing on sequences but can process treo-structured input.
Deep Learning on Tree-Structured Input Various neural architectures have been designed to make use of tree-structured input. [Socher et al. (2013) use neural networks to process trees recursively in a bottom- up manner. Starting from the leave nodes, for each node a representation calculated based on the value of its children. This approach is roughly comparable to RNNs on sequences where the hidden state is updated based on previous elements of the sequence. Taii et al. (2015) build on this idea and incorporate gating mechanisms into the recursive architecture, comparable an LSTM for sequences. Arabshahi et al. (2019) further extend the model and add distinct memory to cach node in the form of a differentiable stack. All these models require recursive function calls and are therefore difficult to parallelize for training on GPUs.
Alternative to the recursive models, Mou et al. (2016) developed a convolutional architecture for trees. Here, learnable filters merge each node with its children, allowing the network to learn a representation of the tree based on local relationships. Furthermore, |aai et al. (2021) incorporate trees into pretrained transformer encoders by masking the self-attention accordance with the structure of the input trec.
However, all above models were designed for classification tasks and are unable to generate output sequences. Thus, in our setting they could only be used to predict an axiom a position, but not both. Furthermore, apart from Bai et al. (2021), none of the models incorporate any form of attention mechanism which is needed to calculate a pointer distribution. Hence, in our setting they could not directly extract the position from the input tree. To the best of our knowledge, the Tree Trenffrrform (Nguyen et al. 2020 is the only architecture that can map tree to an arbitrary sequence and incorporates an attention mechanism. Thus, we decided use it as the base architecture for our TreePointerNet and extend it with the features we need for our SETT task, namely a pointer component. and our custom many-to-one embeddings.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Deep Learning for Symbolic Mathematics Recent research explored the capabilities of deep learning architectures on various tasks from the realm of symbolic. mathematics. Arabshahi et al. (2019) aimed to explore if neural networks can learn the equivalence of mathematical expressions. To this end, they introduced a syuthetically generated data set (Arabshahi et al.) 2018) for mathematical equations from the field of elementary algebra and trigonometry. [all el al. (2021) improved the stale on this dataset. by developing more complex, higher-order models. Wankerl et al. 20021 2023) introduced new datasots that overcame various biases (Davis 2021) and added more mathematical relations and neural architectures.
Apart from classifying mathematical relations, seq2seq models were used for various symbolic and arith- metic math problems (Saxton et al. 2018). for example integration and differential equations (Lample & Charton 2019) and linear algebra (Charton 2022). Other researchers combine natural language with formal mathematics to solve word problems (Hendrycks et al., 2021 Azerbayey et al. 2024 Yu et al.) 2024). All these models are trained end-to-end to output the solution of the given input expression. Here, only the final answer of the LLM is evaluated making possible that intermediate steps are incorrect. In addition, could be shown that transformers do not reliably generalize to numerical lasks like addition or multiplication with specific numbers (Welleck et al. 2022) and frequently make mistakes on non-axiiomaic transformations required for calculating integrals. Hence, we focus on mostly symbolic task and include specific numbers only peripherally. Furthermore, we want to output single steps where each step can be verified.
Finally, there is recent research on proving mathematical theorems (Aerrbayev et al. 2024 Song et al. 2023 by combining LLMs with external prove assistants like Lean (De Moura et al.) 2015) using a framework like Welleck 2023). Here, the LLM is given the current state of the theorem prover and its goal to propose or select next prove step, i.c. axiom. However, this differs from our approach, we use the neural network to simultaneously predict both the axiom and the position at which it is applied. Moreover, their focus lies on verifying or disproving the equality of given expression rather than finding paths to transform one expression into another equal expression. Hence, we did not integrate our research into this setting.
7 Future Applications and Extensions of our Model and Data
The model and dataset we introduced can be beneficial for various further applications and research. In this section, we provide brief outline of potential future work that could benefit from our data or model.
In an educational sctting, the model could be used to help students by providing hints and intermediate steps in interactive teaching tool. Moreover, a learning platform where students solve exercises, the model could help to identify individual knowledge gaps and enhance recommender systems in finding suitable exercises. More precisely, since the model can identify the axioms needed to solve an equation, it could also be used to identify potentially unknown axioms of students or groups of similar students. Then, students could be redirected to appropriate reading material or further exercises which advances their personal learning experience.
Furthermore, the dataset or generator could also be helpful to improve research on word problems (i.e. mathematical problems where the problem is partly formulated using natural language). So far, we input an equation to generate a step-by-sted solution, while other research focusing on word problems (c.f. section ( inputs texts. Yet, in future research both approaches could be combined to allow similarly fine-grained solutions to word problems.
Morcover, our rescarch could be extended to other domains of mathematics. The axioms we use in this work cover polynomials, logarithms, exponentiation, and trigonometry. Thus, we are mainly considering mathematics a level as it is taught in high-school or early college. Furthermore, so far our data contains only a few constants and variables are assumed to be positive real numbers. Yet this does not impose a conceptual limitation on the architecture of PreePointerNet. Future work could therefore extend the set of axioms to more advanced branches of mathernatics and use TreePointerNet on it.
In addition, the setting could also be extended to other mathematical tasks beside finding stops to show equality. For example, seq2seq models were proposed to calculate integrals (Lample & Charton 2019). outputting the final solution in one step. Therefore, one could investigate if our model can be used to
15

---

Published in Transactions on Machine Learning Research (01/2025)
generate step-by-step calculations for finding integrals or derivatives in similarly fine-grained way do it for the equivalence in this work.
8 Conclusion
In this work, we introduce the new Stepwise Equation Transformation Identification (SETI) task of fine- grained prediction lor axiomatic mathematical transformations. Given two equivalent mathematical expres- sions, the task is to iteratively predict sequence of steps for transforming one expression into the other, Each step consists of both the axiom and the position where it must be applied to transfer the first expression towards the second. To this end, we generate a new equation data set consisting of pairs of mathematical equivalent expressions represented by expression trees. We then solve the task using TreePointerNet, a new architecture combining a pointer generator network with a structured input and novel embodding strategy. We show that our model able consistently outperform even strong baselines and conclude that our network benefits from the ability to make of the inherent hierarchical structure of expression trees, the pointer component and the many-to-one embeddings.
model for tree-
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD interna- tional conference on knowledge discovery & data mining, pp. 2623 203126611 2019.
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Combining symbolic expressions and black-box function evaluations for training neural programs. In International Conference on Learning Representations, 2018.
Forough Arabshahi, Zhichu Lu, Pranay Mundra, Sameer Singh, and Animashree Anandkumar. Composi- tional generalization with tree stack memory units. arXiu preprint XrXiv:1911.015455 2019.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Al- bert Q. Jiang, Jia Deng, Stella Biderman, and Scan Wellock. Llemma: An open language model for mathematics. In The Twelfth International Conference Learning Represenlations, 2024. URL https://opennrevee nett/forum/id=4WnqRB000
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. Syntax- BBRT: Improving pre-trained transformers with synlax trees. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaly (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3011-3020, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1 2at/s/3/11/221..aa// maa. 262. URL
org/2021. monel-main.20. 262
Francois Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022.
ISSN 2835-8856. URL
Stéphane D'Ascoli, Pierre-Alexandr Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In Proceedings of the 39th International Conference Machine Learning, 2022.
Ernest Davis. A flawed dataset for symbolic equation verification. arXiv preprint arXiv:2105.11470 2021.
Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob von Raumer. The lean thoorem prover (system description). In Automated Deduction-CADR on Automated Deduction, Berlin, Germany, August 1-7, 2015, Proceedings 25, pp. 378- 378-388. Springer,
25th International Conference
2015.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Seppo Enarvi, Marilisa Amoia, Miguel Del-Agua Teba, Brian Delaney, Frank Dichl, Stefan Hahn, Kristina Harris, Liam McGrath, Yue Pan, Joel Pinto, Luca Rubini, Miguel Ruiz, Gagandeep Singh, Fabian Stem- mer, Weiyi Sun, Paul Vozila, Thomas Lin, and Ranjani Ramanurthy. Generating medical reports from patient-doctor conversations using pequence-to-sqquene models. In Proceedings of the First Workshop on Nalural Language Processing for Medical Conversations, pp. 22 30, Online, July 2020. Association for Computational Linguistics.
Santiago González-Carve Carvajal and Eduardo C. Garrido-Merchái Comparing BERT against traditional ma- chinc learning text classification. Technical report, January 2021. URL http://arxiv.org wrgg/bs//005....... 13012 arXiv:2005.138 13012 [cs, stat] lype: article.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in sequence- to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1631-1640, Berlin, Germany, August 2016. Association for Computational Linguistics.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Compulational Linguistics (Volume Long Papers), pp. 140-149, Berlin, Germany, August 2016. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8):1735 1780, 1997.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Con-
ference on Learning Representations, 2019.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning, In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp- 14631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 /tt22//222aal.//// long.817. URL attps://aclanthooogg) wwwpg/or....2233 httg/2223.acl11000m acl-long. 817
Ankur Mali, Alexander G Ororbia, Daniel Kifer, and C Lee Giles. Recognizing and verifying mathematical equations using multiplicative differential neural units. In Proceedings of the AAAI Conference on Artificial 2021.
Intelligence, volume 35, pp.
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMIT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Mullor, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3, 2017. ISSN 2376-5992.
Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 319 328, Austin, Texas, November 2016. Association for Computational Linguistics.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Thirtieth AAAI conference on artificial intelligence, 2016.
Denise Moussa, Germans Hirsch, Scbastian Wankerl, and Christian Riess. Point to the Hidden: Exposing Speech Audio Splicing via Signal Pointer Nets. In Proc. INTERSPEECH 2023, pp. 5057-5061, 2023. doi: 10.21437/ hwwperseeeehhh2223-2--...
17

---

Published in Transactions on Machine Learning Research (01/2025)
Tapas Nayak and Hwee Tou Ng. Effective modeling of encoder-decoder architecture for joint entity and relation extraction. Proceedings of the A AAAT Conference on Artificial Intelligence, 34(05):8528 8535, Apr. 2020.
Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher. Tree-structured attention with hierarchical accumulation. In International Conference on Learning Representations, 2020.
OpenAl. Learning to reason with llms, 2024.
https://openaai.com/in
[cceessd:: (2024/12/19)-
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapler of the Associalion for Compulational Linguisties (Demonstrations), pp. 48 53, 2019.
Niall O'Mahony, Sean Campbell, Anderson Carvalho, Suman Harapanahalli, Gustavo Velasco Hernandez, Lenka Krpalkova, Daniel Riordan, and Joseph Walsh. Deep Learning vs. Traditional Computer Vision. In Kohei Arai and Supriya Kapoor (eds.), Advances in Computer Vision, Advances in Intelligent Systems and Computing, pp. 128-144, Cham, 2020. Springer International Publishing. ISBN 9783030177959. doi: 10.1007/978-3-0364 1779599 10.
Jan Pfister, Sebastian Wankerl, and Andreas Hotho. SenPoi at SemEval-2022 task 10: Point me to your opinion, SenPoi. In Guy Emerson, Natalie Schluter, Gabriel Stanovaky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (ods.), Proceedings of the 16th Inter- national Workshop on Semantic Evaluation (SemEval-2022)) pp. 1313-1323, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ v1 /0t////1122222//21 s220222.emevll-1..88. URL
semeval-1.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2018.
Abigail See, Peter J. Liu, and Christopher D. Manning, Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Πya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-500 2016. URL http://www nww.re. com coom//aauur//ooraal]] (:29//7787//111//ttrtteee99 html
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631 1642, 2013.
Peiyang Song, Kaiyu Yang, and Anima Anandkunar. Towards large language models as copilots for theorem proving in lean. In The 3rd Workshop on Mathematicul Reasoning and AT al NeurIPS'29, 2023.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree- structured long short-term memory networks. Tn Proceedings of the 53rd Annual Meeling of the Association for Computational Linguisties and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1556 1566, 2015.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaisor, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Oriol Vinyals, Meire Fortunato, and Navdoep Jaitly. Pointer networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Xizhao Wang, Yanxia Zhao, and Farhad Pourpanah. Recent advances in deep learning. International Journal of Machine Learning and Cybernetics, 1144):747-70, April 2020. ISSN 1868-808X.
Sebastian Wankerl, Andrzej Dulny, Gerhard Götz, and Andreas Hotho. Learning mathematical relations us- ing deep tree models. In 2021 20th JEEE International Conference on Machine Learning and Applications (ICMLA); 2021. doi: 10. 10.1109/TCMMI ACCMLIA5253.2222..000
Sebastian Wankerl, Andrzej Dulny, Gerhard Götz, and Andreas Hotho. Can neural networks distinguish high- school level mathematical concepts? In 2023 IEEE International Conference on Data Mining (ICDM), pp. 1397-1402, 2023. doi: /L110//TCDMM65522222333000
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824 2482322--48877, 2022.
Sean Welleck. Neural theorem proving tutorial. httpp://githinn wtw.sh/w..w..eeeelllla/ttttttttt
2023.
Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: on systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8629 8637, 2022.
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. unified generative framework for aspect- based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2416 2429, 2021.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via solf-immpree arXiv preprinl ar Xiv: 12122, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevier. hwtp://www..e.ooorrrr////nonnnnn
19

---

Published in Transactions on Machine Learning Research (01/2025)
A Further Details of the TreePointerNet Architecture
A.1 Visualization of the TreePointerNet Architecture
The left and middle parts show the TreeTransfrrmrr architecture introducee in section 2.1 The equations are input as trees (bottom left.). The right part shows the pointer generator network (section 2.2 which outputs a mix of generated and copied tokens sampled from the vocabulary distribution and the pointer distribution (weighted by Pgen). The many-to-one embeddings (section 2.3) are input to both the encoder and the decoder part of the network visualized in fig. (left and middle, bottom).
Figure 9 provides a visualization of our TreePointerNet as described in section 2
vacabniery cierbrrio
EEN
Sul-Mix
Add Neen
Adc Normi
Add & Noems
EEN
juinter geretator netion
sutiue d'ttibutiion
N'
FEN
X
Adc Norm
Adbi Noen
Crue T'e Altcatous
FFX
Pym
Sdf Tixs Altenkkes
(1 Fam)
Havardioo
Mang Due Enbecalin
Mhuy Toe Enkeoling
Master Actention
Lreves
Nads
Mango TorDue Ersheidee
Doryite p (4ikeiee)))
Figure 9: Overview over our architecture. Both the transformer encoder (left) and decoder (middle) make use of the tree-structured attention. From the last decoder layer the decoder output as well as the cross- attention is fed together with the embedded input tokens to the pointer network (right).
A.2 Complexity of TreePointerNet
With regard to the computational complexity, there is no difference between a transformer and TreePoint- erNet. Let n denote the length of the input sequence (for sequential models like transformers) or the size of the input tree (for treo-based models like TrecPointerNet). The transformer requires On - steps per self-attention layer, where d is the size of the learned representations (Vaswani al. 2017). Assuming that d is set to fixed value, the complexity is dominated by the length of the sequence, and thus O(n²). This is equally true for the Tree Transformer as proven in Nguyen et al. (2020).
Our many-lo-one embeddings do not add any overhead with regard to the computational complexity. Each many-to-one embedding can be computed in O(1) using a hash-table for mapping an input token to its embedding. Since for input tokens an embodding has to be calculatod, this requires O(n) steps in total. Furthermore, the pointer distribution of TrecPointerNet merely requires summation over the already computed attention scores of all occurrences of every input token. Thus, it be computed in O(n) (see
20

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Comparison of Tranfrraasrrrm with an Equal-Sized Transformer
Number Transformation Steps
Model
Average
2
3
4
5
TreeTrransorne Transformer
87.76%==..199
68 20.47%
52.29% +0
42.67%.40.11%
34.10%440.42%
57.14%
Transformer
88.67 38.67%+t0.08%
65.50%%=0.153
49 49.05%%=0.367
39.22%%+.64%%
31.88%:0..73
54.86%
eq. 15 in section 2.2 models, namely On
Concluding, TreePointerNet has the same complexity as the above mentioned bascline
A.3 Parameter Efficiency of TreePointerNet
In section 2 we hypothesized that model explicitly using the tree structure of mathematical equalions can achieve better results with less parameters than a model which has to deduce the grammar of mathematics from sequential input. To confirm this claim, we experimented with a transformer of the same size as our TreePointerNet and compared it with the performance of the TreeTransformef Thus, the transformer in this experiment has only 2,649,901 trainable parameters, roughly 1 of the parameters of the transformer we 5
use in all other experiments (22,768,644).
Table presents the results of this experiment. Although the transformer is able learn equations which require only one transformation step with higher accuracy, proves to be less capable to generalize equa- tions which require more transformation steps. Overall, the Treefraanfrreer outperforms the transformer by 2.28 percentage points. We conclude that using tree-structured input is beneficial for our SETI task and therefore the Transfrrer functions as a suitable basis for our TrreePiiterret..
B Dataset Axioms and Statistics
B.1
Axioms
Table lists all 112 axioms we used for creating our datasets. The axioms are taken from Wankerl et al. (2023). For brevity, the axioms are listed as equivalences. However, in practice every equivalence corresponds to two axioms, one for each direction of application. For example, a/B t a p--31 represents a/3 + a f-1 and a 8-1 + a/3, where the arrow indicates the direction of application, i.c. the left side is replaced with the right side.
The only exceptions are the commutative law for the addition (a+B - a)) and the multiplication (a- 3 G a). These two axioms are not applied from right-to-left because the result would be indistinguishable from applying them left-to-right.
B.2 Dataset Statistics
Our training dataset contains around about 8.5 million samples consisting of trees having a depth up
7. The test dataset for our main experiments (cf. section 5.1 contains 13, 641 samples and comes from the same distribution the training data. In table 5 show the distribution of depth of the trees and the level of the transformation root node within the trees of our test dataset.
It is clearly visible that the number of samples in the dataset increases with the depth of the tree. This is caused by the fact that more possible equations exist the deeper the tree. Especially in the realm of very shallow trees of depth 4, the number of possible equations is very limited and hence we decided not to balance the dataset with regard to the depth.
Considering the level of the root node of the transformation, we observe light tendency towards the nodes middle level and below. This is standing to reason since (on average) more nodes exist at the deeper
3Te Tree Transformer corresponds model without its pointer component and the many-to-one embeddings.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Table Axioms Used to Create the Data Sets
Arithmetic
Exponential and Logarithm
(a+ +
at + a ++1
a+B+
(0.0))77 (8.7)
(3+7) ((9) (a)) (078)7 + (17) (at)7 + (((7) 1-1 +1 a/B
0.3-3.a
a. a.(87) ++ 7
a+0+a
a.0+0
a.1Ha
a/B
1+1#2
+/ (a
2+143
3+1#4
+ (lle))))((()+hiii
-1.10-1
In(1) ++0
2
In(e) +
clu(a) ++
-44-1-4
-1.-161
a - 3 +-++(-1-pp
a 0
Trigonometrical
tan(a 3) ++ (tan(a) tan(3))/(1 tan(a) tan (a))
tan(a 8) ++ (lan(a) + tan(5))/(1 tan(a) -aantan (ann
sin(a B) sin(a) cos( cos((a sin(B)
sin(a - -8) sin(a) cos(8) -oo((r) sin(B)
cos(a + +3) + cos(a) cos(3) - sin(a) sin(3)
cos(a - (3) <> cos(a) cos( + sin(a) sin(3)
tan(2 a) ++ 2 tan(a)/(1 tan(a)²)
(an(-1 + -1 (an((a)
cos(-1..a) + cos(a)
tan(a) ++ sin( cos(a)
cos(TT a) + -1 cos
tan(a/2) ++ sin(a)/(1+ cos(aa))))))
s -1 1----1ssmmte sin(a))))
lan(= + tan(a)
sin( sin(x-a)) sin(a)
tan( -a) -1- -1-tanaa))
cos((T/2) -a) + sin(a)
sin(a) (cs(a))"+
cos(2 (0(2-a) + cos(a)² sin
sin a1.a) + sins
cos(2 a)) + 2- cos(a)2
sin
aa) (T) cos(a)
cos(2. ++ - sin (a)
(2n(2.o) ++ 2- sin(a)) cos(a)
cos(TT + -1 -oce:-looossssss c))((a)
levels and hence there are more positions where axioms could be applied. Furthermore, some axioms require their root node to be several nodes above the leafs of the equation tree because the depth of the subtree they modify. An example for this (tan(a) - ttan(3))/(( + tan(a) tan(3)) - tan(a (3) which enforces the root node to be at most three levels above the leafs.
For our robustness experiments, steps to transform the left side into the right side. Its distribution of depth and level of transformation root is given in table b It reveals similar patterns to the test dataset of our main experiments, although it shifts the data towards deeper trees. This can be explained by the fact that deeper trees allow for more complex equations which consequently allow to derive equal expressions in more steps than the shallow trees.
additionally created dataset whose equations require more than five
For our second robustness experiment, we create dataset that consists of deeper trees than seen during training. Its distribution is shown in table 7 Since there are enough possible expressions for each
22

---

Published in Transactions on Machine Learning Research (01/2025)
Table Count of Samples by Depth of Tree and Level Transformation Root
Level of Transformation Root
Depth of Tree
Sum
2
3
4
5
6
0.65%
1.51%
0.00%
0.00%
0.00%
0.00%
2.16%
4
2.17%
3.20%
3.91%
0.00%
0.00%
0.00%
9.28%
3.47%
4.69%
3.59%
6.46%
0.00%
0.00%
20.22%
4.43%
5.31%
7.68%
6.33%
7.39%
0.00%
31.13%
7
4.31%
4.81%
7.42%
8.17%
6.19%
6.30%
37.21%
Sum
15.03%
19.51%
24.60%
20.97%
13.58%
6.30%
Table 6: Count of Samples by Depth of Tree and Level of Transformation Root for Nine Steps
Level of Transformation Root
Depth of Tree
Sum
2
3
4
5
6
3
0.22%
0.33%
0.00%
0.00%
0.00%
0.00%
0.55%
4
0.84%
1.65%
1.59%
0.00%
0.00%
0.00%
4.08%
2.12%
3.39%
4.49%
3.99%
0.00%
0.00%
13.99%
3.64%
5.62%
8.66%
8.20%
6.46%
0.00%
32.56%
4.21%
6.04%
8.79%
11.39%
10.29%
8.11%
48.82%
Sum
11.02%
17.03%
23.53%
23.57%
16.74%
8.11%
considered depth, we decided keep the dataset balanced with regard the depth. Considering the level of the root node, the dataset shows similar trend as the dataset for the main experiments before which can be explained by the same underlying patterns.
Figure 10 visualizes the distribution of the axioms in our dataset. It is clearly visible that only small fraction of about 25 axioms appear in more than 1.5% of the samples while the vast majority appears much less frequent, with 35 axioms appearing in less than 0.25% of all samples. While creating the dataset, we balance the probabilities for selecting axioms as described section 3 Yet, is standing to reason that in the generated data the axioms are not equally distributed because the number of expressions where each axiom can be applied differs largely between the axioms.
The most common axioms we identify in our dataset are simple identities which only depend on a single variable and few functions or operators, for example a + 0 + a, a + a, a + or el(()) + a. On the other hand, there is a wide range of axioms being used much more seldom. Those axioms generally depend on more variables, functions, and operators or do not have any variables all. Examples include tan(a 3) (3) - (tan(a) (1n(8))(11-tall)
tan(3)), cos(a- B) ~> (of(())cos(()) sin(a) ((na))sss)).......)))))))))))))) 2 - 3, 3 3-+ 33 It is easy see that, compared to the axioms above, those axioms can only match much smaller subset of possible expressions. Consequently, there is a lower chance of sampling them during the data generation process since the generator can only sample one of the matching axioms.
However, the influence of the frequency of an axiom on its prediction accuracy is negligible. Using the predictions obtained from TreePontteNNtt, we only found a very weak correlation of = 0.073 between the fraction of correctly identified instances of an axiom and its share in the dataset. Thus, in line with section 5.4 there are other factors influencing the chance of correctly identifying an axiom like the uniqueness of its tree structure.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Table 7: Count of Samples by Depth of Tree and Level of Transformation Root on Deep Trees
Level of Transformation Root
Depth of Tree
Sum
3
5
8
9
10
8
2.11%
2.11%
3.27%
3.27%
4.26%
2.70%
3.27%
0.00%
0.00%
0.00%
0.00%
20.99%
9
1.67%
1.98%
2.61%
3.27%
3.14%
4.17%
2.39%
2.17%
0.00%
0.00%
21.41%
10
1.32%
1.71%
2.15%
2.52%
2.46%
2.77%
2.66%
1.67%
1.87%
0.00%
0.00%
19.12%
11
1.51%
1.19%
1.80%
2.06%
2.72%
2.55%
2.79%
2.92%
1.62%
1.43%
0.00%
20.59%
12
0.83%
0.68%
1.56%
1.93%
2.06%
2.06%
2.22%
1.82%
2.35%
1.30%
1.08%
17.89%
Sum
7.44%
7.66%
11.39%
13.06%
14.64%
14.25%
13.33%
8.58%
5.84%
2.72%
1.08%
3.5-4.5%
Share in the Dataset
2.5-3.5%
1.5-2.5%
0.5-1.5%
0.25-0.5%
0-0.25%
5
10
15
20
25
30
35
Number of Axioms
Figure 10: Distribution of Axioms in the Dataset
C Hyperparameters
This section lists all relevant hyperparameters we used for training our models and the resulting number of trainable parameters. The hyperparameters were found using Optuna (Akiba al., 2019 with the TPE sampler.
We the identical search space for all models. For the size of embeddings, search powers of two between 32 and 512. For the number of encoder and decoder layers we scarch values between and 8. For the hidden size of the encoder and decoder layers, we search powers of two between 32 and 1024. For the transformer model, we search the number of attention heads for all power two between and 16. In TreePointerNet and SeqPointer, we search the number of pointer heads between and 3. For the LSTM, we check we should use unidirectional a bidirectional encoder. Also, we check if attention is beneficial. Finally, we optimize the ordering of the target sequence to be either axiom position position axiom.
In table table X we list the values of all transformer-based models. The LSTM networks consists of one-layer umidirectional LSTM encoder and decoder with an embedding size of 512 and hidden size of 1024. It has 6,496,256 trainable parameters.
Detailed Results for Main Experiments
In this section, we show the detailed results as discussed in section 5 Tables 9 to 12 show the average
accuracy (# standard deviation over five model runs, each time initialized with a different random sced), i.c. the fraction of correctly transformed equations grouped by the number of required transformations and the macro average all groups. The best results are indicated in bold.
24

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: Hyperparameters Used for our Models
TreePointerNet
SeqPointer
Transformer
Layers
5
5
Encoder
Attention Heads
Hidden Size
1024
512
512
Embodding Size
128
512
512
Layers
5
5
Attention Heads
16
16
Decoder
Heads for Pointer
Hidden Size
1024
1024
Embedding Size
64
512
512
Parameters
2,649,901
22,768,644
22,768,644
Table 9: Results for all Models on Equations of up to 5 Required Transformation Steps
Number of Transfornation Steps
Model
Average
2
3
5
Tee (ours)
88. 43 4.00%%
82. 0..00%%
74.43%%2..o.
66.53 50%.%%8%%%%
59.05 0..555%%.0000 80%%
74.09%
SeqPointer
88.41%20.066
70.83%=20.47%
55.84%20.64%
43.52%=2.666
36.74%220.965
59.47%
Transformer
2.44
70.27%=4.94%%
55.76%641.658
45.20 45.29%
36.65 26.65%%+0.72%%
59.28%
LSTM
85. 85.49%%a..199
70.57%=#0.99%
57. 57.29%sa..41%
46.61% 2..25%
17.40%
59.47%
Table present the main results. The results for robustness experiments are presented in table 10 for the experiments on deeper trees and table 11 for the experiments on equations requiring up to 9 transformation steps. The results of the ablation study on TreePointerNet are given in table 12
E Representing the Index as a Separate Token
experimented with second tokenization strategy where the index is repre- sented as separate token from the operator, constant, function, variable. We evaluated our SeqPointer, transformer, and LSTM baselines with this alternative strategy on the same data and experiments conducted in section The results are presented in table 13 (main experiment), table 14 (deep input trees), and ta- ble 15 more transformation steps). all tables we include TreePointerNel for reference, although it is not affected by the alternative tokenization strategy.
As described in section
With regard to the main experiment, it can be observed that the effect of the tokenization the performance of the models is small. While the performance of the transformer model increases by 0.66 percentage points, the PeqPointer looses 1.32 percentage points. Similarly, the L.STM looses 2.87 percentage points.
Considering the equations requiring up to 9 transformation steps, the same pattern can be observed. Here, the transformer gains 1.35 percentage points while the SeqPointer and the LSTM both loose 1.56 and 3.06 percentage points.
Slightly different results can be observed when considering the experiments on deeper trees. Here, the separate index token seems to be more beneficial. Both the transformer as well as the SeqPointer marginally gain accuracy, namely 3.99 and 1.2 percentage points. However, the LSTM looses again, namely 2.34 percentage points.
Summarizing, one can observe that this tokenization strategy proved to be beneficial for the transformer whose performance increases in all experiments. On the other hand, the LSTM looses accuracy on all experiments. The SeqPointer yielded mixed results, it could only benefit when considering the deep input trees. Most importantly, in all cases the performance of the sequential models distinctly stays below the
25

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: Robustness Study for all Models on Equations of Deeper Parse Trees
Number of Transformation Steps
Model
Average
TreePointerNet (ours)
58.48%%10.77
49.52 10..52%=1..000
40.58%
34.13%##..700
29.71%#11.0m 11111.000%%
42.48%
SeqPointer
50.49%%2.4488
36.34% 7..
26.31 26.311%22.000
19.80%=0.700%
15.21 3.23.8%%
29.63%
Transformer
49.97%
34.09% 00.09%%
24.32%:1
18.73%=0.78%
13.43%o2o.81%
28.11%
LSTM
50.12%:0..85%
31.77%
23.11%441.39%
17.92%:20.58%
12.97%
27.18%
Table 11: Robustness Study for all Models on Equations of up to 9 Required Transformation Steps
Number of Transformation Steps
Model
Average
6
8
TreePointerNet (ours)
58.49%+t..5o
54. 54.63%++1.3%%
51. 51.34%+1.744%
49.14 09.14%##1.s
53.40%
SeqPointer
34.47%+00.m 80%
32.78%=0.47%
30.15%%40..5
27. 27.95%+0..0 86%
31.34%
Transformer
33.16%++1.002
32.80%+0.85%%
29.42%+11.00
27.64%+0.66
30.75%
LSTM
36.18%++..88
32.26%
29.95%
28.74%+4..600
31.78%
accuracy of TreePointerNet and therefore does not qualitatively change the results we obtained for our experiments in section 5
F
Exploring Large Language Models for the SETI Task
Listing 1: Prompt Template for LLMs
Give step towards transforming equation into an equivalent form. Give me the axiom to apply and indicate its exact position, where to apply it.
Example: Input:
sin ( cos (0 } cos ( sin ( 0 (
cos { cos (0))
sin sin (0) ())))== sin ( ( cos_0
sin 1 0 } Answer:
sin (cos (y))) (cos)((cxx)))xii)))))) (v)) (x-x)) Position:
sin 0
Example: Input: sin tan ( 2 ) / cos ( (tan ( ( 2
= sin _ 0 0 0 / 2
tan 0
/ } /_ cos
tan (
) Answer: I-x++0, Position: **_0
Example: Input: ( ((sin sin (( 1))
** 1) ) 1)) (-1) ( sin
( (1100- 11) +.0
1_2 0 (110) Answer: x=*1<-x,
Position: 3
Input: cos 2 (
((-1) (-1) sin 2 ) ) =
) (22- 0
(co8_
cco__ (((-10 +11
sin 0 2 222) ) 0=2) - 0000) Answer:
In this work, we develop TreePointerNet parameter-efficient neural network architectures for the SETT task and compare it to various neural baselines. Both TreePointerNet and the baseline models are trained to explicitly solve the SETI task. However, in recent years, large language models (LLMs) emerged as multi- purpose neural networks that were designed to receive a prompt stating a problem in natural language and then output solution. In this section, we briefly want to explore the capabilities of various LLMs for the SETI task.
As first experiment, used ol (OpenAl 2024 with a prompt like shown in listing I The reasoning time of ol was between 20 and 50 seconds per transformation step. Thus, due to the long reasoning time and the limited access to this model, we could not evaluate it for all equations in our dataset. Instead, we tested it on two equations requiring only one transformation step and two further equations requiring more transformation steps. All of them were solved correctly by TreePointerNet.
We found that ol was able yield an output in the correct format, ie. pair of an axiom and a position, in all tested cases. Moreover, it able to solve all examples which require only transformation step
26

---

Published in Transactions on Machine Learning Research (01/2025)
Table 12: Ablation Study for our TreePoinnerrNet
Number of Transformation Steps
Model
Average
TreePointerNet (ours)
58.43%
82.02 0...5
74.43%%
66.53%440.4%%
74.09%
w/o ME
87.82%=20.34%
69.01 0.22.
52.59%220.855
42.84%220.52%
34.31%=o. 2..5.0
57.31%
w/o pointer
6.76%.20.36%
0.60%.20.08%
0.16%620..6%%
0.07%:20.04%
0.00%%
1.53%
w/o pointer ME
87.76%21.199
00.877%:2.42
52.29%:10.799
42.677:=0.51%
34.10%.20.429
57.14%
Table 13: Results for all Models on Equations of up Roquired Transformation Steps with Separate Index Token
Number of Transformation Steps
Model
Average
2
3
4
5
TreePointerNet (ours)
88.43% 2..4% 0.00%%
82. 0.%1%
74.43%%o.oo
66 00.53%%2..119
59. 39.05%%...66. 80%
74.09%
SeqPointer
$8.58%
69.32%%=0.95%
53.95%::0..8%
14.05%=+1.00
3/.83%++0..1%
58.15%
Transformer
88.57%=0..88%
71.47%==o..0
56.13%%t..45%
45.94%=11.4%%
37.60%#t..6377
59.94%
LSTM
84.18%22..51%
67.81%: 11:1
54.33%21.729%
42.67%:20.96%
34.00%220.929
56.60%
correctly but failed on the equations which require more transformation steps. However, since ol is closed- source model that can change at any time, the acquired results are not representative and can only be considered as rough test towards the capabilities of LLMs.
In addition, we tested two open-source LLMs that were trained for mathematical reasoning tasks and can run locally, namely Llemma (Azerbayev et al. 2024) and Qwen-2.5 Math (Yang et al. 2024h. However, using above prompt none of these models were able to solve our task on various tested equations. Instead of axiom and position for step, they output textual deduction of the equivalence.
To make the task easier, experimented with chain-of-thoughh prompting since it proofed beneficial for rea- soning tasks in other settings (Wei et al. 2022). Preciscly, we added textual descriptions to the input exam- ples like Example: Input: ( sin ( cos ( ) cos ( sin ( ) ( cos cos ( ) sin_0
Answer: The first sin can be transformed using the axiom (sin (x)(x))==(())) (x)++)n()))<<snn (x-y)). The answer is (xin(x)*cos)) - (x) models' output. Finally, we removed the indices from the input expression and let the model deduce the position. However, the models still only yields textual description of the equivalence.
* sin sin (0) (0)))
cos 0_0 -_ sin (0_1)
<-s)))
sin_O. There was no change in the
We conclude that the tested soeresource LLMs are not able to solve our task with simple prompting tech- niques. Further rescarch is necessary to integrate the SETI task into LLMs, be by finctuning them on the task or developing more sophisticated prompting techniques.
G Examples
For illustration, we present few equations solved by our NreePointer in this section. Tables 16 show examples of equations that the model can derive in steps. In tables 20 and 21 show examples our out-of-distribution tests on equations that can be derived in 9 steps.
19
In each lable, the first n rows correspond to a step of the ilerative transformation process and the last row shows the final result, For every step S n, the first column shows the input equation given to the network. The sccond and third columns show the axiom and the position as they are predicted by the network. The subsequent row i+1 presents the equation after applying the predicted axiom the predicted position from step i.
27

---

Published in Transactions on Machine Learning Research (01/2025)
Table 14: Robustness Study for all Models on Equations of Deeper Parse Trees with Separate Index Token
Number of Transformation Steps
Model
Average
TreePointerNet (ours)
58.48%.0..
49.52
19..2%%11.22
40.58%f11.357
34.13%##1.70%
29.71 71%6%.009%
42.48%
SeqPointer
55.82%20.535
39.16%=20.79%
26.81%:110
20.21% 10.21%=21.99%
14.25%220.322
31.25%
Transformer
54.82%620.74%
39.52%:2..d 1.447%
28. 28.22%
21.75%=1.11%
16.20%:2..66
32.10%
LSTM
47.50%22..225
29.37%6:1.000
20.30%:24.7399
15.90%=#1.01%
11.14%42058%
24.84%
Table 15: Robustness Study for all Models on Equations of up to 9 Required Transformation Steps with
Separate Index Token
Number of Transformation Steps
Model
Average
TreePointerNet (ours)
58.499%20..55%
54.63%
51.3 51.34%+11.74%
49.14%%+1.81
53.40%
SeqPointer
32.58 32.58%ot..67%
31.44%.20.25%
28.6 28.64%110.78%
26.46%
29.78%
Transformer
35.25%.40.88%
33.68%:20.69%
30.56%o:0.76%
28.57%
32.01%
LSTM
33.16%ott..25
29.83%::1.66%
26.78%o11.11%
25.12%::1.26%
28.72%
In each equation, we use the symbol A to represent the power instead of the commonly used superscript. For example, we write x A 2 instead of x². By doing so, we are able to indicate the position of the power as predicted by the network the same way we do for the other binary operators (e.g. to. A1).
Since a new expression tree is generated in each transformation step, the indices of the mathematical tokens are reinitialized in each row of the tables. As a consequence, another instance of the token as it did in the step before. For example, in table [6] the rightmost addition summing the constants 2 and 1 is denoted as +1 in step 1 because another addition exists in the same equation. However, in step 2 it becomes to since the other addition is climinated by the axiom chosen in step 1.
any step predicted position might refer to
28

---

Published in Transactions on Machine Learning Research (01/2025)
Table 16: Transformations for eln(1)+l(2++1) =
Step
Equation
Predicted Axiom
Predicted Position
en Ao (Ino(1o) to Ini (2n +1 11))
In(a) [n( + In(a -B)
+
Eo Ao (Lo 10 (2g to 11) = 3
2 2 +
to
eo Ao Iuo [1o - 30) 3
a 8 - 8-a
0
4
eo No In 30 00 lo) =
Q 7 a
U
en no (3p) = 3
(n(()) + a
Ao
Final
30 = 3
Table 17: Transformations for tan(((In(e) + (x²))=²) =tan=)
tan(1 +
Step
Equation
Predicted Axiom Predicted Position
1 tano (ano((()no(eo) to za) to (zo A1 0o)) Ao (2) Ag 01)) tan(1 +1+
a +
Ag
tano(((Ino(e)) (eo) +o 20) 'o (Io Ai 0o)) Ag lo) = tan(1 +2)
at -> a a + x 1 > In(e)
Ap
3
tano (Ing (eu) - 20) 10 (2o Ao 0n]) tan(1
Au
tano (Ing (ca) lo =0) Io) taan1 (1
'0
(Ino +o zu) tam(1 +))
Ino
Final
tano to 2o) tan(1 +
Table 18: Transformations for (in(z/22- sin((-1) z. 2)) = cos(2 sin(2) cos(2))
Step
Equation
Predicted Axiom
Predicted Position
1
simal((nu (())) - sint (o) (Np 21))) cus((2 (2- sin(=)) cus(z))
sin( a.) -
siuj
2
sino((Fo//ll (2)) -0 (( 1o) 10 sin sin 21))) cas((2 (2- sin(z)) -((z)))ccooff cos(z))
sin( 2 - a) + cas(a)
sing
3
cosoll [[ ln) situa
2n1) cos((2 sin(2)) cos(2))
cos( 1 1. -> cos(a)
cosp
4
coso (sing (#o 10 20)) cns((2 sin(=]) cos(s))
a. 8
'0
5
cosy (sino (2g 10 z0) cos((2 sin(z)) cos(2))
sin a) - 2 sin(a) cos(a)
simto
Final
c06o 1 sino(=1) (1)) to (20) cos((2 sin(2)) cos()))
Table 19: Transformations for 2 c.ccs(11 (1-1111__00 x²)² (sin(x)² + cos(x))) = cos(2 x))
Step
Equation
Predicted Axiom
Predicted Position
(20 n (cosol (00% (1n Aa (-1e]) -1 Ag ]] ((sing (Xa)
(2r) 2s)) cos(2
1 +11 slnia] + coe(a) +1 (ene(a)*) cos(2-o)
Ag
(22 vo (00e)(lu 11 m2) Az 2x)) -u ([sinu(onu) Ag 2n) (cosulity) Ag 2x)) cos(2 x)
to
(roea((1a ma)) 2:))
rrs(2.ar)
voeu (2: (1a an)) con(2-r)
a [8 7) (a-
coco 1o) 10 20) cos(2-22 (2)
a.
Final
00so(20 1 cas(2
Table 20: Transformations for (sin((=-((-1)) (-1))) 1) (()( sin cos(
Stop
Equation
Prodlcted Axiom
Predieted Posttion
(singlfo mp [(=10) 1o] (111])]]))))))) (1 To (11, oog (g)t)) Ay (g2)) Ag ([11g) 1()) = oel))
sin o sin(e)
sing
(sinul(-1u (-1a)) -1
([Io ((m) Ag an (m) No ((-1e) As
siml1/(1 cos())
[n(]]
Ino
(sina((-1 (-11]) -1 Tol (0) th cosolfe Ay) Au Agl) (o ([-1g) Ag lall sin[1//1 owlll)
(o*) 0
[singl( 1o) ( 1,)1 lo) ((1, coe
Ooll) ( 1g)
sin(1// cas(1))
0
(sina((-1o) -s (-1i])- -1
(Oh cosatro Ool) Ao [[-[(a le]] sin01// oww())
(-1)-(-1)-+1
(sln,[1,) lo) (C1 conu by (,2) Ao 16) 12)) am((1//1 eos(1)
(sing(ár) to (Ok cosotiy Ge]) (-1o)) (I)) con(l)
a. -0/2
(smo(111) lol/olla ooee (no 06]) sin(1)(( coe(1)
6 11
(sin t Mofla (a)) sinl1/(11 cos(1))
Final
sing (lo)/oll to coso = sim(1/11 (x))
29

---

Published in Transactions on Machine Learning Research (01/2025)
Table 21: Transformations for (((sin(z))) (-1)) (cos(()))) cos((T 2- (T y)) sin)yysn
Equation
Prodicted Axtom
Predicted
([innlay ng 23 (- hull to (f bizy (eo By 2,10) moylla (2, (-: (4) M61) smdy)
a- 2-1 o//
2
(((singlita) Ro) (- lal) mo ([-1g) (ronulin)) 3,00 (oa/// (o
(-()ein(())
con((m/2) -ol kinte)
(asttnfar) Anl (=1a)) the ((-1a) (2s) 2,3) (1a (oa-)))) (-1))-st/a))
a [[-1)-
(x Aa Ro) i (- lof) no (c0so(a) Au Ri]) a asi 0o) (-1)
0-u a
[[0 la) (sin no 203) (cosalnn) An aaim
(-1)-sin(s)
@ d ([-1)- -0 (a-n)
(sinatr))
[-1) -inl))
((-la) Heina and Aa 2x) to (ig ng 2
Do) (-1)-sinig
sinla) coeo))
((-1a) 1s) (in ma (0) ( sin(()
sin/s o0 - (-1)-1-
sito
sinolw)
sinig)
(=1g] mp alma (ao) (-1)+ si((y)
30


Reviews:
[{"review_id": "XiFNZLDXmI", "reviewer": "Reviewer_Z11v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Unlike previous studies, which mainly focused on determining the equivalence of math expressions, the proposed task, SETI, targets finding a sequence of symbolically verifiable steps for equivalence determination. This makes the results more reliable and interpretable.\n- A domain-specific model, TreePointerNet, has been implemented for SETI. Experiment results demonstrate its effectiveness over baselines and the benefits of the ablative improvements.\n- The experiments are solid. They report not only average values but also the standard deviation over 5 runs.\n\n### Weaknesses\n1. Poor organization and writing quality.\n\t1. The content structure is illogical. For instance, the dataset generation is placed in Sec.3. Experiment.\n\t2. The writing is overly verbose.\n\t3. There are several typos:\n\t\t1. In Sec.3.1, since $\\mathbb R^+ = \\\\{x\\in\\mathbb R | x > 0\\\\}$, it is unnecessary to write $x, y, z \\in \\mathbb R^+\\backslash{}\\\\{0\\\\}$\n\t\t2. In Sec.5 L1: \"_To the best **o f** our knowledge, there **exits** no closely related work_\", where \"o f\" and \"exits\" appear to be misspellings.\n2. Absence of important related works.\n\t- The authors should review symbolic methods for determining the equivalence of math equations. For example, [1] can solve equations (and subsequently determine the equivalence of math expressions) in commutative (semi)rings.\n3. All constants are limited to $\\\\{-4, -3, \\dots, 3, 4\\\\} \\cup \\\\{e, \\pi\\\\}$, and variables are assumed to be $x,y,z\\in \\mathbb R^+\\backslash{}\\\\{0\\\\}$, which may lead to\n\t1. Limited applicability. Such a tight limitation makes this dataset deviate far from real-world applications.\n\t2. The operations do not have closure under domain $\\mathbb R^+$. Given constants in $\\\\{-4, \\dots, 4\\\\} \\cup \\\\{e, \\pi\\\\}$ or $\\mathbb R^+$, expressions can sometimes result in negative real numbers and zero, e.g. $\\sin(4) \\approx -0.76 < 0 \\not \\in \\mathbb R^+$, which fall outside the domain of variables $\\mathbb R^+$.\n\t3. Some \"axioms\" are not valid in certain cases, e.g. $\\ln(1) = 0$, thus $\\ln(1)^{\\ln(1)} = 0^0$ is an undefined expression ($\\alpha^0\\leftrightarrow 1$ doesn't hold); furthermore, $\\frac{\\ln(1)}{1} = \\frac{0}{1} = 0$, but $(\\frac{1}{\\ln(1)})^{-1} = (\\frac{1}{0})^{-1}$ is undefined ($\\alpha/\\beta\\leftrightarrow (\\beta/\\alpha)^{-1}$ doesn't hold).\n4. Lack of supporting literature.\n\t- In the abstract, the authors state \"*this problem is expensive to solve with rule-based systems due to the large search space*\". In the introduction, they claim \"_In practice, however, given a large enough equation and set of axioms, this quickly becomes computationally infeasible_\". However, throughout the paper, the reviewer fails to find any supporting literature for these claims.\n5. Insufficient discussion with Formal Theorem Proving.\n\t- In the introduction, the authors mention \"_in remotely related settings like step-by-step solutions to word problems, only the final answer is checked, but not the predicted intermediate steps_\". But in formal theorem proving, all steps are formal-verifiable.\n6. Incomplete Baselines.\n\t- Symbolic methods should have been included.\n7. The tokenization strategy for baselines may result in unfair comparisons.\n\t- Each token (operator, variable, and constant) is tokenized differently based on its position, as shown in Sec.3.2 \"_the equation `ln(1)/1 = 0` is tokenized to `[=, /_0, ln_0, 1_0, 1_1, 0]` for the sequential baselines_\". The more frequently a particular token appears, the lower the probability of generating the corresponding equation. For example, $1_0 +_0 1_1 +_1 1_2 +_2 1_3 = 4_0$ has a low generation probability. Hence, tokens that occur multiple times in one expression, such as the $+_2$ and $1_3$ in the above example, are scarce in the training set, and baselines may underfit them. This unfairness is more severe in Sec.4.3.Robustness Study.\n\n[1] Grégoire, Benjamin, and Assia Mahboubi. \"Proving equalities in a commutative ring done right in Coq.\" _International Conference on Theorem Proving in Higher Order Logics_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Introduced a **new task**, namely Stepwise Equation Transformation Identification (SETI): given two equivalent math expressions, generate a sequence of (axiom, position) pairs to transform one expression into the other;\n2. Constructed a **dataset** for model training and evaluation on SETI;\n3. Implemented a **novel model**, TreePointerNet, which is specifically designed for SETI by incorporating a tree-structured transformer with a copy-pointer mechanism and a custom embedding. Ablation experiments demonstrate the effectiveness of these enhancements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is advisable to refine the writing to improve its clarity and conciseness. (critical)\n2. It is recommended to incorporate a literature review of symbolic methods for equivalence determination. (recommended)\n3. Expanding the domains and reconsidering the axioms could enhance the generalizability and significance of this work. (critical)\n4. The addition of symbolic baselines and a fairer tokenization strategy would make the experimental results more convincing. (critical)", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "znSlHXDgnp", "reviewer": "Reviewer_nGRS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n**1. Novel Task Design:**\n\nSETI is a task that shifts the focus from simple equivalence classification to iterative, interpretable reasoning steps. This direction aligns with real-world applications where stepwise explainability is essential, such as in automated theorem proving.\n\n**2. Innovative Architecture:**\n\nTreePointerNet demonstrates strong architectural novelty through its integration of tree-structured input, pointer mechanisms, and Many-To-One embeddings. These components are shown to enhance performance and robustness.\n\n**3. Scalable Dataset Construction:**\n\nThe dataset construction process is highly efficient and automated, leveraging extensions of existing generation frameworks.    The diversity of included transformations (polynomials, trigonometry, logarithms, etc.) ensures broad applicability.\n\n**4. Lightweight Model with Robust Performance:**\n\nTreePointerNet requires significantly fewer parameters than baselines while achieving state-of-the-art performance.    Its robustness to increasing transformation steps and tree depth is particularly noteworthy.\n\n**Weaknesses:**\n\n**1. Unclear Motivation and Application Scenarios for the SETI Task:**\n\nThe paper lacks a detailed discussion on the motivation for introducing the SETI task and its potential real-world applications. This requires further elaboration.\n\n**2. Limited Visualization:**\n\nDespite the technical depth of the paper, many sections rely heavily on textual descriptions. For example, the explanation of data generation and the TreePointerNet architecture could be significantly enhanced with additional figures or flowcharts.\n\n**3. Outdated Baselines:**\n\nWhile TreePointerNet is based on tree neural networks, the baselines used for comparison rely on traditional neural networks, some of which are outdated.  This raises concerns about the fairness of the comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**1. Introduction of the SETI Task:**\n\nThe paper introduces a novel task called Stepwise Equation Transformation Identification (SETI), which focuses on iteratively transforming a mathematical expression into an equivalent form using a sequence of axiomatic steps.  This task is significant, as it bridges the gap between equivalence classification and interpretable reasoning, enabling deeper exploration into step-by-step mathematical problem-solving.\n\n**2. TreePointerNet Architecture:**\n\nThe authors propose TreePointerNet, a novel neural network architecture that effectively combines a TreeTransformer-based encoder, a pointer generator network for position identification, and a Many-To-One Embedding Layer.  This integration enables efficient and accurate modeling of hierarchical mathematical transformations.\n\n**3.  Dataset Creation:**\n\nA comprehensive dataset tailored for the SETI task is presented, featuring equations with varying complexities and up to five transformation steps.  The dataset, built on previous generation techniques (e.g., Wankerl et al., 2023), is unique in its explicit focus on iterative transformations, filling a gap in symbolic mathematics datasets.\n\n**4. Experimental Benchmarks:**\n\nExtensive experiments demonstrate the superior performance of TreePointerNet over multiple baselines, with robust generalization to deeper parse trees and higher transformation complexity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1. Reorganize the Paper for Clarity:**\n\nImprove the overall structure of the paper.  For example, integrate figures to explain the network structure concisely, focusing on the pointer mechanism and the Many-To-One Embedding Layer (ME) for better readability.\n\n**2. Simplify Dataset Generation Explanation:**\n\nUse figures to illustrate the dataset construction process, reducing the reliance on textual explanations.\n\n**3. Enhance Motivation and Applications of the SETI Task:**\n\nProvide a more thorough discussion on the motivation behind introducing SETI and highlight its real-world applications.\n\n**4. Standardize Terminology:**\n\nEnsure consistent usage of key terms like “ME” throughout the paper.  For instance, “me” is used in lowercase on page 6 but appears as “ME” in uppercase in Figure 3.  Consider using italicization or another consistent format to denote this component.", "broader_impact_concerns": "**1. Scalability Challenges for Complex Problems:**\n\nThe model’s performance degradation with increasing transformation steps is evident. In competitive mathematical problem-solving, proofs often require ten or more steps. Whether this lightweight tree-based model can handle such problems remains uncertain and warrants further exploration.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h6iANMyLwt", "reviewer": "Reviewer_XrD7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nClear Motivation: The work tackles a well-defined problem in symbolic mathematics, emphasizing iterative, interpretable transformation steps over direct equivalence classification.\n\nNovel Architecture: TreePointerNet effectively integrates hierarchical structural information and positional embeddings, outperforming baselines.\n\n\nweaknesses:\n\nInsufficient Mathematical or Algorithmic Explanation: The paper primarily relies on empirical results to demonstrate the effectiveness of TreePointerNet, but it lacks a systematic theoretical analysis explaining why the model outperforms baseline models (e.g., SeqPointer) on specific tasks. \nFor example, the paper does not clearly explain how TreePointerNet's architectural components (e.g., tree structures, hierarchical embeddings) theoretically enhance its ability to handle deep transformations.\n\nUnclear Fairness in Complexity and Parameters: The paper claims efficiency for TreePointerNet but does not comprehensively analyze its parameter count, computational complexity, or inference speed compared to baselines. This omission makes it difficult to fairly evaluate the trade-offs.\n\nShallow Analysis of Misclassifications: While the paper identifies some misclassified axioms and provides examples (e.g., structurally similar trees), it does not deeply investigate the root causes of these errors. \nFor instance: Are the errors due to failures in hierarchical embeddings or attention mechanisms? How do misclassifications differ across trees of varying depth and complexity?\n\nIncremental Contribution: The main innovations of TreePointerNet (e.g., tree-structured transformers, pointer generation, hierarchical embeddings) are primarily adaptations or combinations of existing techniques, rather than entirely novel contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Introduction of SETI Task: The authors propose a novel task named Stepwise Equation Transformation Identification (SETI), which focuses on identifying axiomatic steps to transform one mathematical equation into an equivalent form iteratively.\n\nTreePointerNet Architecture: They design a novel neural network, TreePointerNet, which leverages:tree-structured transformer model, a pointer generator for identifying transformation positions, and custom embeddings for distinguishing mathematical tokens.\n\nNew Dataset Creation: The authors develop and release a dataset specifically tailored for the SETI task, featuring equations of varying complexity and up to five transformation steps.\n\nExperimental Benchmarking: Extensive comparisons against strong baselines (e.g., SeqPointer, Transformer, and LSTM models) and ablation studies demonstrate the superior performance of TreePointerNet.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Strengthen Theoretical Explanation\nDiscuss the theoretical advantages of TreePointerNet’s architectural components (e.g., tree structures, hierarchical embeddings, and pointer generation) over baseline models such as SeqPointer or Transformers.\n\nProvide Comprehensive Comparisons of Efficiency\nCompare TreePointerNet’s parameter count and computational complexity with baseline models. Clearly show how its efficiency is achieved without sacrificing accuracy.\n\nConduct Deeper Error Analysis\nInvestigate and categorize the root causes of misclassifications, particularly in the cases of structurally similar trees or transformations.\nFor instance:\nAre the errors related to specific components, such as hierarchical embeddings or attention mechanisms?\nHow do tree depth or transformation complexity influence error rates?\n\nClarify and Enhance Novelty\nClearly articulate which aspects of TreePointerNet are novel compared to existing techniques (e.g., tree transformers, pointer networks). \nExpand the related work section to include a more detailed comparison with similar tree-based models or pointer networks. Discuss how TreePointerNet’s contributions build upon or differ from these prior approaches.", "broader_impact_concerns": "Scalability Challenges\nIncreasing Computational Complexity: The model’s performance decreases significantly as tree depth and transformation steps increase. While TreePointerNet is efficient relative to baselines, its hierarchical embeddings and attention mechanisms may still pose challenges when deployed in resource-constrained environments or scaled to handle millions of equations.\n\nInterpretability and Trust Issues\nOpaque Decision-Making: The model’s internal mechanisms (e.g., hierarchical embeddings and attention weights) are not fully interpretable, which might make it difficult for users to trust its predictions.\n\nLimited Generalizability\nRestricted Applicability Across Mathematical Domains: The reliance on a predefined set of axioms and tree structures limits the model’s ability to generalize to new mathematical domains or problems that involve different rules or representations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5WM6TLJgup", "reviewer": "Reviewer_EmxS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Stengths**\n\n1. The framework and SETI task can be employed in symbolic mathematics to provide interpretable intermediate steps for mathematical problem-solving.\n2. The model significantly outperforms baselines (such as sequential transformers and LSTMs) on accuracy, especially for equations requiring multiple transformation steps.\n\n**Weaknesses**\n\n1. Lack of Integration with Contemporary LLM Approaches:\n   1. While the paper introduces an innovative task and architecture, it does not explore how large language models (LLMs), such as LLaMA and Qwen could be adapted to the SETI task. These models are much more popular in recent research areas and daily lifes.\n   2. Current advances in **Chain of Thought (CoT)** prompting, which allow LLMs to generate intermediate reasoning steps, could have been explored as a baseline or supplementary approach. This omission leaves a gap in comparing TreePointerNet to state-of-the-art methods for interpretability and stepwise reasoning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThe paper introduces the Stepwise Equation Transformation Identification (SETI) task, which is designed to generate a sequence of axiomatic steps to transform one mathematical expression into another equivalent form. This task addresses limitations in existing research that often treats mathematical equivalence as a binary classification without providing insight into intermediate steps.\n\n**Contributions**\n\n1. **TreePointerNet Model**: The paper proposes TreePointerNet, which combines hierarchical tree-structured transformers with pointer networks and many-to-one embeddings to predict transformation steps efficiently.\n2. **Dataset Development**: The authors generate a new dataset comprising diverse equations, transformations, and varying complexities, ensuring robustness and benchmarking.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Incorporate Experiments with Contemporary LLMs: add experimental baselines using large language models (LLMs) with Chain of Thought (CoT) prompting techniques. \n2. Update Baseline Models: I suggest the author include more competitive models. Maybe compare the GPT4-o1 with TreePointerNet on one or two examples selected from the SETI task.\n3. Discuss Broader Implications: add a section discussing the potential applications of TreePointerNet in education, research, and real-world symbolic computation tools.", "broader_impact_concerns": "No need.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MM0TPDg03e", "reviewer": "Reviewer_Dovy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed Stepwise Equation Transformation Identification (SETI) task is novel and significant, opened up a new path in the field of automated mechanical theory proving. The proposed method is not uncessarity relied on LLMs and ceatively combined TreeTransformer and Pointer Network to guarantee the precise appropriation of the formula terms. All of the contributions make this work quite interesting. \n2. The theory construction part in section 2 is concrete, with enough formular  representations to clarify the proposed methods. \n3. The proposed TreePointerNet has a parameter size of 2.6M, which is far more smaller than the existed methods, which indicates the imporvement of the architecture and training data are effective in the SETI task.\n4. The experiment shows the proposed TreePointerNet achieves the best results over other baseline methods on various experiment settings. The ablation of different proposed components successfully exhibites the effectiveness of each of them. \n\nWeaknesses：\n1. The major weakness is the language presentation of this paper. The writing style is not quite easy to follow.  Three major types of pitfalls are found: (1) Some of the complex sentences are too long to clearly represent the author's idea. For example, \"This approach has the davantage that ...... nodes observed during training\" at page 3.  (2) There exists some unnecessary sentenses, some of them  confuse the reader. For example, \"We make our code and dataset ...... (after acceptance of the paper)\" and \"As with all deep learning architectures ...... to an embedding\" at page 3 . (3) The excessive use of the linking words cannot individually make the paragraph more logical. For example, in page 2 it includes \"In theroy, ... In practice, ... however,  ... Empirically, ... Hence, ... However, ... So far, ... More over , ... Thus, ... Besides ...\" in a paragraph with 13 lines. \n2. The concept of the proposed task is poorly defined in the Introduction section. Why do not the author put the first paragraph of Section 2, a very clear definition of the task, into the introduction section to clarify the task? \n3. The organization of the sections is confusion and do not well paralleled with the introduction. To be psecific, Why other's work about TreeTransformer are described in section 2.1 with so many details? And the dataset constraction part is unexpectedly placed in the experiment section. \n4. Not compare the proposed methods with that proposed in recent years. The latest baselines are from the following years, 2017, 2020, 1997.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Introducing the SETI Task: The authors introduce the new Stepwise Equation Transformation Identification (SETI) task in the field of deep learning for symbolic mathematics. This task focuses on predicting a sequence of axiomatic mathematical transformation steps to transform one mathematical expression into an equivalent one. It fills a gap in the research area by providing a more detailed and step-by-step approach to understanding the equivalence of mathematical expressions.\n2. Proposing the TreePointerNet Architecture: A novel neural network architecture, TreePointerNet, is designed. It combines a pointer generator network with a hierarchical-accumulation model for tree-structured input and a unique embedding strategy. This architecture is tailored to solve the SETI task efficiently and effectively, taking advantage of the inherent structure of mathematical expressions.\n3. Generating a New Dataset: A new equation dataset is created for the SETI task. The dataset consists of pairs of mathematically equivalent expressions represented as expression trees and includes a wide range of mathematical functions and equations with varying complexity. This dataset is crucial for training and evaluating models on the SETI task and provides a valuable resource for future research in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The contents are good, but the structure of the paper need to be re-organized.\n2. The problem statement might need to be put in an independent section, and improve the clarity of the problem statement in the introduction. \n3. The formulation expression of the \"ME\" operation need to be unified over the whole paper for more clearly presentation. For example, \"Our many-to-many embedding layer me then ......\". \n4. Investivate more recent methods about the axiom proving problem, and represent the experiment results woth more detailed tables and figures.", "broader_impact_concerns": "1. The proposed axiom transformation method can be used to generate intermediate stepped axiom proving data for mathematical LLMs with significantly low costs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please see above. 

After the rebuttal, the reviewers are positive about the manuscript and recommend acceptance of the paper. The authors are recommended to further consider enhancing the versatility of the task and using more up-to-date baselines.
Decision: Accept (poster)