Venue: NAACL 2025

# E-Gen: Leveraging E-Graphs to Improve Continuous Representations of

Symbolic Expressions

 Hongbo Zheng Suyuan Wang Neeraj Gangwar Nickvash Kani

University of Illinois at Urbana-Champaign

{hongboz2,suyuan2,gangwar2,kani}@illinois.edu

Equal Contribution.

###### Abstract

Vector representations have been pivotal in advancing natural language processing (NLP), with prior research focusing on embedding techniques for mathematical expressions using mathematically equivalent formulations. While effective, these approaches are constrained by the size and diversity of training data. In this work, we address these limitations by introducing E-Gen, a novel e-graph-based dataset generation scheme that synthesizes large and diverse mathematical expression datasets, surpassing prior methods in size and operator variety. Leveraging this dataset, we train embedding models using two strategies: (1) generating mathematically equivalent expressions, and (2) contrastive learning to explicitly group equivalent expressions. We evaluate these embeddings on both in-distribution and out-of-distribution mathematical language processing tasks, comparing them against prior methods. Finally, we demonstrate that our embedding-based approach outperforms state-of-the-art large language models (LLMs) on several tasks, underscoring the necessity of optimizing embedding methods for the mathematical data modality. The source code and datasets are available at [https://github.com/MLPgroup/E-Gen](https://github.com/MLPgroup/E-Gen).

## 1 Introduction

While large language models (LLMs) Jiang et al. (2023); Dubey et al. (2024); Hurst et al. (2024); Jaech et al. (2024) have demonstrated effectiveness in processing natural language, these methods still perform suboptimally with math-based content which plays an important role across numerous domains Zanibbi et al. (2024); Rohatgi et al. (2019). For instance, even a state-of-the-art LLM like GPT-4V only reaches 49% accuracy in MathVisa Lu et al. (2023), a comprehensive benchmark to evaluate the mathematical reasoning capabilities of LLMs. In another instance, as tested in Frieder et al. (2024), GPT-4's performance in computing integration is dominated by specialized embedding-based models Lample and Charton (2019); Lample et al. (2022). Therefore, effective approaches to process semantically rich mathematical notation are necessary.

One approach is to compute a semantic representation of mathematical content based on textual context and treat mathematical expressions as a sentence without learning mathematical semantics Krstovski and Blei (2018). This has been used in various mathematical language processing (MLP) tasks, such as mathematical information retrieval Topic et al. (2013); Mansouri et al. (2022), identifier definition extraction Pagael and Schubotz (2014); Hamel et al. (2022); Zou et al. (2024), mathematical reasoning Geva et al. (2020); Nye et al. (2021) and theorem proving Wang et al. (2020); Wu et al. (2022). But this approach has two limitations: (1) textual descriptions are lacking in some math content like textbooks or mathematical derivations in academic publications, and (2) don't really explore relations between symbolic operators from a mathematical perspective.

To address these problems, recent studies have focused on equivalence and mathematical manipulation between expressions to derive the meaning of mathematical expressions independent of context. However, these approaches remain limited, primarily due to the quality of available data. Allamanis et al. (2017) introduces EqNet, a TreeNN-based Socher et al. (2013) model to group equivalent expressions together, but their datasets are limited to arithmetic and boolean expressions. SemembGangwar and Kani (2023) computes vector-based semantic representations by learning to generate mathematically equivalent expressions but is significantly limited by the SymPy-generated dataset Meurer et al. (2017). SymPy functions are designed for simplification and directly derive the most simplified expression without intermedi

ate steps, leading to a dataset that lacks enough rewrites per expression. Hence, a more efficient mathematical data generation scheme is required.

In this work, we propose E-Gen, a highly scalable and efficient corpus generation scheme based on e-graphs (Willsey et al., 2021). Leveraging a collection of mathematical transformations, E-Gen facilitates the creation of synthetic datasets with large clusters of semantically equivalent expressions. This approach significantly improves upon prior SymPy-based methods, overcoming limitations in the number of rewrites per expression while improving flexibility and scalability. In summary, we highlight the following contributions of E-Gen:

1. We introduce E-Gen, a novel scheme for generating a cluster-based mathematical expression dataset, along with a high-diversity mathematical corpus.
2. We evaluate two types of embedding models based on seq2seq and contrastive learning respectively, showing improved semantic representation performance over prior works in quantitative and qualitative tests.
3. The embedding models are evaluated on two out-of-distribution downstream tasks, to demonstrate models' generalizability and robustness.
4. Finally, we compare our models with GPT-4o across multiple tasks, demonstrating the effectiveness of embedding-based approaches.

## 2 Related Work

In MLP (Meadows and Freitas, 2022), semantic representation shows strong potential across various problems. A representative application is mathematical information retrieval (MIR) (Kristianto et al., 2016; Zanibbi et al., 2016a; Mansouri et al., 2022c), where expressions or keywords are ranked based on their relevance to a query. Gao et al. (2017) proposes Symbol2Vec, a mathematical symbol representation generation scheme to group LaTeX symbols having similar contexts together. They further extend this approach to a MIR scheme (Formula2Vec) to prove its effectiveness. Krstovski and Blei (2018) converts symbolic layout trees (Zanibbi et al., 2016b) of mathematical expressions into token sequences and generates representation based on word2vec. Mansouri et al. (2019) further improves this approach by combining the symbol layout tree and operator tree representations of an expression. Peng et al. (2021) proposes MathBERT, a pretrained model based on MIR and formula headline generation task (Yuan et al., 2020).

Identifier definition extraction (Kristianto et al., 2014) is another promising application, which aims to align identifiers found in scientific text with their definitions. Popovic et al. (2022) utilizes a transformer-based method to develop an end-to-end joint math entity and relation extraction approach. Jo et al. (2021) provides a mathematical understanding pretrained model by fine-tuning BERT on masked mathematical identifier prediction task. In the theorem proving field, Welleck et al. (2021b, a) generate theorem proof by retrieving relevant references to a given query theorem. However, as discussed in Section 1, MLP studies rely on semantic representation but mostly focus on establishing homomorphism between context and mathematical expressions to construct semantic representation, which is limited by the incapacity to process pure math content and can not really understand mathematical transformations. Even though some recent studies (Allamanis et al., 2017; Meidani et al., 2023) have focused on generating semantic representation based on intrinsic features of mathematical expressions to address this problem, their performance and scope of application are still limited by existing dataset, motivating this work's demonstration of a more effective mathematical corpus generation scheme.

## 3 Corpus of Equivalent Expressions

### Corpus Generation

To generate a diverse set of mathematical expressions, we manually design templates containing placeholders for arithmetic operators, functional operators, and numerical values. Expressions are systematically instantiated by replacing the placeholders in templates with all possible operators and values, resulting in approximately 5,000 initial expressions in prefix notation. The templates are carefully designed to cover fundamental arithmetic and functional operators in various formats while maximizing the potential for mathematical transformations. Each of the resulting initial expressions is processed by E-Gen, which applies around 800 mathematical rules to generate clusters of semantically equivalent expressions.

The core of our E-Gen is _e-graph_, an advanced data structure designed to efficiently manipulate collections of terms under a congruence relation.

 

An e-graph is composed of e-classes, each containing a set of equivalent e-nodes. An e-node can be linked to one or more child e-classes, depending on the operator's arity. From a mathematical perspective, child e-classes represent the arguments of their associated e-node, typically corresponding to a mathematical operator. Consequently, any subgraph originating from an e-node within the same e-class represents equivalent expressions.

Figures 0(a) to 0(d) show the e-graph saturation process. In the initial e-graph (Figure 0(a)), the e-node "+" is linked to two child e-classes "\(x\)" and "\(8\)", respectively, and the e-node "\(-\)" is linked to a subgraph "\((x+8)\)" and a child e-class "\(8\)", together forming the initial expression "\((x+8)-8\)". In Figure 0(b), a new e-node "+" is added to the top e-class after the associative law is introduced. The two e-nodes, "\(-\)" and "\(+\)" in the top e-class represent subgraphs for expressions "\((x+8)-8\)" and "\(x+(8-8)\)" which are equivalent rewrites generated by the associative law. Similarly, transformations such as "\((x-x)\to 0\)" and "\(x+0\to x\)" are embedded into the e-graph in the steps shown in Figures 0(c) and 0(d). The e-graph is iteratively expanded by applying each applicable rule, thereby capturing all possible equivalent expressions.

A context-free grammar is created after e-graph saturation in prefix notation, based on e-classes and the connections of e-nodes in it, as shown in Figure 0(e). E-classes will be represented as variable symbols denoted by "\(e0\)", "\(e1\)", "\(e2\)", "\(e4\)" in Figure 0(d). Variables and numbers comprise the terminal set in the grammar. The production rules are determined by the edges and take the form:

\[e{\langle}\text{index}{\rangle} \rightarrow[var/num]\] \[e{\langle}\text{index}{\rangle} \rightarrow[op~{}e{\langle}\text{index}{\rangle}~{}e{\langle} \text{index}{\rangle},~{}\ldots]\]

where \(e{\langle}\text{index}{\rangle}\) denotes eclass with corresponding index, \(op\) and \(var/num\) denote encodes representing operators and variables/numbers.

Equivalent expressions are extracted from this grammar using a recursive rewrite algorithm. The process begins at a designated root e-class (e.g., "\(e0\)") and traverses the grammar, replacing e-classes with their corresponding expansions until no e-classes remain. For instance, "\(e2\)", in the second grammar "\(-~{}e2~{}e1\)" of "\(e0\)", can be expanded to "\(+~{}e0~{}e1\)" and generate "\(-~{}+~{}e0~{}e1~{}e1\)". To avoid excessively long rewriting, a token length limit of 25 and a time limit of 600s are imposed. Eventually, clusters of semantically equivalent expressions for all initial expressions are generated by E-Gen and form a new corpus.

### Corpus Analysis

The new corpus encompasses a comprehensive set of arithmetic and functional operators as detailed in Table 1. Table 2 presents examples from three clusters of equivalent expressions generated by E-Gen. The corpus is structured into clusters, each containing numerous mathematically equivalent expressions derived through a series of flexible transformations. These transformations span

\begin{table}
\begin{tabular}{l l} \hline \hline
**Category** & **Operators** \\ \hline Arithmetic & \(+\), \(-\), \(\times\), \(\div\) \\  & pow, abs, sqrt, \(\frac{d}{dx}\) \\ Logarithmic/Exponential & ln, exp (as pow e) \\ Trigonometric & sin, cos, tan \\  & csc, sec, cot \\ Inverse Trigonometric & sin\({}^{-1}\), cos\({}^{-1}\), tan\({}^{-1}\) \\  & scsc\({}^{-1}\), sec\({}^{-1}\), cot\({}^{-1}\) \\ Hyperbolic & sinh, cosh, tanh \\  & csch, sech, cath \\ Inverse Hyperbolic & sinh\({}^{-1}\), cosh\({}^{-1}\), tanh\({}^{-1}\) \\  & csch\({}^{-1}\), sech\({}^{-1}\), coch\({}^{-1}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Operator coverage of E-Gen Corpus.

Figure 1: Illustration of e-graph saturation (0(a) to 0(d)) and grammar creation (0(e)). An e-graph consists of e-classes (dashed boxes) containing equivalent e-nodes (solid boxes). Edges connect e-nodes to their child e-classes. Applying mathematical rules to an e-graph adds new e-nodes and edges (0(b) and 0(c)), or merges e-classes (0(d)). Additions and modifications are emphasized in black. In 0(e), the saturated e-graph is converted into a context-free grammar, where each grammar is defined using e-class id and the e-nodes with their child e-classes.

 

both simple and complex mathematical relationships, significantly enhancing expression diversity and enabling models to efficiently learn underlying mathematical rules with fewer initial expressions.

As detailed in Table 3, the E-Gen corpus supports a wider range of operators and achieves a significantly higher average cluster size of 102, compared to the _Equivalent Expressions Dataset_ (EED) Gangwar and Kani (2023) that is generated using SymPy and limited to a cluster size of 2 (pairs). This substantial increase in cluster size enables models to develop a deeper semantic understanding of the diversity of mathematical transformations.

Training Data.The training dataset is provided in two formats: expression pairs and triplets, corresponding to the two training methodologies described in Section 4. For the seq2seq model, equivalent expression pairs are generated by permuting expressions within each cluster. For contrastive learning, expression triplets are constructed by treating expressions within the same cluster as positive pairs, while randomly sampling expressions from different clusters to serve as negative examples. This process results in a training set comprising 55 million equivalent expression pairs and 50 million expression triplets as shown in Table 3. For the validation set, a subset of clusters is randomly sampled from the corpus, yielding a total of 8,077 expressions. Following Lample and Charton (2019) and Gangwar and Kani (2023) , we use the prefix notation to encode the expressions.

## 4 Methodology

As illustrated in Figure 2, we employ two well-established approaches for learning representations of mathematical expressions: sequence-to-sequence (seq2seq) Sutskever (2014); Cho (2014) equivalent expression generation and contrastive learning (CL) Wu et al. (2018); Chen et al. (2016)

\begin{table}
\begin{tabular}{l l} \hline \hline
**Initial expression**: & \(\left(\frac{x}{6}+2\right)^{9}+3x\) \\ \hline \(-(-2-\frac{x}{6})^{9}+(-3(-x))\) \\ \(3x-(-2-\frac{1}{6}x)^{9}\) \\ \((-1)(-3x-(2+\frac{x}{6})^{9})\) \\ \(1/(\frac{6}{6}+2)^{-9}+3x\) \\ \((-2+(-1)x/6)^{9}+3x\) \\ \hline \multicolumn{2}{c}{**Initial expression**: & \(\tanh(3x-(-4))-6\)} \\ \hline \(\tanh(3x+4)-6\) \\ \(1/\coth(3x+4)-6\) \\ \(\sinh(3x+4)\operatorname{sech}(3x+4)-6\) \\ \(\tanh(3/\operatorname{csc}(\sin^{-1}(x))+4)-6\) \\ \(\sinh(3\cos(\cos^{-1}(x))+4)/\cosh(3x+4)-6\) \\ \hline \multicolumn{2}{c}{**Initial expression**: & \(\frac{d}{dx}-2(\ln x/7)^{-2}\)} \\ \hline \(4/\left(x(\ln(x/7))^{3}\right)\) \\ \(-2/\left(x/(\ln(x)-\ln(7))^{-3}\right)\times(-2)\) \\ \(-4x^{-1}/\ln(7\times x^{-1})^{3}\) \\ \(-4/(x(\ln(7)-\ln(x))^{3})\) \\ \(4\cot(\tan^{-1}(x))(\ln(x/7))^{-3}\) \\ \hline \end{tabular}
\end{table}
Table 2: Examples of equivalent expressions generated with E-Gen. Expressions listed below each initial expression are part of the equivalent rewrites. Additional examples are in Appendix A.1.

Figure 2: Overview of E-Gen and the cluster-based training framework. Equivalent expressions of each initial expression are generated using an e-graph-based approach, forming clusters of equivalent expressions. The seq2seq model is trained on equivalent expression pairs, while the contrastive learning model is trained on triplets, with each triplet containing a reference expression \(\mathbf{x}\), an equivalent positive sample \(\mathbf{x}^{+}\), and a non-equivalent negative sample \(\mathbf{x}^{-}\). \(\mathbf{z}\) is the latent space representation of the corresponding input \(\mathbf{x}\).

 

2020), both leveraging the transformer architecture (Vaswani, 2017). These methods aim to capture the underlying semantic relationships between mathematical expressions by generating meaningful embeddings in a high-dimensional latent space.

### seq2seq Expressions Generation

Following prior work (Gangwar and Kani, 2023), we employ the seq2seq framework to learn representations of mathematical expressions by training the model to generate mathematically equivalent expressions. Specifically, given a pair of equivalent expressions, the model takes one as input and is tasked with predicting the other as output. The encoder, during this process, learns to map the input expressions into a latent space where semantically equivalent expressions are clustered together.

### Contrastive Learning

The other promising approach to learning mathematical expression representations is _contrastive learning_, a technique that has gained significant traction in the domain of representation learning. The primary objective is to learn a latent space where semantically equivalent expressions are embedded closer together, while semantically distinct expressions are pushed apart. This is achieved using a contrastive loss function, such as InfoNCE (Oord et al., 2018) or SimCSE (Gao et al., 2021).

In this manuscript, we use a variation of the InfoNCE loss formulated as:

\[{\mathcal{L}}(f)=\mathbb{E}\left[-\ln\frac{e^{f^{\intercal}(\mathbf{x})f(\mathbf{z}^ {+})}/\tau}{e^{f^{\intercal}(\mathbf{x})f(\mathbf{x}^{+})}/\tau+e^{f^{\intercal}(\mathbf{ x})f(\mathbf{x}^{-})}/\tau}\right] \tag{1}\]

where \(f\) is a transformer encoder \(f:{\mathcal{X}}\rightarrow{\mathcal{Z}}\) that maps an tokenized input expression \(\mathbf{x}\in{\mathcal{X}}\) to its latent representation \(\mathbf{z}\in{\mathcal{Z}}\). The terms \(\mathbf{x}^{+}\) and \(\mathbf{x}^{-}\) correspond to positive (equivalent) and negative (non-equivalent) samples respectively, with \(\tau\) as a temperature hyperparameter.

### Representation Vector

To derive the representation vector from the transformer encoder, an additional step is required to convert the output matrix from the encoder's last layer \(\mathbf{X}\in\mathbb{R}^{S\times D_{\text{model}}}\) to a one-dimensional embedding vector \(\mathbf{x}\in\mathbb{R}^{D_{\text{model}}}\) for each expression. Here, \(S\) and \(D_{\text{model}}\) represent the sequence length of the input and the model dimension. We experiment with two common pooling strategies: average pooling and max pooling over the hidden states of the last encoder layer. Our empirical analysis and prior work (Gangwar and Kani, 2023) indicate that max pooling consistently outperforms average pooling for the seq2seq model. In the case of contrastive learning, we train two separate models--one with average pooling (CL Mean) and one with max pooling (CL Max)--to evaluate the impact of these strategies on embedding quality. In all models, special tokens, such as the start-of-expression and end-of-expression tokens, are excluded from the pooling process.

## 5 Experiments

### Evaluation Tasks

The models trained on the E-Gen synthetic dataset, are compared against the prior Semmb model (Gangwar and Kani, 2023) trained on SymPy-generated corpus, on both in-distribution and out-of-distribution tasks, demonstrating the efficacy of the newly presented methods.

K-Means Clustering.K-Means clustering (Macqueen, 1967) is employed to evaluate the performance of both seq2seq and CL models. The test set comprises 8,077 expressions in 279 clusters, with cluster sizes ranging from 20 to 40 expressions. The K-Means algorithm is applied to group the expressions into clusters. Clustering performance is evaluated by mapping the predicted cluster labels to the corresponding ground truth labels. Accuracy is computed as the proportion of expressions correctly assigned to their respective clusters. A visualization of clusters is shown in Figure 3.

As shown in Table 4, both the seq2seq and CL models trained on the E-Gen cluster-based dataset exhibit strong performance in clustering semantically equivalent expressions while effectively separating non-equivalent ones. The CL models slightly outperform the seq2seq model due to their explicit training objective of grouping semantically equivalent expressions in the latent space. In contrast, Semmb, trained on a SymPy-generated corpus,

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **Accuracy (\%)** \\ \hline seq2seq & 96.72 \\ CL Mean & 97.61 \\ CL Max & 97.30 \\ Sememb & 37.70 \\ \hline \hline \end{tabular}
\end{table}
Table 4: K-Means clustering accuracy (%) of seq2seq, CL Mean, and CL Max, compared against prior Semmb model.

 

achieves only 37.70% accuracy. This significant performance gap underscores the limitation of the SymPy-generated dataset, which lacks the diversity of mathematical transformations present in the E-Gen corpus. Consequently, Sememb struggles to accurately capture and differentiate both subtle and substantial syntactic variations among equivalent expressions.

Semantic Understanding Beyond Syntactic SimilarityTo assess whether models trained on the E-Gen corpus capture mathematical semantics rather than relying on syntactic similarity, we design an experiment where the model must identify the single semantically equivalent expression among syntactically similar candidates. Specifically, we sample 1,000 query expressions from the test set. For each query, one correct answer is introduced, which is semantically equivalent but structurally distinct from the query. Additionally, six distractors are generated by making minor syntactic modifications to both the query and the correct answer (e.g., altering a single operator or numerical value). These distractors are divided into two groups: three closely resembling the query and three closely resembling the correct answer. The model is tasked with selecting the correct equivalent expression from seven candidates.

As illustrated in Table 5, the correct answer (bolded) is syntactically distinct from the query. Candidate 2, 4, 7 exhibit structural similarity to the correct answer, while candidate 1, 3, 6 closely resemble the query. As shown in Table 6, models trained on the E-Gen corpus significantly outperform Sememb, demonstrating their ability to differentiate semantically equivalent expressions despite structural variations, as well as to distinguish between syntactically similar yet semantically non-equivalent expressions.

Mistake Detection.Mistake detection in mathematical derivations is an out-of-distribution downstream task, with the test set generated using SymPy to assess the robustness and generalizability of the models.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **Accuracy (\%)** \\ \hline seq2seq & 76.41 \\ CL Mean & 50.36 \\ CL Max & 48.19 \\ SemEmb & 28.62 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Semantic understanding beyond syntactic similarity accuracy (%) of seq2seq, CL Mean, and CL Max, compared against prior Sememb model.

Figure 3: Visualization of representation vectors for 17 different single-operator mathematical expressions and their equivalent forms using our method (left) and SemEmb (right). t-SNE Van der Maaten and Hinton (2008) is applied to reduce the dimensionality of the embeddings from 512 to 2.

 

For this task, step-by-step mathematical derivations are generated for each expression in the E-Gen corpus, and mistakes are introduced into randomly selected steps within these derivations. The goal is to ask the models to identify erroneous steps by classifying transformations between consecutive steps as either "mistake" or "no mistake". To achieve this, a semantic similarity threshold for binary classification is calculated using only derivations generated from expressions in the training set of the E-Gen corpus, as described in Algorithm 1. Specifically, cosine similarities are computed between consecutive steps in each derivation, excluding steps with mistakes. The minimum cosine similarity among the correct steps is recorded for each derivation, and the average of these minimum values is used as the threshold. A step is flagged as a mistake if its cosine similarity with the preceding step falls below this threshold.

```
1:input: set of derivations \(\{\mathbf{D}_{i}\}^{N}_{i=1}\), each containing a sequence of derivation steps \(\mathbf{D}_{i}=\{\mathbf{d}_{k}\}^{M}_{k=1}\)
2:output: threshold value \(t\)
3:\(f\): transformer encoder
4:9: cosine similarity function
5:\(x\leftarrow[]\)
6:for all\(\mathbf{D}_{i}\in\{\mathbf{D}_{1},\dots,\mathbf{D}_{N}\}\)do
7:\(\mathbf{Z}_{i}\gets f(\mathbf{D}_{i})\)
8:\(c\gets g(\mathbf{Z}_{i}[\texttt{!}\cdot\texttt{!};\texttt{)},\mathbf{Z}_{i}[ \texttt{!}\text{--}\texttt{!}\texttt{!}],\texttt{dim--}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!} \texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{}\texttt{!} \texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{}\texttt{!} \texttt{!}\texttt{}\texttt{!}\texttt{!}\texttt{}\texttt{!}\texttt{{!}} \texttt{!}\texttt{{!}\texttt{}}\texttt{{!}\texttt{!} \texttt{!}{}\texttt{!}\texttt{{!}}\texttt{{!}}\texttt{{!}\texttt{{!}} \texttt{!}{!}\texttt{{!}}\texttt{{!}}\texttt{{!}} \texttt{{!}}\texttt{{!}}\texttt{{!}}\texttt{{!}}\texttt{{!}} \texttt{{!}}\texttt{{}}{!}\texttt{{}!}\texttt{{}}\texttt{{}!} \texttt{}{}{!}\texttt{{}}\texttt{{}}{!}\texttt{{}}\texttt{{}}{!}\texttt{}{} \texttt{}{}}\texttt{}{}}\texttt{}{}}\]

\begin{table} \end{table}
Table 7: Example comparison of mistake detection in mathematical derivations between seq2seq and GPT-4o. Errors in the derivations are in red. The � symbol indicates that the respective model has predicted the step to contain a mistake.

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Derivation** & **seq2seq** & **GPT-4o** \\ \hline \(-\frac{1}{8}(\sinh(\cosh^{-1}(7-\frac{\pi}{s})))^{-1}\) & & \\ \(-\frac{1}{8}(\sinh(\cosh^{-1}(7+\frac{\pi}{s})))^{-1}\) & & \\ \(-\frac{1}{8}(\sqrt{6+\frac{\pi}{s}}\sqrt{8+\frac{\pi}{s}})^{-1}\) & & \\ \(-\frac{\pi}{8}(6+\frac{\pi}{s})^{-\frac{1}{2}}(8+\frac{\pi}{s})^{-\frac{1}{2}}\) & & \\ \(-\frac{1}{8}(6+\frac{\pi}{s})^{-\frac{1}{2}}(8+(1-8)x)^{-\frac{1}{2}}\) & \multicolumn{1}{r}{�} & \\ \(-\frac{1}{8}(6+\frac{\pi}{s})^{-\frac{1}{2}}(8-7x)^{-\frac{1}{2}}\) & & \\ \hline \(5/\operatorname*{ccsc}(\operatorname*{cc}^{-1}(1/\ln(\frac{\pi}{-5})))\) & & \\ \(5/\operatorname*{ccc}(\operatorname*{cc}^{-1}((\ln(\frac{1}{-5}x))^{-1}))\) & & \\ \(5/(1-(\ln(\frac{1}{-5}x))^{2})^{-\frac{1}{2}}\) & & \\ \(5\sqrt{1-(\ln(\frac{1}{-5}x))^{2}}\) & & \\ \hline \hline \end{tabular}
\end{table}
Table 8: Mistake detection evaluation results precision, recall, and F1 (%) scores of seq2seq, CL Mean, and CL Max, compared against prior Semmb model.

 

\(f(\hat{y}_{2})\) is selected as the predicted answer, excluding the original expressions \(x_{1}\), \(y_{1}\), and \(x_{2}\) from consideration.

For this experiment, 584 analogy examples are manually constructed. The entire E-Gen corpus serves as the search pool, with expressions equivalent to \(x_{2}\) and \(y_{2}\) removed to ensure uniqueness of the correct answer. Any necessary expressions are added to complete the analogy.

As shown in Table 10, the seq2seq model achieves the highest accuracy, indicating its ability to learn underlying mathematical rules and handle basic substitutions effectively. In contrast, Sememb exhibit low accuracy on this task and have a tendency to imitate the transformation between \(x_{1}\) and \(y_{1}\), leading to "look-alike" predictions rather than true understanding of the mathematical rules, as seen in Test 1, 4, 5 in Table 9.

### Comparison with GPT-4o

To further assess the quality of vector representations of symbolic expressions, a comparative analysis is conducted between the seq2seq model, trained on the E-Gen corpus, and the state-of-the-art large language model GPT-4o [3] on two tasks.

Mistake Detection.For mistake detection, a small test set is randomly sampled from the mistake detection test set described earlier. This subset consists of 322 derivation steps, including 50 steps containing mistakes, while the remainder are error-free. GPT-4o is first provided with an example derivation in the prompt and asked to identify potential mistakes. If it fails to detect the mistake, explicit feedback indicating the erroneous step is provided until it correctly understands the task. Once verified, GPT-4o is tested on queries from the sampled test set.

As shown in Table 11, our approach outperforms GPT-4o across all evaluation metrics. While GPT-4o demonstrates a recall of 66%, suggesting its capability of detecting a notable portion of errors, its low precision indicates a tendency to misclassify correct transformations as mistakes, leading to a high rate of false positives.

Table 7 illustrates examples of GPT-4o's misjudgments, which commonly occur in complex transformations, particularly those involving function operators or relatively intricate arithmetic computations. For instance, in Example 1, GPT-4o fails

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **Accuracy (\%)** \\ \hline seq2seq & 70.38 \\ CL Mean & 64.73 \\ CL Max & 50.34 \\ Sememb & 54.85 \\ GPT-4o & 39.60 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Embedding algebra accuracy (%) of seq2seq, CL Mean, and CL Max, compared against prior Sememb model and GPT-4o.

\begin{table}
\begin{tabular}{c l l l l l l} \hline \hline  & **Model** & **Precision** & **Recall** & **F1** \\ \hline \multirow{3}{*}{**no**} & seq2seq & 96.98 & 94.49 & 95.72 \\  & CL Mean & 97.64 & 91.18 & 94.30 \\ \cline{1-1}
**mistake** & CL Max & 97.78 & 91.93 & 94.77 \\  & GPT-4o & 93.58 & 90.84 & 92.19 \\ \hline \multirow{3}{*}{**mistake**} & seq2seq & 73.68 & 84.00 & 78.50 \\  & CL Mean & 64.71 & 88.00 & 74.58 \\ \cline{1-1}  & CL Max & 67.96 & 89.14 & 77.12 \\ \cline{1-1}  & GPT-4o & 56.90 & 66.00 & 61.11 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Mistake detection evaluation results precision, recall, and F1 (%) scores of seq2seq, CL Mean, and CL Max compared with GPT-4o.

 

to recognize the equivalence between syntactically different expressions (e.g., \(\sinh({\cosh^{-1}(x)})=\sqrt{x+1}\sqrt{x-1}\)). In Example 2, it incorrectly classifies an operator substitution in step 3 as a valid derivation, failing to detect the mistake.

Embedding Algebra.We also conduct the embedding algebra task on GPT-4o, which performs worse than both the seq2seq and CL models, achieving only 39.60% accuracy on the test set as shown in Table 10. Table 9 provides examples comparing the performance of the seq2seq model and GPT-4o on embedding algebra tests. Similar to the mistake detection test, an example query is provided to GPT-4o to verify its understanding before proceeding with the test.

While the seq2seq model consistently makes accurate predictions by adhering to mathematical rules, GPT-4o demonstrates only a partial understanding of certain mathematical properties. For instance, it correctly predicts the periodicity of "\(\csc(x)\)" but often mimics the structure of "\(y_{1}\)" rather than applying the underlying mathematical transformations. In Test 5, GPT-4o incorrectly subtracts "\(\pi/2\)" from "\(x\)" in "\(\csc(x)\)", imitating the structure of "\(y_{1}\)", while the correct answer should be "\(\csc(x+\pi/2)\)", which is mathematically equivalent to "\(\csc(x)\)". A similar error occurs in Test 8, where GPT-4o predicts "\(y_{2}\)" as "\(\csch^{-1}(x)\)" by replicating "\(y_{1}:\csc^{-1}(x)\)" instead of applying the correct mathematical transformation. This comparison highlights the effectiveness of E-Gen corpus in helping models to build up understanding in mathematical rules and transformations.

## 6 Conclusion

In this work, we enhance semantic representations of symbolic expressions by developing E-Gen, a novel mathematical corpus generation framework based on the e-graph data structure. E-Gen shows strong capability in the generation of a scalable, cluster-based corpus with high diversity in mathematical transformation. To evaluate its effectiveness, two training approaches based on seq2seq and contrastive learning respectively are implemented to capture equivalence relation between expressions. Our experimental results demonstrate the efficacy of these embedding models across a variety of downstream tasks, including clustering, semantic understanding beyond syntactic similarity, mistake detection, and mathematical analogies. Notably, these semantic representations outperform prior approach and GPT-4o in both quantitative and qualitative tests. This work provides an algorithmic foundation for processing symbolic mathematics, and the vector-based representations can easily integrate with vector embeddings of other data modalities.

### Limitations

This work introduces and evaluates the potential of a novel mathematical corpus generation framework to augment mathematical semantic understanding. While the results are promising, there are several areas for improvement.

First, extending the dataset to include a wider range of mathematical operators used in published datasets, such as ArXMLiv Kohlhase et al. (2024) and ARQMath Mansouri et al. (2022b), would improve the applicability of the learned embeddings to real-world mathematical data.

Following that, more efficient grammar enumeration techniques, such as those based on the Earley algorithm Earley (1970), could facilitate the extension of the E-Gen corpus to support more complex operators and higher-arity expressions. In this manuscript, equivalent expressions are generated by enumerating the grammar obtained from the e-graph using a simple recursive function. Additionally, incorporating variable characteristics, such as dimensionality, phase, and/or bounds, into the embeddings remains an unexplored but promising direction for improving their expressiveness.

Another limitation is the scarcity of real-world datasets that focus on symbolic mathematics. While tasks like mathematical information retrieval have been explored in competitions such as NTCIR Zanibbi et al. (2016a) and ARQMath Mansouri et al. (2022b), these typically involve a mixture of symbolic mathematics and natural language in queries and results. Integration of our embedding methods with natural language vectors remains an open research challenge with significant potential for advancing mathematical information retrieval and reasoning.

## Acknowledgments

We would like to thank the University of Illinois for its support in facilitating this research. We would also like to extend our gratitude to the National Center for Supercomputing Applications (NCSA) for providing access to high-performance computing resources 



## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Allamanis et al. (2017) Miliadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. 2017. Learning continuous semantic representations of symbolic expressions. In _International Conference on Machine Learning_, pages 80-88. PMLR.
* Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In _International conference on machine learning_, pages 1597-1607. PMLR.
* Cho (2014) Kyunghyun Cho. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Juhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahahe, Aiesha Lettman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Earley (1970) Jay Earley. 1970. An efficient context-free parsing algorithm. _Communications of the ACM_, 13(2):94-102.
* Frieder et al. (2024) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. 2024. Mathematical capabilities of ChatGPT. _Advances in neural information processing systems_, 36.
* Gangwar and Kanti (2023) Neeraj Gangwar and Nickvash Kanti. 2023. Semantic representations of mathematical expressions in a continuous vector space. _Transactions on Machine Learning Research_.
* Gao et al. (2017) Liangcai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. 2017. Preliminary exploration of formula embedding for mathematical information retrieval: can mathematical formulae be embedded like a natural language? _arXiv preprint arXiv:1707.05154_.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. _arXiv preprint arXiv:2104.08821_.
* Geva et al. (2020) Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. _arXiv preprint arXiv:2004.04487_.
* Hamel et al. (2022) Emma Hamel, Hongbo Zheng, and Nickvash Kanti. 2022. An evaluation of NLP methods to extract mathematical token descriptors. In _International Conference on Intelligent Computer Mathematics_, pages 329-343. Springer, Springer.
* Hurst et al. (2024) Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o system card. _arXiv preprint arXiv:2410.21276_.
* Jaech et al. (2024) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. OpenAI o1 system card. _arXiv preprint arXiv:2412.16720_.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Jo et al. (2021) Huwyeol Jo, Dongyeop Kang, Andrew Head, and Marti A Hearst. 2021. Modeling mathematical notation semantics in academic papers. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3102-3115.
* Kohlhase et al. (2024) Michael Kohlhase et al. 2024. arxmliv project. [https://kwarc.info/projects/arXMixV/](https://kwarc.info/projects/arXMixV/). Accessed: 2024-09-17.
* Kristianto et al. (2014) Giovanni Yoko Kristianto, Akiko Aizawa, et al. 2014. Extracting textual descriptions of mathematical expressions in scientific papers. _D-Lib Magazine_, 20(11):9.
* Kristianto et al. (2016) Giovanni Yoko Kristianto, Goran Topic, and Akiko Aizawa. 2016. MCAT math retrieval system for NTCIR-12 MathIR task. In _NTCIR_.
* Krstovski and Blei (2018) Kriste Krstovski and David M Blei. 2018. Equation embeddings. _arXiv preprint arXiv:1803.09123_.
* Lample and Charton (2019) Guillaume Lample and Francois Charton. 2019. Deep learning for symbolic mathematics. _arXiv preprint arXiv:1912.01412_.
* Lample et al. (2022) Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ebner, and Xavier Martinet. 2022. Hypertree proof search for neural theorem proving. _Advances in neural information processing systems_, 35:26337-26349.
* Loshchilov (2017) I Loshchilov. 2017. Decoupled weight decay regularization. _arXiv preprint arXiv:1711.05101_.
* Loshchilov and Hutter (2016) Ilya Loshchilov and Frank Hutter. 2016. SGDR: Stochastic gradient descent with warm restarts. _arXiv preprint arXiv:1608.03983_.
* Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. _arXiv preprint arXiv:2310.02255_.



J Macqueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press, Cited by: SS2.
* B. Mansouri, A. Agarwal, D. W. Oard, and R. Zanibbi (2022)Advancing math-aware search: the ARQMath-3 lab at CLEF 2022.. In European Conference on Information Retrieval, pp. 408-415. Cited by: SS2.
* Behrooz Mansouri, V. Novotny, A. Agarwal, D. W. Oard, and R. Zanibbi (2022)Overview of ARQMath-3 (2022): third CLEF lab on answer retrieval for questions on math. In International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 286-310. Cited by: SS2.
* Behrooz Mansouri, S. Rohatgi, D. W. Oard, J. Wu, C. Lee Giles, and R. Zanibbi (2019)Tangent-CFT: an embedding model for mathematical formulas. In Proceedings of the 2019 ACM SIGIR international conference on theory of information retrieval, pp. 11-18. Cited by: SS2.
* J. Meadows and A. Freitas (2022)A survey in mathematical language processing. arXiv preprint arXiv:2205.15231. Cited by: SS2.
* Kazem Meidani, P. Shojaee, C. K. Reddy, and A. Barati Farimani (2023)SNIP: bridging mathematical symbolic and numeric realms with unified pre-training. arXiv preprint arXiv:2310.02227. Cited by: SS2.
* A. Meurer, C. P. Smith, M. Paprocki, O. Certik, S. B. Kirpichev, M. Rocklin, A. Kumar, S. Ivanov, J. K. Moore, S. Singh, T. Rathnayake, S. Vig, B. E. Granger, R. P. Muller, F. Bonazzi, H. Gupta, S. Vats, F. Johnson, F. Pedregosa, M. J. Curry, A. R. Terrel, S. Roucka, A. Saboo, I. Fernando, S. Kulat, R. Cimrman, and A. Scopatz (2017)SymPy: symbolic computing in python. PeerJ Computer Science3, pp. e103. Cited by: SS2.
* Tomas Mikolov (2013)Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. Cited by: SS2.
* Maxwell Nye, A. Johan Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieper, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. (2021)Show your work: scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114. Cited by: SS2.
* A. van den Oord, Y. Li, and O. Vinyals (2018)Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: SS2.
* Robert Pagael and Moritz Schubotz (2014)Mathematical language processing project. arXiv preprint arXiv:1407.0167. Cited by: SS2.
* Shuai Peng, K. Yuan, L. Gao, and Zhi Tang (2021)MathBERT: a pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377. Cited by: SS2.
* Jeffrey Pennington, Richard Socher, and Christopher D Manning (2014)Glove: global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543. Cited by: SS2.
* Nicholas Popovic, W. Laurito, and M. Farber (2022)ALFB-WebScience at SemEval-2022 task 12: relation extraction first-using relation extraction to identify entities. arXiv preprint arXiv:2203.05325. Cited by: SS2.
* Shaurya Rohatgi, W. Zhong, R. Zanibbi, J. Wu, and C. Lee Giles (2019)Query auto completion for math formula search. arXiv preprint arXiv:1912.04115. Cited by: SS2.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts (2013)Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642. Cited by: SS2.
* I Sutskever (2014)Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215. Cited by: SS2.
* Goran Topic, Giovanni Yoko Kristianto, Minh-Quoc Nghiemi, and Akiko Aizawa (2013)The MCAT math retrieval system for NTCIR-10 math track. In Proceedings of 10th NICIR Conference, Tokyo, Japan, pp. 680-685. Cited by: SS2.
* Laurens Van der Maaten and Geoffrey Hinton (2008)Visualizing data using t-SNE. Journal of machine learning research9 (11). Cited by: SS2.
* A. Vaswani (2017)Attention is all you need. Advances in Neural Information Processing Systems. Cited by: SS2.
* Qingxiang Wang, C. Brown, C. Kaliszyk, and J. Urban (2020)Exploration of neural machine translation in autoformalization of mathematics in imazar. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs, pp. 85-98. Cited by: SS2.
* Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho (2021)NaturalProofs: mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112. Cited by: SS2.
* Sean Welleck, Jiacheng Liu, Jesse Michael Han, and Yejin Choi (2021)Towards grounded natural language proof generation. In MathAI4Ed Workshop at NeurIPS, 

Max Willsey, Chandrakana Nandi, Yisu Remy Wang, Oliver Flatt, Zachary Tatlock, and Pavel Panchekha. 2021. Egg: Fast and extensible equality saturation. _Proceedings of the ACM on Programming Languages_, 5(POPL):1-29.
* Wu et al. [2022] Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jannik, and Christian Szegedy. 2022. Autroformization with large language models. _Advances in Neural Information Processing Systems_, 35:32353-32368.
* Wu et al. [2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 3733-3742.
* Yuan et al. [2020] Ke Yuan, Dafang He, Zhuoren Jiang, Liangcai Gao, Zhi Tang, and C Lee Giles. 2020. Automatic generation of headlines for online math questions. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 9490-9497.
* Zanibbi et al. [2016a] Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topic, and Kenny Davila. 2016a. NTCIR-12 MathlR task overview. In _NTCIR_.
* Zanibbi et al. [2016b] Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Wm Tompa. 2016b. Multi-stage math formula search: Using appearance-based similarity metrics at scale. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_, pages 145-154.
* Zanibbi et al. [2024] Richard Zanibbi, Behrooz Mansouri, and Anurag Agarwal. 2024. Mathematical information retrieval: Search and question answering. _arXiv preprint arXiv:2408.11646_.
* Zou et al. [2024] Jiaru Zou, Qing Wang, Pratyush Thakur, and Nickvash Kani. 2024. STEM-PoM: Evaluating language models math-symbol reasoning in document parsing. In _The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24_.

 



## Appendix A Cluster-Based Corpus

### Cluster Example

As discussed in Section 3.2, the key distinction of the E-Gen corpus from prior datasets is its cluster-based organization of equivalent expressions, rather than equivalent expression pairs. Table 12 presents additional examples of these clusters. Each cluster consists of an initial expression along with numerous equivalent rewrites. This diverse set of transformations significantly enhances pretrained models' ability to understand and generalize mathematical semantics, leading to improved performance in downstream tasks.

### Generation Efficiency

Efficiency is a key property of any corpus generation scheme. To assess the computational performance of E-Gen, we measure the average time required for e-graph saturation and expression rewrite extraction. On an Intel Xeon E5-2666 CPU, the e-graph saturation process takes 36.83ms on average, and the extraction takes an average of 1.46s per expression. These evaluations are conducted with a 25-token length limit and a 600s timeout per initial expression.

## Appendix B Training Details

Both the seq2seq and contrastive learning models are trained using a transformer architecture, optimized with the AdamW optimizer Loshchilov (2017), and scheduled with CosineAnnealingWarmRestarts scheduler Loshchilov and Hutter (2016). The specific training parameters and time are detailed in Table 13 and Table 14 respectively.

\begin{table}
\begin{tabular}{l c} \hline \hline
* \\ \hline \multicolumn{3}{c}{**Optimizer**} \\ \hline Optimizer & AdamW \\ Learning Rate & \(1\times 10^{-4}\) \\ Weight Decay & \(1\times 10^{-2}\) \\ \hline \multicolumn{3}{c}{**Scheduler**} \\ \hline Scheduler & CosineAnnealWarmRestarts \\ \(T_{0}\) & 10 \\ \(T_{\text{mah}}\) & 2 \\ \(\eta_{\text{min}}\) & \(1\times 10^{-8}\) \\ \hline \multicolumn{3}{c}{**Training Parameters**} \\ \hline Label Smoothing & 0.1 \\ Batch Size & 256 \\ Epochs & 20 \\ Gradient Clipping & 4.0 \\ \hline \multicolumn{3}{c}{**Hardware Configuration**} \\ \hline CPU & AMD EPYC 7763 (64-Core) \\ GPU & NVIDIA L40S (46GB) \\ \hline \hline \end{tabular}
\end{table}
Table 13: Hyperparameters and hardware specifications for training the seq2seq, CL Mean, and CL Max models. The transformer-based models are optimized using AdamW with a cosine annealing warm restart scheduler. Training was conducted on an AMD EPYC 7763 CPU and a Nvidia L40S GPU.




## Appendix C Experiments

### K-Means Clustering

Accuracy Calculation.Since all expressions from the same cluster are labeled in the same class, the accuracy for K-Means clustering in Section 5.1 is calculated as follow.

\[acc=\frac{1}{K}\sum_{i=1}^{K}acc_{i} \tag{3}\]

where \(acc_{i}\) denotes the accuracy of cluster \(i\), which is computed as follow.

\[acc_{i}=\frac{1}{|c_{i}|}\sum_{x_{j}\in c_{i}}\mathds{1}_{\left\{g(x_{j})=c_{ i}\right\}} \tag{4}\]

where \(g(x_{j})\) denotes the cluster predicted by K-Means algorithm, and \(c_{i}\) is the ground truth cluster of expression \(x_{j}\).

### Formula Retrieval

Formula retrieval is implemented as an additional task to assess the models' semantic understanding of mathematical expressions. As discussed in Section 1, prior MIR studies heavily rely on contextual cues for semantic representation Gao et al. (2017); Krstovski and Blei (2018) rather than directly capturing the intrinsic mathematical property of expressions. This limitation results in suboptimal performance, particularly in scenarios with limited surrounding text, such as textbook or mathematical derivations. To evaluate the models' performance in this scenario, we design a pure formula retrieval task using the E-Gen corpus. Given a query expression, the top-\(k\) most similar expressions are retrieved based on cosine similarity in the latent space.

We evaluate formula retrieval using the E-Gen test set, which contains 8,077 expressions. Each expression serves as a query, while the remaining expressions act as retrieval candidates. Figure 4 illustrates an example where the top-\(4\) most similar candidates to \(\cos x\) are retrieved in the latent space. As shown in Table 15, both seq2seq and CL models trained on E-Gen corpus effectively retrieve semantically relevant expressions from the candidate pool as \(k\) increases. Conversely, Semmb, which is constrained by a limited number of equivalent rewrites per expression, struggles to identify semantically equivalent expressions when \(k\) exceeds 10.

### Embedding Algebra

Table 16 and 17 provide additional examples from embedding algebra task. The seq2seq model correctly predicts most of the answers, outperforming other models. CL Mean slightly outperforms, CL Max, Sememb, and GPT-4o. Notably, Semmb and GPT-4o use a similar strategy to predict "\(\mathbf{y}_{2}\)". For instance, Tests 6 to 11 specifically evaluate the models' understanding of function periodicity, where the seq2seq model trained on the E-Gen corpus accurately predicts "\(\mathbf{y}_{2}\)" in most cases. In contrast, as discussed in Section 5.2, Semmb, similar to GPT-4o, tends to imitate the transformation between "\(\mathbf{x}_{1}\)" and "\(\mathbf{y}_{1}\)" rather than correctly applying the periodicity of the function. For instance, in Test

Figure 4: Example of the formula retrieval with seq2seq model. The query expression is \(\cos(x)\) in darker blue, and the rest are candidates. Top-\(4\) expressions are retrieved. t-SNE is applied to reduce the dimensionality of the embeddings from 512 to 2.

\begin{table}
\begin{tabular}{l r r} \hline \hline
**Model** & \multicolumn{1}{c}{**Training Time (h:min)**} \\ \hline seq2seq & 38:04 \\ CL Mean & 31:56 \\ CL Max & 29:59 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Training time of seq2seq, CL Mean, and CL Max models.



6, "\({\mathbf{x}_{1}}:\sin(x)\)" and "\({\mathbf{y}_{1}}:\sin(x+2\pi)\)" are equivalent due to the \(2\pi\) period of "\(\sin(x)\)". However, since "\({\mathbf{x}_{2}}:\cot(x)\)" has a period of \(\pi\), the correct "\({\mathbf{y}_{2}}\)" should be "\(\cot(x+\pi)\)". The Sememb model just simply imitates the transformation from "\({\mathbf{x}_{1}}\)" to "\({\mathbf{y}_{1}}\)" by adding "\(2\pi\)" to "\(x\)" in "\(\cot(x)\)", and incorrectly converts "\(\cot(\ldots)\)" into "\(\csc(\ldots)\)". This comparison highlights that the models trained with E-Gen corpus shows a better understanding of mathematical rules and transformations, rather than relying on superficial "looks-like" predictions.

 

* [105] M. C. Dickson, J. R. Friedman, and A. K. S. Blackwell, "A survey of the statistical properties of a population with large numbers," in _Proceedings of the 27th International Conference on Statistical Learning Theory (ICML)_, volume 364 of _Lecture Notes in Computer Science_, pages 89-96. Springer, Berlin Heidelberg, 2000.
* [106] G. F. Hinton, P. H. Fox, and L. E. Hinton, "The classification of images by clustering," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 103-114, 1996.
* [107] T. W. Teh, "Classification of images from incomplete data," _IEEE Transactions on Pattern Analysis and Machine Intelligence_, vol. 10, no. 1, pp. 1-10, Jan 1996.
* [108] Y. Bengio, "Generational networks for image recognition," _Advances in neural information processing systems_, vol. 10, no. 1, pp. 1-10, Jan 1996.
* [109] B. C. Davis, "An introduction to machine learning," _arXiv preprint arXiv:1902.00990_, 2019.
* [110] V. I. Goodfellow, J. Pouget-Abadie, and J. Schmidhuber, "Learning deep representations via deep convolutional nets," in _NIPS'04. CVPR'04. CVPR'04. CVPR'04.
* [111] X. Chen, Z. Li, and Q. Weinberger, "Deep residual learning for image recognition," _CoRR_, abs/1412.0338v1, 2014.
* [112] X. Chen, Z. Li, and Q. Weinberger, "ImageNet: A review of deep architectures for image recognition," _ACM SIGKDD international conference on Knowledge discovery and data mining_, pp. 1-10, Jan 1996.
* [113] X. Chen, Z. Li, and Q. Weinberger, "ImageNet: A review of deep architecture for image recognition," _CVPR'04. CVPR'04.

 



\begin{table}
\begin{tabular}{c|l l l l l l} \hline \hline  & \(\mathbf{x}_{1}\) & \(\mathbf{y}_{1}\) & \(\mathbf{x}_{2}\) & \(\hat{\mathbf{y}}_{2}\) (eq2seq) & \(\hat{\mathbf{y}}_{2}\) (CL Mean) & \(\mathbf{y}_{gt}\) \\ \hline
1 & \(\sin(x)\) & \(-\sin(-x)\) & \(\cos(x)\) & \(\cos(-x)\) & \(\cos(-x)\) & \(\cos(-x)\) \\
2 & \(\sin(x)\) & \(-\sin(-x)\) & \(\tan(x)\) & \(-\tan(-x)\) & \(-\tan(-x)\) & \(-\tan(-x)\) \\
3 & \(\cos(x)\) & sec(\(x\)) & tanh(\(x\)) & coth(\(x\)) & cath(\(x\)) & cath(\(x\)) \\
4 & \(\sin^{-1}(x)\) & scsc\({}^{-1}(1/x)\) & cos\({}^{-1}(x)\) & sec\({}^{-1}(1/x)\) & sec\({}^{-1}(1/x)\) & sec\({}^{-1}(1/x)\) \\
5 & \(\sinh^{-1}(x)\) & csch\({}^{-1}(1/x)\) & tanh\({}^{-1}(x)\) & coth\({}^{-1}(1/x)\) & cath\({}^{-1}(1/x)\) \\
6 & \(\sin(x)\) & sin(\(x+2\pi\)) & cot(\(x\)) & cot(\(x+\pi\)) & cot(\(x+\pi\)) & cot(\(x+\pi\)) \\
7 & tan(\(x\)) & tan(\(x+\pi\)) & csc(\(x\)) & csc(\(x+2\pi\)) & csc(\(x+2\pi\)) & csc(\(x+2\pi\)) \\
8 & \(-\cos(x)\) & cos(\(x+\pi\)) & \(-\cot(x)\) & cot(\(x\)) x 1 & cos(\(x\))(1/\(\sin(x)\)) & tan(\(x+\pi/2\)) \\
9 & \(\sin(x)\) & cos(\(x-\pi/2\)) & sec(\(x\)) & csc(\(x+\pi/2\)) & csc(\(x+\pi/2\)) & csc(\(x+\pi/2\)) \\
10 & \(-\mathrm{csc}(x)\) & csc(\(x+\pi\)) & \(-\sec(x)\) & sec(\(x+\pi\)) & cot(\(x\)) csc(\(\pi/2+x\)) & sec(\(x+\pi\)) \\
11 & \(-\cos(x)\) & cos(\(x+\pi\)) & \(-\tan(x)\) & tan(\(\pi+x\)) & sin(\(-\,\mathrm{csc}^{-1}(\cot(x))+\pi\)) & cot(\(x+\pi/2\)) \\
12 & \(x\) & ln\(\,x\) & sin(\(x\)) & ln sin(\(x\)) & sin(ln(\(x\))) & ln sin(\(x\)) \\
13 & \(x\) & ln\(\,x\) & coth(\(x\)) & ln coth(\(x\)) & ln coth(\(x\)) \\
14 & \(x\) & ln\(\,x\) & cos\({}^{-1}(x)\) & cos\({}^{-1}(\ln x)\) & ln sec\({}^{-1}(x)\) \\
15 & \(x\) & sin\({}^{-1}(x)\) & cos(\(x\)) & sin\({}^{-1}(\cos(x))\) & sin\({}^{-1}(\cos(x))\) \\
16 & \(x\) & esc\({}^{-1}(x)\) & csch\({}^{-1}(\)cc(xc)\) & esc\({}^{-1}(\sinh(x))\) & csc\({}^{-1}(\mathrm{csc}(x))\) \\
17 & \(x\) & \(x+1\) & tan\({}^{-1}(x)\) & tan\({}^{-1}(x)\times 1\) & tan\({}^{-1}(x)+1\) & tan\({}^{-1}(x)+1\) \\
18 & \(x\) & \(x-1\) & sin(\(x\)) & sin(\(x\)) & sin(\(x-1\) & sin(x)-1 \\
19 & \(x\) & \(x/1\) & tan(\(x\)) & tan(\(x\))/1 & tan(\(x\))/1 \\
20 & \(x\) & \(x^{3}\) & csch\((x)\) & csch\({}^{3}(x)\) & csch\({}^{3}(x)\) \\
21 & \(x\) & \(x^{3}\) & sinh\({}^{-1}(x)\) & cosh\({}^{-1}(x)\) & sinh\({}^{-3}(x)\) \\ \hline \hline \end{tabular}
\end{table}
Table 16: Additional examples from the embedding algebra evaluation comparing the seq2seq and CL Mean models, both trained on the E-Gen corpus. \(\hat{\mathbf{y}}_{2}\) represents model’s prediction, while \(\mathbf{y}_{gt}\) denotes the ground truth. Incorrect predictions are highlighted in red. The seq2seq model offers superior performance over the CL Mean model.

\begin{table}
\begin{tabular}{c|l l l l l l l} \hline \hline  & \(\mathbf{x}_{1}\) & \(\mathbf{y}_{1}\) & \(\mathbf{x}_{2}\) & \(\hat{\mathbf{y}}_{2}\) (CL Max) & \(\hat{\mathbf{y}}_{2}\) (Sememb) & \(\hat{\mathbf{y}}_{2}\) (GPT-4o) & \(\mathbf{y}_{gt}\) \\ \hline
1 & \(\sin(x)\) & \(-\sin(-x)\) & cos(\(x\)) & cos(\(-x\)) & \(-\tan(-x)\) & cos(\(-x\)) & cos(\(-x\)) \\
2 & \(\sin(x)\) & \(-\sin(-x)\) & tan(\(x\)) & \(-\tan(-x)\) & \(-\tan(-x)\) & \(-\tan(-x)\) & \(-\tan(-x)\) \\
3 & \(\cos(x)\) & sec(\(x\)) & tanh(\(x\)) & coth(\(x\)) & sech(\(x\)) & cath(\(x\)) \\
4 & \(\sin^{-1}(x)\) & csc\({}^{-1}(1/x)\) & cos\({}^{-1}(x)\) & sec\({}^{-1}(1/x)\) & - & sec\({}^{-1}(1/x)\) & sec\({}^{-1}(1/x)\) \\
5 & \(\sinh^{-1}(x)\) & csch\({}^{-1}(1/x)\) & tanh\({}^{-1}(x)\) & coth\({}^{-1}(1/x)\) & - & coth\({}^{-1}(x)\) & cath\({}^{-1}(1/x)\) \\
6 & \(\sin(x)\) & sin(\(x+2\pi\)) & cot(\(x\)) & cot(\(x+\pi\)) & csc(\(x+2\pi\)) & cot(\(x+\pi\)) & cot(\(x+\pi\)) \\
7 & tan(\(x\)) & tan(\(x+\pi\)) & csc(\(x\)) & csc(\(x+2\pi\)) & cot(\(x+\pi\)) & csc(\(x+2\pi\)) & csc(\(x+2\pi\)) \\
8 & \(-\cos(x)\) & cos(\(x+\pi\)) & \(-\cot(x)\) & sec(\({}^{-1}(\cot(x))\)) & sec(\(x+\pi\)) & cot(\(x+\pi\)) & tan(\(x+\pi/2\)) \\
9 & \(\sin(x)\) & cos(\(x-\pi/2\)) & sec(\(x\)) & cos(\(x\)) & sec(\(x-\pi/2\)) & csc(\(x-\pi/2\)) & csc(\(x+\pi/2\)) \\
10 & \(-\mathrm{csc}(x)\) & csc(\(x+\pi\)) & \(-\sec(x)\) & csc(\(x+\pi\)) & sec(\(x+\pi\)) & sec(\(x+\pi\)) \\
11 & \(-\cos(x)\) & cos(\(x+\pi\)) & \(-\tan(x)\) & sin(\(\mathrm{csc}^{-1}(\cot(x))\)) & tan(\(x+\pi\)) & tan(\(x+\pi\)) & cot(\(x+\pi/2\)) \\
12 & \(x\) & ln\(\,x\) & sin(\(x\)) & ln(\(\mathrm{csc}(x)\)) & ln sin(\(x\)) & sin\({}^{-1}(x)\) & ln sin(\(x\)) \\
13 & \(x\) & ln\(\,x\) & coth(\(x\)) & ln coth(\(x\)) & cath\(\ln(x\)) & cath\({}^{-1}(x)\) & ln coth(\(x\)) \\
14 & \(x\) & ln\(\,x\) & cos\({}^{-1}(x)\) & sec\({}^{-1}(\ln