Semantic Representations of Mathematical Expressions in a Continuous Vector Space
Authors: Neeraj Gangwar
Nickvash Kani
Venue: Accepted by TMLR ()

Abstract: Mathematical notation makes up a large portion of STEM literature, yet finding semantic representations for formulae remains a challenging problem. Because mathematical notation is precise, and its meaning changes significantly with small character shifts, the methods that work for natural text do not necessarily work well for mathematical expressions. This work describes an approach for representing mathematical expressions in a continuous vector space. We use the encoder of a sequence-to-sequence architecture, trained on visually different but mathematically equivalent expressions, to generate vector representations (or embeddings). We compare this approach with a structural approach that considers visual layout to embed an expression and show that our proposed approach is better at capturing mathematical semantics. Finally, to expedite future research, we publish a corpus of equivalent transcendental and algebraic expression pairs.

Published in Transactions on Machine Learning Research (09/2023)
Semantic Representations of Mathematical Expressions in Continuous Vector Space
Neeraj Gangwar
ganguar26ellinois.ddd
Electrical and Computer Engineering University of Illmoiis Urbona- Champaign
Nickvash Kani
kanidhllinois. edu
Electrical and Computer Engineering University of Mlioois Urbena- Chaaa-Cn
Reviewed on OpenReview: oppnreviee. net/ forum? EdEENTTTZzUU
Abstract
Mathematical notation makes up a large portion of STEM literature, yet finding semantic representations for formulae remains a challenging problem. Because mathematical notation precise, and its meaning changes significantly with small character shifts, the methods that work for natural text do not necessarily work well for mathematical expressions. This work describes tor space. We use the encoder of a sequence-to-sequenne architecture, trained on visually different but mathematically equivalent expressions, to generate vector representations (or embeddings). We compare this approach with a structural approach that considers visual layout to embed an expression and show that our proposed approach is better at captur- ing mathematical semantics. Finally, expedite future research, we publish a corpus of equivalent transcendental and algebraic expression pairs.
approach for representing mathematical expressions in continuous
1
Introduction
Despite there being well-established search technologies for most other modes of data, the processing of mathematical content remains an open problem (Larson et al. 2013). Effeetive search technologies require semantically rich and computationally efficient reprosentations of mathematical data. Most equation em- bedding methods have focused on establishing a homomorphism between an equation and ils surrounding mathematical text (Zanibbi al. 20166) Krstovski & Blei 2018).. While this approach can help find equations used in similar contexts, it is less effective in the following situations: (1) surrounding text may be limited, for example, consider math textbooks that contain equations with minimal explanation, and (2) in scientific literature, an equation may be used in variety of disciplines with different contexts, and encoding equations based on textual context may hamper cross-disciplinary retrieval.
We argue that context alone is not sufficient for finding representations of mathematical content, and em- bedding methods must understand equations addition to the surrounding context. To this end, we present novel embedding method generate semantically rich representations of mathematical formulae without the aid of natural language context. In our proposed approach, we train sequence-te sequence-to-ssquenee woctrnnceett-ssuueeee model equivalent expression pairs and use the trained encoder to generate vector representations or embeddings. Figure shows an example of our approach that embeds expressions according to their semantics. We com- pare our proposed approach with a structural encoder, that considers the layout structure of an expression, on a variety tasks to determine the usefulness of each. Furthermore, we compare our semantic embed- ding method with two prior works proposed for an analogous problem of clustering equivalent expressions: EQNET (Allamanis al. 2017) and EQNET-L (Liu 2022), further proving our model's ability to capture semantics.

---

Published in Transactions on Machine Learning Research (09/2023)
EQE
div
sin(x)
cos
Transformer Encoder
Transformer Decoder
cos(x)
x
X
Polish Notation
SOE
50E
x
x
Figure An overview of our approach. The model trained to generate an expression mathematically sin(a] equivalent to the input. For example, given an input expression cos(ø) the model learns to generate tan(z). We use max pooling on the hidden states of the last encoder layer corresponding to the input tokens and use the result as the continuous vector representation of the input expression. Here, "SOE", "EOE", and "div" represent the start token, the end token, and the division operation, respectively.
The contributions of our work are threefold:
We show that sequence-to-sequeee (SeQ2SDQ) model can learn generate expressions that are mathematically equivalent to the input.
We use the encoder such a model to generate semantically rich vector representations of mathe- matical expressions. They better clustering and retrieving similar mathematical expressions. We expand on the notion of semantics subsequently.
We publish a corpus of equivalent transcendental and algebraic expression pairs that can be used to develop more complex mathematical embedding approaches.
In this work, consider the SEM VEC datasets that have previously been used by EQNET and EQNET-L to infer if representations are semantically rich. In addition, we perform distance analysis and embedding algebra and analyze embedding plots. For distance analysis, the similarity between two expressions is defined using the tree edit distance (Zhang & Shasha 1989) between their operator trees. We ignore constant additions and multiplications while computing the tree odit distance and put emphasis on the operators. This is achieved by considering f(x) and a (z) + b to be
distance zero when a and are constants.
We end this manuscript with comprehensive study of our proposed approach, detailing its potential for general information processing and retrieval tasks and noting its limitations. The datasets and source code are available on GitHub.
2 Related Work
While information processing for mathematical expressions still a relatively new field, other groups have attempted to create embedding methods for mathematical content.
Starting with embedding schemes for mathematical tokens, Gao et al. 20117 embedded mathematical sym- bols using WORD2VEC (Mikolov et al. 2013) on the Wikipedia corpus. They created a mathematical token embedding scheme (SYMBOL2VEC) in which tokens used in similar contexts (sin & cos, = =) were grouped together. Using these symbol embeddings, they extended their approach embed entire formulae using the paragraph-to--octor approach (PORMULA2VEC).. More commonly, other approaches have attempted extract textual descriptors of equations using surrounding text and use pre-trained embeddings of those textual keywords to create embedding of the equation in question (Kristianto et al. 2014 Schubotz al. 2017
2

---

Published in Transactions on Machine Learning Research (09/2023)
Expanding token descriptors to represent equations is more complex. Schubotz et al. (016) created equa- tion descriptors by combining the textual keyword descriptors of mathematical tokens included within an expression. By organizing the equation descriptors into keyword pairs, clustering algorithms grouped equa- tions according to Wikipedia categorics. In a similar approach, Kristianto et al. (2017) viewed equations as dependency relationships between mathematical lokens. Nominal natural language processing (NLP) methods were used to extract textual descriptors for these tokens, and the equation was transformed into a dependency graph. A combination of interdependent textual descriptors allowed the authors to derive better formula descriptors that were used by indexers for retrieval tasks.
Alternatively, multiple groups have attempted to represent equations as feature sets; soquences of symbols that partially describe an equation's 's visual layout (Zanibbi et al. 2015: Fraser et al., 2018). Krstovski & Blei 2018)) used WORD2VEC in two different manners to find equation embeddings. In one approach, they treated equations as tokens. In their second approach, they treated variables, symbols, and operators as tokens and equations as sequences of tokens. Equation embeddings were computed by taking the average of individual token embeddings. Mansouri et. al. (2019) used modified version of the latter method that extracted features from both the symbol layout tree and operator tree representations of expression. Ahmed et al. (202)) used a complex schema where an equation was embedded using a combination of a pussage-passing network (to process its graph representation) and a residual neural network (to process its visual representation). Peng et al. (2021) trained BERT (Devlin et al. 2019 on mathematical datasets to create a pre-trained model for mathematical formula understanding.
Most of these approaches depend on embedding mathematical tokens and expressions using the surrounding textual information, effectively establishing homomorphism between mathematical and textual information. While these approaches have produced crucial initial results, there are still two important limitations that need to be addressed. Firstly, an embedding scheme should be able to process equations without surrounding text, such as in the case of pure math texts like the Digital Library of Mathematical Functions (DLMF)) Lozier 2003 Socondly, expressions may be written in a multitude of mathematically cquivalent ways (consider x = or sin(x) = cos(x - -)). Embedding methods should recognize that such expressions are mathematically equivalent and should produce similar embeddings. Russin et al. (2021) and [Schlag ct al. (2019) have studied the representations learned by Transformer model, trained on math-reasoning dataset. Lample & CCarrton (2019) showed that their model generated multiple mathematically equivalent solutions for a. first-order differential equation, showing degree of semantic learning. Analogous to our approach, Zhang et al. (2017) trained a SEQ2SEQ model on sentence paraphrase pairs and used the last encoder hiddon state as the vector reprosentation of the input sentence.
Allamanis et al. (017 and Liu (2022) have previously proposed FONET and EQNET-L, respectively, for finding semantic representations of simple symbolic expressions. For EQNET, Allamanis al. (2017) used the tree representation of formula with modified version of the recursive neural network (TrecNN). Their dataset is partitioned into equivalent classes, and they used this information while computing the training loss. They also introduced a regularization term in their objective, called subexpression autoencoder. EQNET-L Liu 2022) extended EQNET by adding a dropout layer and introducing a stacked version of the subexpression autoencoder. However, these approaches only focus on ensuring that the embeddings of equivalent expressions are grouped together. They do not consider or explore semantically similar but non- oquivalent expressions. We define semantic similarity based on the operators present in expressions. For example, we consider sin(x 1) to be more similar to (1: than log(z) or sinh(x + 1) (Section 5.2
To this end, we look at mathematical expressions in isolation without any context and propose approach to semantically represent these expressions in a continuous vector space. Our approach considers semantic similarity in addition to mat hetaticaaaal equivalence. Learning semantically rich representations of mathemat- ical content would be useful in sarming-based applications, like math problem-solving, as well as information retrieval.
3 Proposed Approach
We frame the problem of embodding mathematical expressions as a SEQ2SEQ learning problem and utilize the encoder-decoder framework. While natural language embedding approaches, like WORD2VEC, assume
3

---

Published in Transactions on Machine Learning Research (09/2023)
that proximity in the text suggests similarity, we contend that for mathematical expressions, mathematical equivalence implies similarity. Furthermore, we hypothesize that if we train a SEQ2SEQ model to generate expressions that are mathematically equivalent to the imput, the encoder would learn to generate embeddings that map semantically equivalent expressions together in the embedding vector space. Figure example of our approach. To accomplish this, we need machine learning model capable of generating equivalent expressions and a dataset of equivalent expression pairs (Section
I shows an
Model. In encoder-decoder architectures, the encoder encodes the input sequence, and the decoder is conditionod on the encoded vector to generate an output sequence. In our approach, the encoder maps the input expression to a vector that is referred to as the continuous vector representation of the expression.
There are several choices for modeling encoders and decoders. We use the Transformer architecture (Vaswani et al. 2017) in this work. It has been successfully used in various NLP applications, like machine translation, language modeling, and summarization, to name few, and it has also been shown to work with mathematical expressions [Lample & Charton 2019
Every encoder layer in the Transformer architecture generates hidden states corre- sponding to each token in the input sequence. We use max pooling on the hidden states of the last. encoder layer to generate the continuous vector representation the embedding vector of the input expression. The special tokens, like the start-of-expressson and end-of-cxpression tokens, are not considered for max pooling, Furthermore, we considered the following other candidates to generate the embedding vectors but chose max pooling based on its performance in our initial experiments (1) average pooling of the hidden states of the last encoder layer and (2) the hidden states of the last encoder layer corresponding to the first and the last tokens.
Embedding Vector.
Data Formatting. Mathematical expressions are typically modeled as trees with nodes describing a vari- able or an operator (Ton et al. 1998). Since we are using a SEQ2SEQ model, we use the Polish (prefix) notation sin(a) to convert tree into sequence of tokens. For example, cos(z) is converted to the sequence [div, sin, x. cos, æ! (Figure Th. The primary reason for using the Polish notation is that we refer to the datasets created by Lample Charton (2019) which are Polish notation (dctails in Scction 4). Furthermore, their work shows that the Transformer model is capable of generating valid prefix expressions without requiring constraints also find that invalid output prefix expressions become increasingly rare as the training progresses. Hence, we use the Polish notation for simplicity and leave the exploration of tree-based representations of mathematical expressions for future work. The tokens are encoded as one-hot hot vectors and passed through an embedding layer and positional encoding before being fed to the encoder or decoder.
during decoding. In our experiments,
Decoding. We use the beam search decoding [Keenn 2004 to generate expressions at inference. The beam search algorithm is essential in finding an output with as high a probability as computationally possible. As the main focus of this work is vector representations and the beam search decoding is only used equivalent expression generation, we use a simple version of beam search and do not employ length penalty, diverse beam scarch, or any other constraint during decoding.
4 Equivalent Expressions Dataset
Training our model requires a dataset of mathematically equivalent expression pairs. Working off a known collection of simple mathematical expressions, we use SymPy (Meurer et al. 2017) to create the Equivalent Expressions Dataset, consisting of ~4.6M equivalent expression pairs. This dataset is available publicly our project page.
Generation. To generate valid mathematical expressions, we refer to the work of Lample & Charton (2019). We take expressions from their publicly available datasets and perform pre- pre-procsssing to remove the parts that are not necessary for our use case. This process gives a set of valid mathematical expressions.
4

---

Published in Transactions on Machine Learning Research (09/2023)
Operator Type
Arithmetic
Number of Expressions
1M
Hyperbolic
Dogarithmic/Exponeee
2-
Trigonometric
5-
10k
5
Inu
ppe
Mod
Ajp
dia
2ibs-
yuss
qaant
ueje
U0004
100
595
383
102
103
sqR
1000
Operator
Figure 2: The number of expressions containing an operator in the Equivalent Expressions Dataset. Note
that the dataset represents as x + (-y), hence there is no subtraction operator in the plot.
Table 1:
The number of operators and sequence lengths of training, validation, and test sets of the Equivalent Expressions Dataset.
SET
# OPERATORS
SEQUENCE LENGTH
TRAINING
5.68 + 1.32
16. + 6.28 15.03 + 4.31
VALIDATION
5.60 1.29
TEST
5.98 + 1.22
16.20 + 4.13
For these expressions, we use SymPy to generale matherntticalyy equivalent but visually different counter- parts (see Appendix A for details). For each pair of equivalent expressions X1 and X2, we add examples (x1, X2) and (x2, to the dataset. In our data generation process, we encounter expressions that result in NaN (Not Number) when parsed using SymPy. We exclude examples that contain such expressions.
Dataset. Our training set has 4,662,300 input-output pairs comprised 2,744,824 unique expressions. Note that the number of equivalent expression pairs per unique expression straightforward compute because some expressions result in more equivalent expressions than others. All expressions are univariate and consist of the following operators:
Arithmetic: +, x, abs, pow, sqrt.
Trigonometric: sin, cos, tan, cot, sec, esc, sin -1 (asin), cos (acos), tan (atan).
Hyperbolic: sinh, cosh, tanh, coth, sinh-1 (asinh), cosh -1 (acosh), tanh-1 (atanh).
In, exp.
The variable in the expressions is considered to be positive and real. For embedding analysis, this work only considers simple polynomial and transcendental mathematical expressions, and such, we limit the input and output expressions to have a maximum of five operators. Figure 2 shows the number of expressions containing an operator. Note that one expression can contain multiple operators. The big disparity bet ween the arithmetic and other operators arises because the arithmetic operators occur in almost all expressions. Other reasons for disparitics are the ability of SymPy to generate equivalent expressions and our constraint of maximum of five operators in each expression. Table shows the number of operators in the training, validation, and test. sets and corresponding sequence lengths.

---

Published in Transactions on Machine Learning Research (09/2023)
Our validation and test sets contain single expression per example instead of pair. This is because there are multiple possible correct equivalent expressions for given input expression. At inference, we use SymPy to determine if an output expression is equivalent to the input, instead of fixing one or multiple outputs for an input. The validation and test sets contain 2,000 and 5,000 expressions, respectively.
Note that an expression will only be present in one set, but an equivalent expression may be present another set. For example, if sin(x) present in the training as either input or output, it will not be present in the validation and test sets. But cos - z)) may be present in either validation or test set, but not both.
5 Experiments
We consider two decoder settings for our experiments:
Equivalent expression setting, or SEMEMB, in which the model is trained on disparate but mathe- matically equivalent expression pairs.
Structural setting, STRUCTEMB, in which the model is trained autoencoder to output the same expression as the input.
The proposed SEMEMB embeds an expression based on mathematical semantics, while STRUCTEMB only considers the visual layout of the expression. Though the focus of this work to demonstrate the efficacy of SEMFMB, STRUCTEMB serves important contrast, demonstrating that SEMEMB yields representations that better describe semantics and are superior for clustering, retrieval, and analogy tasks.
5.1
Training Details
We use the Transformer architecture with 6 encoder layers, 6 decoder layers, attention heads, and the ReLU activation. The model dimensions, indicated by Dmodel, are 512 and 64 for the experiments in Sections 5.2 and 5.3 respectively. The layer normalization is performed before other attention and feedforward operations. We use the Adam optimizer (Kingma & Ba 2015) with learning rate of 10-4 and do not consider expressions with more than 256 tokens. Refer to Appendix C for more details.
5.2
Embedding Evaluation
In this section, we evaluate the usefulness of the representations generated by the SEMFMR model. We train SEMEMB and STRUCTEMB on the Equivalent Expressions Dataset. As the dataset consists of equivalent expression pairs, consider each expression an independent example for training STRUCTEMB. The details and results of the training are presented in Section 5.4 For the experiments of this section, we disregard the decoder and use the encoder of the trained models to generate vector representations of input expressions.
Unlike the natural language embedding problem (Wagg et al. 2018 there do not exist standard tests to quantitatively measure the embedding performance of methods built for mathematical expressions. Hence, our analysis in this section must. be more qualitative in nature. These experiments show some interesting properties of the representations generated by SEMEMB and STRUCTEMB and demonstrate the efficacy of the proposed approach. The data used for the experiments are described in their respective sections.
Embedding Plots. To gauge if similar expressions clustered in the embedding vector space, plot and compare the representations generated by the SEMEMB and STRUCTEMB models. For this experiment, we use a set of 7,000 simple expressions belonging to four categories: hyperbolic, trigonometrie, polynomial, Each expression is either a polynomial or contains hyperbolic, trigonometric, logarithmic, or exponential operators. Each expression contains only one category of operators. 35% of these expressions are from the training set and the remaining 65% are unseen expressions. Below are few examples of expressions belonging to each class:
and

---

Published in Transactions on Machine Learning Research (09/2023)
Trigonometric Hyperbolic Polynomial Log/ Exp
Trigonometric Hyperbolic Polynomial Log/Ew
Component 2
Component 2
-4
Component 1
Component
(a) STRUCTEMB
(b) SEMEMB
Figure 3: Plots for the embedding vectors generated by STRUCTEMB (autoencoder) and SEMEMB (proposed equivalent-oxpression method). PCA is used to reduce the dimensionality from 512 to 2. SEMEMB groups expressions with operators of the same lype logether, indicating its ability to understand semantics, whereas STRUCTEMB groups expressions mainly based on their visual structure. The interactive versions of these plots are available on our project page.
Polynomial: x² 2x + 5, 2x +
Trigonometric: sin (x) tan (x)) cos (4x)
Hyperbolic: cosh (I - 4), sinh cos (2))
Log/Exp: 2z-44 log (x + 3)
We use Principal Component Analysis (PCA) for dimensionality reduction. Figure 3 shows the plots for STRUCTEMB and SEMEMB. We observe from these plots that the clusters in the SEMEMB plot distinguishable compared to the STRUCTEMB plot. SEMEMB does better job grouping similar expres- sions together, suggesting its ability to understand semantics. For SEMFMB, there is an overlap bet expressions belonging to hyperbolic and penprtthmmieexxxeentiiial classes. This is expected because hyperbolic operators can be written in terms of the exponential operator. Furthermore, STRUCTEMB focuses on the overall token similarity. For example, representations generated by STRUCTEMB for tan (x (2) x² x),
sinh
2>)))= and log (ar v2 are grouped together. On the other hand, representations generated by SE- MEMB capture semantics in addition to the syntactic structure.
Distance Analysis. To understand the applicability of the proposed approach for the information retrieval task, we perform distance analysis on the embeddings generated by STRUCTEMB and SEMEMB. We use all expressions from the training set of the Equivalent Expressions Dataset the pool of expressions for this experiment. The similarity between two expressions is defined the inverse of the cosine distance between their embedding vectors. We find the five closest expressions to given query expression. Table shows the results of this experiment. We observe that the closest expressions computed using SEMEMB are more similar to the query in terms of the synlactic structure and operators. On the other hand, STRUCTEMB focuses on the overall token similarity. For example, the first query in Table 2 consists of polynomial and trigonometric expressions. The closest expressions computed using SEMEMB follow the same structure, whereas STRUCTEMB seems to put more emphasis on the polynomial multiplier. This behavior is also apparent in the second example. We believe that the ability of SEMEMB to group similar expressions, in

---

Published in Transactions on Machine Learning Research (09/2023)
Table 2: Expressions closest to a given query based on the representations generated by StrucTEMB and SEMEMB. is evident that SEMEMB does a better job at learning semantics and the overall structure of the expressions. Refer Appendix E for more examples.
QUERY
STRUCTEMB
SEMEMB
10x cos (x 10)
4x 2z-1
10x² cos (x + 10)
dx² cos (3x
1x 2x-1
cos (x 10)
1x e -2--
x² cos (xr + 1)
1x2e--
x² cos (xr 30)
x
+
x²
+ log (z)
x² log (ιr) + 1
Vx
x²
+ ex
x² log (1) +
x² log (x) +1
V
x²
+ acos (ar)
x² + log (x) +
log
2
+ sinh (x)
x² + log (z) +
log
2
+ tan (x)
x² + (z)
terms of operators and the syntactic structure, can prove useful in information retrieval problems, where the aim is to find similar expressions to a given query. Refer to Appendix D for more examples.
To quantify the notion of similarity, we use the tree edit distance between the operator trees of two expres- sions We consider 2,000 query expressions and analyze the most similar expression returned by STRUcTEMB and SEMEMB for given query. A cost of one unit is used for inserting, deleting, or updating a token while computing the distance. The tree edit distance between the query expression and the result shows the following:
Scenario 1. If we compare expressions in their original form, STRUCTEMB returns expression strictly closer to the query in 1,044 cases and SEMEMB in 261 cases.
Scenarto 2. When we exclude constant multipliers and additions, i.c., we consider af + b and [(x) to be al a distance of vero, SEMEMB relurns an expression strictly closer to the query in 683 cases and STRUCTEMB in 402 cases. Note that a and b could be simple constants, like 1, v2, etc or constant expressions, like cos(1), cos(log())) etc.
It is interesting to see the change in results as we move from the first scenario to the second one. These results indicate that SEMEMB puts more emphasis on operators compared to STRUCTEMB, and STRUcTEMB considers the overall token similarity. The five closest expressions to the query show a similar trend.
Embedding Algebra. Word embeddings generated by mothods like WORD2VEC (Mikolov et al.) 2013) and GloVe (Pennington al., 2014) exhibit an interesting property that simple algebraic operations on the embedding vectors can be used to solve analogies of the form "x is to yn 22 is to y2". Following along the same lines, we perform simple algebraic operations on the embeddings generated by StrUCTEMB and SEMEMB. For given triplet of expressions ø1, V1, and y/, we compute
2 = emb(21) - emb(y1) + emb(z2)
(1)
where "emb" represents a function that returns the embedding vector of an input expression. We then find an expression with the embedding vector closest to in terms of cosine similarity, excluding 3 Vi, and y2-
To create pool of expressions for this experiment, missing expressions that are required for an analogy. We create 20 analogy examples, 12 of which are based on mathematical identities, and the remaining are based on simple substitutions. Table 3 shows the results for STRUCTEMB and SEMEMB. It is interesting to observe that SEMEMB works for nine examples that are
all expressions from the training set and add any
immplementation Python: https//ppyhhonnhcteeorr///ssss

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3: Examples of embedding algebra on the representations generated by STrUcTEMB and SEMEMB.
The correct predictions are shown in bold.
(PREDICTED)
1/2
22 (EXPECTED)
STRUCTEMB
SEMEMB
co8 (x)
sin
csc (ar)
sec (x)
sec (æ)
sec (x)
cos (x)
sin (2)
(2:
tan (x)
vz
cot (x)
sin (x)
cos (x)
cosh (z))
sinh (2)
tanh (ar)
sinh (æ)
sin (x)
CSC
sec (x))
cos (2)
cos (x)
cos
sin (ar)
csC (x:
cot (a:)
tan (a:)
cos (z)
tan (ar)
sin (-x)
sin (xr
cos (x:
cos (-xr)
sin (I))
cos (-z))
tan (-z) tan (x)
cot (x:)
cot (-x)
sinh (-x)
cot (-x)
sin (-x) sin (x)
cot (x))
cot (-x)
tan (-r)
cot (-x)
sinh sinh (x)
cosh (2))
cosh (-x)
acosh (x)
cosh (-2:
sinh sinh (a)
tanh (1r)
tanh (-x)
tanh (-w)
tanh (-a)
sinh sinh
cosh (x)
cosh (-x)
lan (-x)
cosh (-x)
sinh
sinh
tanh (ar)
tanh (-x)
acosh (x)
tanh (-x)
2
(z)
sin (x)
sin" (x)
x² sin (a))
2
cos (x)
cos (x)
cos² (x)
cos (x)
log (z)
log (x))
log (v2)
log (2")
2
(e*)²
x²
a²c"
2x
e
22
2e"
+
(x + 2)²
tan (x)
x² ++2
log (x)
sin (x)
log (sin (z))
log (sin (a))
sin (log (a))
sin (3r)
2 log (r)
sin (log (I))
log (sin (I))
sin (log (x))
based mathematical identities. It demonstrates degree of semantic learning. STRUCTEMD performs poorly for the identity-based examples and gets four substitution-base examples right, demonstrating that STRUCTEMB understands the visual structure to a certain extent. The results of this analysis further bolster the efficacy of SEMEMB for learning seranttically rich embeddings.
5.3 Comparison with Existing Methods
As discussed in Section prior works (Allamanis et al. 2017 Liu 2022 have examined machine learning models can generate similar representations of mathema ically equivalent expressions. should be noted that these prior works only focus on generating similar representations for mathematically equivalent expressions and do not explore the applicability of their methods to the similarity of non-equivalent expressions. But this historical perspective serves as an established evaluation of the representations generated by SEMEMB and PRRUCTEMB.
If we refer to all mathematically equivalent expressions as belonging to a class, this property is mcasured as the proportion nearest neighbors of each test expression that belong to the same class (Allamanis et al. 2017). For test expression q belonging a class e, the score defined as
Nx(g) nel score: min (k: |cl)
(2)
where Ne(g) represents k nearest neighbors of based on cosine similarity.
We use the datasets published by Allamanis al. (2017) for this evaluation. These datasets contain equiv- alent expressions from the Boolcan (BOOL) and polynomial (POLY) domains. In these datasets, class is defined by an expression, and all the equivalent expressions belong to the same class. The datasets are split into training, validation, and test sets. The expressions in the training and validation sets come from the same classos. The datasct contains two test sets: (1) SEENEqCLASS containing classes that are present the training set, and (2) UNsEENEQCLASE containing classes that are not present in the training (to measure the generalization to the unseen classes).

---

Published in Transactions on Machine Learning Research (09/2023)
Table 4: scores on UNCEEN of the SEM VEC datasets. are from their published work (Allamanis
The scores for EQNET and EQNET-L
2017 Liu 2022].
EQNET
EQNET-L
STRUCTEMB
SEMEMB
DATASET
scores (aecc%
scores
SCOres (%
TRAINING SET SIZE
scores (%)
TRAINING SET SIZE
BOoL8
58.1
30.6
146,485 488
100.0
16, 16,143,072
ONEV-POtY13
90.4
38.7
60,128
99.6
9,958,406
SIMPPOLY10
99.3
40.1
31,143
99.8
6,731,858
SIMPBOOL8
97.4
36.9
21,604
99.4
4,440,450
BoOl10
71.4
10.8
25,560
91.3
3,041 4.041,640 ,640
StMPBooL.10
99.1
24.1
13,081
95.5
1,448,801
BooLL5
75.2
38.3
23,219
36.0
552,642
POLYS
86.2
87.
32.7
6,785
87.3
257,190
SIMPPOLY8
98.9
98.0
47.6
1,934
98.9
113,660
SIMPBoOLL5
85.0
72.1
55.1
6,009
71.1
66,876
ONEV-POLY10
81.3
80.0
59.8
767
74.1
25,590
Bool5
65.8
73.7
36.4
712
57.7
17,934
PoLY5
55.3
5.7
352
45.2
1,350
SIMPPOLY5
65.6
56.3
28.1
156
20.8
882
Our approach requires data to be in the input-output format. For SEMEMB, the input and output are mathematically cquivalent expressions. To transform the training sot into the input-output format for SEMEMB, we generale all possible pairs for the expressions belonging the same class. To limit the size of the generated training set, select maximum of 100,000 random pairs from each class. For STRUCTEMB, the input and output are the same expressions. For this setting, treat each expression present in the training set as an independent example. Table shows the training set sizes obtained by these transformations for SEMEMB and STRUCTEMR, We use the validation and test sets in their original form.
As mentioned in Section 5.1 both STRUCTEMB and SEMEMB are trained with a model dimensionality of 64. To enable fair comparison with the existing approaches, we use the same model configuration and hyperparameter values for all 14 SEMVEC datasets. Refer to Appendix C for detailed description of the model configuration and hyperparameter values.
For evaluating our models, we use the UNSEENEQCLASS test set. Table E shows the scores achieved by our approach. We observe that the representations learned by SEMEMB capture semantics and not just the syntactic structure. should be noted that our approach does not use the distance between representations al the time of training and only trains the model to generale equivalent expressions, whereas the training for EQNET and EQNET-L explicitly pushes the representations of equivalent expressions closer in the embedding vector space. We further observe that SEMEMB performs better than the existing approaches the datascts with sufficiently high mumber of training examples. For other datasets, the model does not perform well but does well on SRENEqCLASS and the validation set (See Appendix D for details). We believe that this is because of not having enough examples in the training set for generalization to the unseen classes, and these datasets may perform better with simpler SEQ2SEQ models and different hyperparameters. We leave this exploration for future work.
In our experiments, STRUCTEMB does not perform well SEMEMB. One possible reason for this behavior may be the less number of examples for training the model in the autoencoder setting. To rule out this possibility, we train models with three model dimensions of 32, 64, and 128 with 179K, 703K, and 2.8M parameters, respectively. We do not observe any significant difference in the performance across these model dimensions. The model also does not achieve good scores on the training and validation sets (See Appendix D for details). Furthermore, SEMEMB performs better than STRUCTEMB on the datasets with comparable training set sizes, for example, consider STRUCTEMB on BoOL8 vs SEMEMB on SIMPPOLY8. With these observations, we conclude with good certainty that the representations learned by STRUCTEMB are inferior to those learned by SEMFMB in terms of capturing semantics.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Table 5: Accuracy of STrUCTEMB and SEMEMB on the Equivalent Expressions Dataset. STRUCTEMB
has an easier task of generating the input at the decoder, while SEMEMB must generate a mathematically equivalent expression.
BEAM SIZE
STrUCTEmB
SEMEMB
0.9996
0,7206
10
0.9998
0,8118
50
0.9998
0.8386
5.4 Training Efficacy
This section presents the efficacy of training SEMEMB and STrUCTEMB models on the Equivalent Expressions Dataset that are used in Section 5.2 It is important to emphasize that SEMEMB and STRUCTEMB are trained on fundamentally different problems. STRUCTEMB an autoencoder and is trained to generate the input expression exactly, while SEMEMB is trained to generate visually different but mathematically equivalent expressions. We show that SEMEMB can learn to generate mathematically equivalent expressions for given input. To evaluate if two expressions
and are mathematically equivalent, we simplify their difference X2 using SymPy and compare it to O, In this setting, if the model produces an expression that is the same as the input, we do not count it as a model success. There are instances in which SymPy takes a significant time to simplify an expression and eventually fails with out-of-memory errors. To handle these cases, we put a threshold on its execution time. If the simplification operation takes more time than the threshold, count it model failure. At inference, we use the beam scarch algorithm generate output expressions. As an output can be verified programmatically, the outputs are correct, we count it as model success.
consider all outputs in beam. If any of
The accuracy of these models is shown in Table and generate the input expression and achieves a near-perfect accuracy with greedy decoding (beam size -). Second, the SEMFMB results demonstrate that generating mathematically equivalent expressions is significantly harder task. For this setting, we observe an improvement of 11% with a beam size of 50. However, we also see increase in the number of invalid prefix expressions being assigned high log probabilities with this beam size. This experiment demonstrates that both STRUCTEMB and SEMEMB are capable of generating mathematical expressions, and SEMEMB can learn to generate expressions that are mathematically equivalent to the input expression. These results also indicate that it is harder to train a model to generate mathematically equivalent expressions compared to training it as an autoencoder.
5 First, we note that STRUCTEMB is easily able to encode
6 Conclusion
In this work, developed framework to represent mathematical expressions in continuous vector space. We showed that SEq2SEQ model could be trained to generate expressions that are mathematically equiva- lent to the input. The encoder of this model could be used to generate vector representations of mathematical expressions. We performed quantitative and qualitative experiments and demonstrated that these represen- tations were semantically rich. Our experiments also showed that these representations were better at grouping similar expressions and performed better on the analogy task than the representations generated by an autocncoder.
We also note certain limitations. There is need for downstream tasks better evaluation metrics to evaluate the quality of the learned representations of mathematical expressions. Though evaluated our approach on the SEMVEC datascts, we could not perform similar evaluation on the Equivalent Exprcssions Datasel due to a limited number of equivalent expressions per expression. Furthermore, the notion of similarity is not well-defined for mathematical expressions. Though the analysis with the tree edit distance shows an interesting contrast between STRUCTEMB and SEMEMB, only looks at tokens and does not consider semantics. More work is needed to establish a comprehensive and quantifiable measure of similarity for mathematical expressions. Lastly, the dataset. presented in this work consists of univariate expressions
11

---

Published in Transactions on Machine Learning Research (09/2023)
that contain simple algebraic and transcendental operators. It may be extended to include expressions with multiple variables and more complex operators. We leave these avenues for future research.
Acknowledgments
We thank Prof. Shaloo Rakheja (University of Illinois Urbana- (aaaaaa-hhmpaignn for providing computing resources for this work. This work also utilizes resources supported by The Grainger College of Engineering, the National Science Foundation's Major Research Instrumentation program, grant #1725729, and the University of Illinois Urbana-Chaie -haapaan. Finally, we thank the anonymous reviewers and the action editor at TMLR for insightful comments and feedback, which helped us improve this work.
References
Salcem Ahmed, Kenny Davila, Srirangaraj Setlur, and Venu Govindaraju. Equation Attention Relationship Network (EARN) A Geometric Deep Metric Framework for Learning Similar Math Expression Embed- ding. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6282 6289, January /TTC///RR08022221...4040 000000000000///RR440000000000000
2021. doi: 10.
Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. Learning continuous semantic representations of symbolic expressions. In International Conference Machine Learning, pp. 88. PMLR, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding. crXfc:1810.04805 [cs/, May 2019.
Dallas Fraser, Andrew Kane, and Frank Wm. Tompa. Choosing Math Features for BM25 Ranking with Tangent-L. In Proceedings of the ACM Symposium Document Engineering 2018, mumber 17, pp. 1-10. Association for Computing Machinery, New York, NY, USA, August 2018. ISBN 978-1-4503-5769-
Liangeai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. Preliminary Exploration of Formula Embedding for Mathematical Information Retrieval: Can mathematical formulae be embedded /os], August 2017.
like natural language?
Patrick lon, Robert Miner, Ron Ausbrooks, et al. Mathematical markup language (mathml) version 3.0. 1998.
Dicderik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs htts/1412.6980 2015.
Philipp Kochn. Pharaoh: beam search decoder for phraso-based statistical machine translation models. In
Conference of the Association for Machine Translation in the Americas, pp. 115 124. Springer, 2004.
Giovanni Yoko Kristianto, Akiko Aizawa, et al. Extracting textual descriptions of mathematical expressions in scientific papers. D-Lib Magazine, 20(11):9, 2014.
Giovanni Yoko Kristianto, Goran Topić, and Akiko Aizawa. Utilizing dependency relationships between math expressions in math IR. Information Retrieval Journal, 20(2):132-167, April 2017. ISSN 1386-4564, 1573-7659. doi: 10. 10.1007 0/0///177910017992000
Kriste Krstovski and David M. Blei. Equation Embeddings. arXiv:1803.09123 fes, statj, March 2018.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Con- ference on Learning Representations, 2019.
Ray Larson, Chloe Reynolds, and Fredric C Gey. The abject failure of keyword ir for mathematics scarch: Berkeley n1 math. In NTCIR, 2013.
Jiaxin Liu. EqNet-L: A new representation learning method for mathematical expression. In 2022 Inter- national Conference on Big Data, Information and Computer Network (BDICN), pp. 547-551. IEEE, 2022.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Daniel W. Lozier. NIST Digital Library of Mathematical Functions. Annals of Mathematics and Artificial Intelligence, 38(1):105-119, May 2003. ISSN 1573-7470. doi: 10.1023/ A:11029111229993911
Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C. Lee Giles, and Richard Zanibbi. Tangent- CFT: An Embedding Model for Mathematical Formulas. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '19, pp. 11-18, Santa Clara, CA, USA, September 2019. Association for Computing Machinery. ISBN 978-1-4503-6881-4 doi: 10.1145/ 3341981.3344233
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cinrman, and Anthony Scopalz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peerj-os.100 1033. URL https: //doi. 7717p//rrr////......
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
Shuai Peng, Ke Yuan, Liangeai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. arxiv preprint 2Xx::2055000777 2021.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Repre- sentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-115
Jacob Russin, Roland Fernandez, Hamid Palangi, Eric Rosen, Nebojsa Jojie, Paul Smolensky, and Jianfeng Gao. Compositional processing emerges in neural networks solving math problems. In CogSci... Annual Conference of the Cognitive Science Society. Cognitive Science Society (US). Conference, volume 2021,
pp. 1767. NIH Public Access, 2021.
Tmanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. Enhancing the transformer with explicit relational encoding for math problem solving, orXiv preprint arXiv:1910.06611 2019.
Moritz Schubotz, Alexcy Grigorev, Marcus Leich, Howard S. Cohl, Norman Meuschke, Bela Gipp, Abdou S. Youssef, and Volker Markl. Semantification of Identifiers in Mathematics for Better Math Information Retrieval. In Proceedings of the 39th International A ACM STGIR Conference on Research and Develop- ment in Information Retrieval, SIGIR '16, pp. 135 144, New York, NY, USA, July 2016. Association for Computing Machinery. ISBN 777-114508-4069-41 doi: 10.1145/2911451.222
Moritz Schubotz, Leonard Krämer, Norman Meuschke, Felix Hamborg, and Bela Gipp. Evaluating and Improving the Extraction of Mathematical Identifier Definitions. In Gareth J.F. Jones, Séamus Law- less, Julio Gonzalo, Liadh Kelly, Lorraine Goeuriot, Thomas Mandl, Linda Cappellato, and Nicola Ferro (eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction, Lecture Notes in Com- puter Science, pp. 82 94, Cham, 2017. Springer International Publishing, ISBN 978-3-319-65813-1. doi: 010077997833319-65811
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackbozNL.P. Analyzing and Interpreting Neural Networks for NLP, pp. 353- 355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-5446. URL htw.aalaannnooolooooog////e/eee
13

---

Published in Transactions on Machine Learning Research (09/2023)
Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Tompa. The Tangent Search Engine: Improved [es], July 2015.
Similarity Metries and Scalability for Math Formula Search.
Richard Zanibbi, Akiko Aizawa, and Michael Kohlhase. NTCIR-12 MathIR Task Overview. pp. 10, 2016.
Chi Zhang, Shagan Sah, Thang Nguyen, Dheeraj Peri, Alexander Loui, Carl Salvaggio, and Raymond Ptucha. Semantic sentence embeddings for paraphrasing and text summarization. In 2017 IEFE Global Conference on Signal and Information Processing (GlobalSIP), pp. 705 709. IEEE, 2017.
Kaizhong Zhang and Dennis Shasha. Simple fast algorithms for the editing distance between trees and related problems. SIAM journal on computing, 18(6):1245 1262, 1989.
14

---

Published in Transactions on Machine Learning Research (09/2023)
A Equivalent Expression Generation
We use SymPy to generate mathematically equivalent expressions for a given formula. We apply the opera- tions shown in Table G get mathematically equivalent expressions. The examples shown the table are intended to give an idea about each function. The actual behavior of these functions is much more com- plex. Refer to the SymPy documentation for more details We use the REWRITE function for expressions containing trigonometric or hyperbolic operators. Specifically, we use this function to rewrite an expression containing trigonometric (or hyperbolic) operators in terms of other trigonometric (or hyperbolic) operators.
Table 6: List of SymPy functions with examples that are used to generate equivalent expressions.
FUNCTION
INPUT
OUTPUT
SIMPLIFY
sin(x)² + cos(2)²
EXPAND
(x 11²
x² + 2x
FACTOR
x² + 5x +
(x + 2)(x + 3)
CANCEL
(x² + +2))
x² +2 cos(2)
TRIGSIMP
sin(x) cot(x)
EXPAND_LOP
In(x*)
2 lu(())
LOGCOMBINE
In(x) ln(2)
In(2x)
sin( cos
cos(x T/2)
REWRITE
sinh(z), tanh
2aanh (1) tanh? (5))
B Equivalent Expressions Dataset
The expressions in the Equivalent Exprcssions Dataset may contain operators from multiple operator types (shown in Table D. For example, 1,424,243 expressions contain operators from a single operator type, 1, 1,18 expressions contain two types of operators, 130,547 expressions contain three types of operators, and on. Table shows few examples from the Equivalent Expressions Dataset.
C Training Details
We use the PyTorch implementation of the Transformer architecture for our experiments with a modified version of the decoder to enable caching to speed up the generation process at inference We use 6 en- coder layers, decoder layers, 8 attention heads, 0.1 dropout probability, and the ReLU activation for our experiments. The layer normalization is performed and feedforward operations. We use the Adam optimizer with a learning rate of 10- fixed seed of 42 for reproducibility, and label smoothing of 0.1 while computing the loss.
the encoder and decoder layers before other attention
For the Equivalent Expressions Dataset, we use model dimension of 512, feedforward dimension of 2048, and a batch size of 256. These experiments were run on two 32GB V100 GPUs. We use early stopping with 300K minimum steps, 1M maximum steps, and patience of 30K steps. In terms of epochs, this translates to 17 minimum epochs, 55 maximum epochs, and patience of epochs for SEMEMB and 28 minimum epochs, maximum epochs, and patience of 3 epochs for STRUCT We evaluate the model at the end of every epoch.
For the SEMVEC datasets, of 512. These experiments were run on one 16GB V100 GPU. We use early stopping with 50K minimum steps, 1M maximum steps, and patience of 20K steps or 2 epochs (whichever is smaller). Table shows these values in terms of epochs for different datasots. We evaluate the model at the end of every epoch.
model dimension of 64, feedforward dimension of 256, and batch size
http:///winttnorrr//o/////////ee Transtormer.htm
15

---

Published in Transactions on Machine Learning Research (09/2023)
Table Examples from the Equivalent Expressions Dataset.
INPUT
OUTPUT
1. 1.302 5nd
1 f (x - 5)
2
1 1 log (a) 2x
1 2 (ar log (x) + dx +)
+ c" 20 +
e sin (a) -0
s00 (r- +)
x² (acos (x) + 1) + x +
x (x acos (x) + xr) + +
6mr"
tTT
1 JT (5x + +z)
x + log (x²) - -1+ 2og (5)
2 + log (25x²) -1
5e log 2
5-126" +log (=))
acos (x) +
3x (1 acos (1r) +
2 (atanh (x) + 1) log (2)
(atanh (x) + 1) log (4)
10w o0th (sin (ain (103)
10
10 (xr tanh (sin (10)) 1)
-20 (x) cosh (x)
-20 cos (x 3) + 20 cosh (æ)
(-x asinh (3)) 11)
(x + 1) (x asinh (3))
cos (a) + acosh (x:)
x ssin (x 1 m)) + acosh (x)
1 1 +1
log (x)
1 1- 1 (x² + + log (1:)
2
æatan (3) 1 atan (3)
(x + 1. (1) atan (3)
Table The minimum, maximum, and patience epochs for early stopping used in the SEm training.
STRUCTEMB MAXIMUM
SEMEMB MAXIMUM
DATASET
MINIMUM
PATTENCE
MINIMUM
PATTENCE
BooLS
175
3,485
70
2
32
2
ONEV-POLY13
424
8,475
170
SIMPPOLY10
820
16.394
328
4
77
2
SIMPBOOLS
1,163
23.256
166
BOOL10
1,000
20,000
400
9
169
4
SIMPBOOL10
1,924
38,462
770
18
354
BOOLL5
1,087
21,740
435
47
926
19
PoLYS
3,572
71,429
1,429
100
1,989
40
SIMPPOLY8
12,500
250,000
5,000
226
4,505
91
SIMPBOOLL.5
4,167
83,334
1,667
382
7,634
153
ONEV-POLY10
25,000
500,000
10,000
1,000
20,000
400
BooL5
25,000
500,000
10,000
1,389
27,778
556
POLY5
50,000
1,000,000
20,000
16,667
333,334
6,667
SIMPPOLY5
50,000
1,000,000
20,000
25,000
500,000
10,000
D Results for the SemVec Datasets
To compute scorex(g) for our model, we use the source code provided by Allamanis et al. (0177
Tables 9 and 10 show the scores achieved by STRUCTEMB and SEMEMB the validation and SEENEqCLASS test sets of the VEC datasets, respectively. For STRUCTEMB, we try three model dimensions of 32, 64,
16

---

Published in Transactions on Machine Learning Research (09/2023)
Table 9: scores (%) achieved by STrUcTEMB and SEMEMB on the validation sets of the SEMVEC datasets.
SMrUCTEMB
DATASET
SEMEMB
DAODAL 32
DADDEL 64
DISOEL
128
BooL8
36.2
36.3
35.5
100.0
ONEV-POLY13
38.3
39.0
38.5
99.9
SIMPPOLY10
37.5
31.5
34.0
99.9
SIMPBOOLS
42.3
46.7
44.5
99.8
BoOL10
9.6
9.9
9.1
93.9
SIMPBOOL10
23.2
26.6
27.2
96.3
BOOLL5
47.4
47.6
47.7
98.2
POLYS
36.4
36.5
36.0
98.6
SIMPPOLY8
36.5
35.4
34.6
99.8
SIMPBOOLL5
75.0
76.1
74.4
99.5
ONEV-POLY10
42.1
46.3
42.5
83.0
Bool5
42.4
42.1
35.4
81.0
POLY5
7.4
43.2
SIMPPOLY5
20.0
28.8
32.5
43.8
Table 10: (%) achieved by STRUCTEMB and SEMEMB on SEENEqCLASS of the SEMVEC datasets.
STRUCTEMB
DATASET
SЕMEMB
DASCEL 32
DaLDEL 64
DIODEL 128
BooL8
36.1
36.1
35.5
100.0
ONEV-PotY13
38.0
38.5
38.1
99.9
SImPPotY10
37.7
32.2
34.5
99.9
SIMPBOOLS
41.6
46.1
43.5
99.8
BooL10
9.6
9.8
8.9
94.0
SIMPBOOL10
23.0
26.8
27.3
96.5
BOoLL5
40.5
42.0
42.2
68.7
POLYS
34.9
35.0
34.5
95.2
SIMPPOIN8
38.5
36.7
36.1
99.6
StMPRooL.L5
58.2
59.3
58.3
80.9
ONEV-POLY10
36.1
39.6
37.9
77.3
BooL5
41.1
43.2
34.6
80.2
PoLy5
14.7
8.2
4.2
46.2
SIMPPOLY5
30.9
32.6
42.2
47.9
and 128 to rule out the effect of underfitting and overfitting while training the model. Table 11 shows the scores achieved by these models on UNSEENEQCLASS. It can be seen that changing the number of parameters in the model does not have significant effect on the scores.
We also perform an evaluation of compositionality on the SEMVEC datasets Allamanis et al. 2017 For this evaluation, we train our model on a simpler dataset and evaluate it on a complex dataset. For example, model trained on BoOL.5 evaluated on BoOL.10. We use UNSPENEoCLLASS for the evaluation. The results of this experiment are shown in Table 12
E
Distance Analysis
Table 14 shows more examples of the distance analysis discussed in Section 5.2
17

---

Published in Transactions on Machine Learning Research (09/2023)
Table 11: scores achieved by STRUCTEMB on UNSEENEQCLASS of the SEMVEC datasets for different model dimensions.
DATASET
Dxcoa 32
Duumm 64
D
128
BooL8
31.1
30.6
25.9
ONEV-POLY13
37.7
38.7
38.4
SIMPPOLY10
47.4
40.1
44.9
SIMPBOOLS
33.1
36.9
34.1
BooL10
10.7
8.6
SIMPBOoL:0
20.5
24.4
25.4
BooLL5
35.6
38.3
38.2
POLY8
32.2
32.7
32.1
SIMPPOLYS
50.4
47.6
47.3
SIMPBOOLL5
55.2
55.1
54.9
ONEV-POLY10
58.3
59.8
59.7
BooL5
36.7
36.4
28.1
Poiv5
14.9
5.7
6.6
SIMPPOLY5
18.8
28.1
17.7
Table 12: Compositionality test of SEMEMB. score;(%) achieved on UNSEENEQLLASS of SEMVEC dataset reprosented by To by model trained on the datasct represented by FROM. For scores achieved by EQNET, refer to Figure 7 in Allamanis et al. (2017).
FROM + To
scores (%)
BooL5 BooL10
12.5
BooL5 BoOLS
32.5
BooLL5 BooLS
29.3
ONEV-POLY10 ONEV-POLY13
29.8
POLY5 - POLY8
32.9
POLY5 + SIMPPOLY10
POLYS ONEV-POLY13
29.9
POLYS + SIMPPOLY10
58.9
POLYS SIMPPOLY8
98.9
SIMPPOLY5 - SIMPPOLy10
45.5
SIMPPOLY8 SIMPPOLY10
65.9
F Equivalent Expression Generation
For the Equivalent Expressions Dataset, Table 13 shows the accuracy of STRUCTEMB and SEMEMB the validation set of the Equivalent Expressions Dataset with bean sizes of 1, 10, and 50.
Table 13: Accuracy of STRUCTEMB and SEMEMB the validation set of the Equivalent Expressions Dataset.
BEAM SIZE
STRUCTEMB
SEMEMB
0.9995
0.7615
10
0.9995
0.8365
50
0.9995
0.8665
18

---

Published in Transactions on Machine Learning Research (09/2023)
Table 14: Exprcssions closest to given query computed based on embeddings generated by STRUCTEMB and SEMEMB.
QUERY
STRUCTEMB
SεmEMB
x log (x) + 21r
210x + 210 (1)
accsh (z) + 21r
60z + 20 20 (x)
21.r 3 (x))
2r 16 lag (a)
11x sin (x)
10x cos (z) + 21x
21x + 21 cos (x)
zxe" + 21.r
70z + 10 (a:)
cosh² (x)) + cosh (x)
cosh 2 (x)
+ cosh (x:)
sin (x) cos" (z)
sin (a:) + cos (2)
sin (a)) + cosli" (x)
sin (a) + lan 2 (æ)
sin" (a) + sin (a)
sin (a) + sin (1)
(x)
(z)
acos² (x) + acos (x)
log (x) sin (x)
x²
x² +
x² atanh (a:)
+5
x² +5
x² + tanh (x)
+ x²
x² sinh (a)
x² + x²
x² cosh (x)
(a² 1)
V x² log (9)
atanh (x² +1)
log (a² log (9)
atanh (x² +1)
sinh (x²
log (x² log (8)
atanh (x +5)
log (x" ex)
atanh (x +
+e+
atanh
(x² +4)
log (Ir) log (2) +
x² 2x
log (2x) te
x² + 2x + 14e
2x +1+1 ++
+e
2x e" +
(-r log (2x)))
x² + 3x + em
(-x + 22)
3x + e" -
log (x*)
+ _
sin (x) + ee 102
log 5
+
x + sin (a)
10r
sin" (x) t
log [x
-
20r + sin (z)
log + +e
10x + sin (z)
log (2x)
+
x + e 10 + sin (x)
x² acosh (x)
x² log (x)
3 log (x)
x log (x)
x² log (x)
x² atanh (æ:
2x log (æ)
acos (x)
3x" log (x)
x² asinh (x)
1x log (2r)
+ log (2)+3
log
x
+
J.
10 +
ese (I0)
log (x + vx2 +
log
Fec (10)
log
csc (10)
log
37 #4.24142loy
(20)
log
+ (250)
log
+
2 (2)
19


Reviews:
[{"review_id": "KxuDaAPtfZ", "reviewer": "Reviewer_xb2r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper proposes an approach, EXPEMB-E, to represent mathematical expressions *independent from* their surrounding text to support cross-disciplinary retrieval purposes. \n- The paper demonstrates the advantage of the proposed EXPEMB-E approach via a suite of qualitative evaluations including 1) the emergence of clusters correponding to different operator types; 2) the alignment between embedding distance and semantic distance for retrieval tasks; 3) the possibility to perform arithmetics in the representation space and retain semantic meaningfulness. E.g. x1 is to y1 as x2 is to y2 --> derive x2 as emb(x1) - emb(y1) + emb(y2).\n- The paper quantitatively benchmarks the performance of EXPEMB-E on a previous equivalent expression dataset (Allamanis et al. 2017) and shows that EXPEMB-E outperforms both the baseline EXPEMB-A and previous math-expression representation methods.\n- The paper releases a corpus of equivalent transcendental and algebraic expression pairs.\n\nWeaknesses\n- The released training corpus only contains univariate algebraic and transcendental operators. Future work is expected to extend it to multi-variate expressions with more complex operators.\n- There lacks a comparison between encoding standalone expressions vs. encoding expressions conditioned on the surrounding text.\n- The Transformer enc-dec was trained from scratch whereas there could be a potential to utilize mathematical capacity already present in large-scale pre-trained Transformers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new approach for representing mathematical expressions in a continuous vector space, such that distances in the embedding space reflect more about semantic similarity in math than similarity in the surface form.\n\nThe paper proposes to obtain such representations by training an enc-doc Transformer to generate mathematically equivalent expressions, and demonstrates that the resulting encoder hidden states (EXPEMB-E) turn out to be reasonable representations for downstream retrieval and analogy tasks. The training task of generating mathematically equivalent expressions is shown to be more effective than an autoencoding task (EXPEMB-A)  because the resulting representations are better structured according to operator types.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper mentions that encoding expressions using surrounding textual information could potentially hinder downstream retrieval performance. But there isn't a comparison with such a text-conditioned baseline. \n\nIt would also be great to investigate if it's easier to adapt a pre-trained Transformer into encoding expressions by injecting certain abilities through fine-tuning.\n\nThe proposed adjustments won't affect my overall rating for acceptance, but will greatly strengthen the work in my view.", "broader_impact_concerns": "No particular concern on ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lGLqa8nN4y", "reviewer": "Reviewer_evvu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper is attempting to solve an important problem - finding a useful embedding of mathematical expressions.\n2. Paper is clearly written and easy to understand. The authors do a good job of motivating the problem of mathematical expression representation and why it differs from language modeling.\n3. The authors include a variety of ablations and compare against reasonable baselines.\n4. The authors include a sincere discussion of limitations in the conclusion.\n\n**Weaknesses**\n1. The notion of \"similarity\" the authors are after is unclear. This is the most serious issue with the current work, and I go into more detail on this in the \"Requested Changes\".\n2. The models chosen seen suboptimal for the author's purpose. At a high level, ExᴘEᴍʙ-A is trained as an autoencoder on mathematical expressions without any signal for equivalent mathematical structure, and ExᴘEᴍʙ-E essentially treats the problem as autoregressive language modeling on strings which start with the Polish notation form of an expression followed by \"=\", but neither of these seem ideal for learning representations of mathematical expressions with some level of semantic similarity. (More on this in \"Requested Changes\".)\n3. The qualitative analysis of the embeddings is a bit weak. For example, the authors state \"ExᴘEᴍʙ-E groups expressions with operators of the same type together, indicating its ability to understand semantics, whereas ExpEmb-A groups expressions mainly based on their visual structure\", but no crisp definition of visual structure was given. Does this mean, for example, that ExᴘEᴍʙ-A considers $(x+1)^2$ to be more similar to $(x+5)^2$ than, say, $x^2 + 2x + 1$? Formalizing this would also present more quantitative analyses.\n4. The \"analogy\" task also fits in the above point, but deserves it's own discussion. I think the authors would be hard-pressed to defend this interpretation of the vector space structure. It seems as though the analogy has two interpretations: (1) $y_1$ is equivalent to $x_1$, with some particular style of equivalence semantically clear to the authors which they would like to apply to $y_2$ to obtain $x_2$, or (2) $y_1 = f(x_1)$, and the authors would like $y_2 = f(x_2)$. Consider what the vector $v = x_1 - y_1$ means in both cases. The first interpretation is hard to justify - recall that semantically equivalent expressions are already intended to be nearby each other. What happens if we apply this vector more than once? For example, if $v$ is the vector for $x_1 = \\sin(-x)$ and $y_1 = -\\sin(x)$, then if we add this to $\\cos(x)$ we expect to end up at $\\cos(-x)$. What happens if we add this to $\\cos(-x)$? It seems reasonable to expect that we end up at $\\cos(x)$ again, but clearly we can't. The second case has a similar problem: consider $v$ to be the vector for $x_1 = x^2$ and $y_1 = x$, then the vector $v$ would represent the sqrt operation (as indicated by the fact that the desired $x_2 = \\sin^2(x)$ for $y_2 = \\sin(x)$). If we assume such a $v$ exists, we can easily see it must be $0$, as applying it to $y_2 = 1$ would need to result in $x_2 = 1$, but then this is problematic since it would imply that the square of any expression has the same representation as the expression itself. There are other issues apart from this (eg. what happens when the same operation is achievable by two different vectors, such as a $v'$ representing division by $x$, in which case $v = v'$ if we consider $x_1 = x^2$ and $y_1 = x$, but also $v = 2v'$ if we consider $x_1 = x^4$ and $y_1 = x^2$) which makes the analogy setup quite problematic.\n5. The Equivalent Expressions Dataset seems to be a rather weak evaluation of the desired goal. It seems like the expressions are fairly clearly separated into a few classes (based on Table 1) making it easy to distinguish many non-equivalent expressions purely based on the tokens they contain. Additional statistics (eg. number of equivalence classes, histogram based on size of equivalence class) would be useful in getting a high-level view of the difficulty. Some examples of equivalence classes added in the appendix would also be appreciated. Personally, I feel like the dataset itself misses the mark when it comes to semantically equivalent mathematical expressions. ML models which operate on statistical co-occurrance patterns are ill-suited to encoding equivalence which can be obtained symbolically, as is the case with the samples in this dataset. More often than not the difficulty is not that equations are written unreduced or expanded, in which case they can be simplified to the same form. In most cases there is a canonically or conventionally \"simplest form\", but different authors may use different names for variables, functions, and notation, which seems better suited to statistical co-occurrance models but which this dataset does not address.\n6. The model performance on the only task which arguably has a well-defined objective (identifying equivalent mathematical expressions, section 5.4) is relatively poor. It outperforms the baseline on 6/14 datasets, 2 of which only marginally so. There is a correlation with training set size with the exception of the BᴏᴏʟL5 dataset, which is particularly notable since it so drastically underperforms whereas it achieves perfect results on Bᴏᴏʟ8. In my opinion, it would be worth trying to understand why it achieves perfect results on Bᴏᴏʟ8 beyond scale of the training set - i.e. did the model actually learn to generalize appropriately, or is it just a very easy dataset/task for some reason?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors main goal is to obtain representations of mathematical expressions which capture some notion of semantic similarity. To do so, they first represent a mathematical equation as a sequence of tokens using Polish notation, and then propose two models, both using a transformer architecture: ExᴘEᴍʙ-A is trained in an autoencoder fashion, whereas ExᴘEᴍʙ-E is trained using pairs of equivalent mathematical expressions. (More on this later.) The authors then evaluate these models in several ways:\n\n1. **Equivalent Expression Generation:** Given an input expression $x$, they obtain $K$ output expressions ${y_k}_{k=1}^K$ using beam search on the trained transformer, and evaluate how often an expression in this set is mathematically equivalent to $x$, in the sense that $y_k - x$ can be symbolically simplified to $0$.\n2. **Embedding Evaluations:** The authors qualitatively inspect a 2-dimensional PCA projection of the encoding of a variety of mathematical expressions and observe some patterns. Given a query, they also inspect the top 5 embeddings which are closest in cosine distance to the query embedding, and note some patterns. Finally, they explore what is captured in the vector space structure of the embeddings using an analogy-like task.\n3. **Similar Representations of Mathematically Equivalent Expressions:** They compare against existing baselines which attempt to learn representations of expressions such that equivalent expressions are most similar to one another. Given a test equation, they calculate the proportion of the $k$ nearest neighbors which are mathematically equivalent to the test equation. The authors underperform the baselines in the majority of these evaluations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The notion of similarity that the authors are attempting to encode is never defined. The authors do explicitly exclude contextual similarity (i.e. expressions used in similar contexts have similar representations) and say they want to consider \"semantic similarity\" in addition to mathematical equivalence (end of section 2), however exactly what this means is unclear. The lack of clarity on this point becomes more problematic in the qualitative evaluation (section 5.3). For example, in Table 4 it is stated that \"ExᴘEᴍʙ-E does a better job at learning semantics and the overall structure of the expressions\", and one of the supporting examples is that ExᴘEᴍʙ-E $4x^2 \\cos(3x-1)$ matches to expressions which generally have the form $\\alpha x^2 \\cos(x + \\beta)$, whereas ExᴘEᴍʙ-A matches to expressions which are of the form $4x^2 e^{\\alpha x+\\beta}$. It is unclear to me why the former is preferred to the latter, and to my understanding there is no clear definition of similarity aside from some gut feeling. This could be excused if the paper was proposing to learn some general representations from a large amount of unsupervised pretraining and then leverage those representations for some downstream tasks, in which case the lack of a well-defined notion of similarity could be justified by saying that a rigorous definition is not possible but clearly the representation is useful in tasks for which some notion of similarity is required, however this is not present in this work. I could imagine several notions of similarity in mathematical expressions. For example, edit distance of the Polish notation where different penalties apply for different types of edits seems closest to what the author's mean by \"similarity\", and this could be formalized. In other settings, one may be more interested in predefined distances on function spaces (eg. $\\mathcal L^k$ distance). In practice, the sort of function representation I find myself most often in need of is the ability to describe a set of properties and obtain functions with those properties, and this sort of similarity would seemingly be currently best supported by the sort of contextual embedding eschewed by the authors at the start. This point is a significant weakness of the paper, in my opinion, and the authors should either seek to define what sort of similarity they are after or defend the lack of such a definition by demonstration of the usefulness of their representation on other tasks of interest which can be well-defined.\n2. It is somewhat unsurprising that the representations for ExᴘEᴍʙ-A are relatively weak, as (to my understanding) the model is simply an autoencoder and does not receive any information about mathematical equivalence whatsoever. As such, the relatively strong performance of ExᴘEᴍʙ-A on any tasks should be viewed not as a strength, but rather as a useful critique of an evaluation which actually does not require knowledge of mathematical equivalence, and instead can be solved simply by token similarity. In this light, the relative performance of ExᴘEᴍʙ-A compared to ExᴘEᴍʙ-E in table 3 vs. table 6 makes complete sense. As such, I take issue with the statement at the end of page 6 that \"both ExᴘEᴍʙ-A and ExᴘEᴍʙ-E are capable of learning the mathematical structure\", and would encourage the use of ExᴘEᴍʙ-A not as a baseline representation model but rather as a model which is able to discern which test examples do *not* require mathematical understanding in order to get them correct (eg. as a consequence of unintended dataset / task bias).\n3. I would like clarification on how ExᴘEᴍʙ-E is trained. As I understand it, given a set of equivalent expressions $\\{x_k}_{k=1}^K$ then during training ExᴘEᴍʙ-E would attempt to model $f(x_i) = x_j$ for every $i \\ne j$. One way to interpret this is thinking of ExᴘEᴍʙ-E as an autoregressive language model, where the data looks like strings $x_i = x_j$, and thus for input $x_i$ we want the continuation $x_j$. Is this correct? If so, why not try a contrastive training process, i.e. minimize $d(f(x_i), f(x_j))$ while maximizing $d(f(x_i), f(y))$ for some $y$ which is not equivalent to $x_i$? It is not that your current approach for ExᴘEᴍʙ-E is wrong, per-se, but the properties of natural language vs. mathematical expressions seem significantly different enough that it almost seems like banging a nail in with a drill. A contrastive loss seems much more in-line with your stated objective. Moreover, if you had a crisp definition of similarity, you could even sample $y$ in accordance with this (with probability inversely proportional to the edit distance, for example).\n4. As mentioned above, the qualitative analysis needs to be improved, and the analogy task needs to be rethought.", "broader_impact_concerns": "I do not feel there are any ethical implications of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JufzKTIUvh", "reviewer": "Reviewer_WSit", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- S1: The approach is clear and intuitive - it makes a lot of sense as a way to create semantic representations.\n- S2: The dataset that they produce will likely be useful to future researchers working in this domain\n- S3: The analyses are extensive, interesting, and convincing for the purpose of demonstrating that the learned representations capture semantics. I especially enjoyed Table 4 and Table 5 (the analyses of nearest neighbors and analogies). Another point I especially liked about these analyses is the comparison between ExpEmb-A (a baseline model trained on autoencoding) and ExpEmb-E (their actual model, trained on producing equivalent expressions), since this comparison provides a nice control.\n- S4: The paper is clear and well-written.\n\nWeaknesses:\n- W1: I believe that the framing is somewhat misleading with respect to prior work and novelty. The intro starts by saying “there exist no major algorithms for processing mathematical content”, which seems unnecessarily dismissive of the several well-received papers that are cited which do process mathematical content, such as Lample and Charton, Allamanis et al., and Liu 2022. Similarly, parts of the introduction seem to suggest that all prior methods for representing mathematical expressions have encoded them based on context alone, when in fact there have been several prior papers which encoded them based on their structure and/or semantics - these include the three papers mentioned earlier in this paragraph, as well as the following 2 papers which perform analyses that are in a similar spirit to the submitted paper’s analyses: https://arxiv.org/abs/1910.06611, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8491571/. Also, regarding Lample and Charton’s work, even though their focus is not representations, I think that their model’s performance (particularly Section 4.6 “Equivalent Solutions”) gives reason to believe that their system learned reasonably sophisticated semantic representations, which is why I included it in the list of papers that are relevant for learning semantic mathematical representations.\n- W2: The motivation for this work is unclear - why do we want semantic representations of mathematical expressions? I can imagine answers to this question (e.g., standard LLMs are bad at math, and we could solve this problem by giving them better representations), but it would be nice to directly motivate this in the paper instead of leaving it to the reader to figure it out.\n- W3: All of the evaluations are intrinsic (just analyzing the structure of the learned representations) rather than extrinsic (testing the usefulness of the representations for downstream tasks). It would be nice to also have extrinsic evaluations. However, I do not believe that this weakness is a serious one, as the paper already has a sufficient quantity of analyses. (Also, note that the authors acknowledge this concern). \n- W4: There is no description of the strategies used by the relevant prior work (Allamanis et al., and Liu 2022). This paper compares to those papers, which is excellent, but it would be good to also briefly state how the proposed model differs from what they did, to motivate why we would expect the proposed model to differ from these prior models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an approach for creating representations of mathematical expressions. These representations are intended to be semantic - i.e., they capture the expression’s meaning, rather than its surface form (e.g., the same representation should be produced for “tan(x)” and “sin(x)/cos(x)”). The authors achieve this by training encoder-decoder models whose input-output pairs are pairs of semantically equivalent expressions.\n\nThe contributions are:\n- C1: They create a dataset of equivalent expressions which can be used to train systems under their approach\n- C2: They show that encoder-decoder models can succeed on this task of producing expressions equivalent to the input.\n- C3: They perform several analyses of the representations (e.g., nearest-neighbor analyses and analogies) showing that the representations learned in their approach capture the meaning of expressions\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- RC1: Relating to W1 above, I would encourage the authors to revise the framing to more clearly acknowledge the existence of prior work with similar aims (i.e., prior work on processing mathematical expressions). For a paper to be valuable, there is no need for it to be the first one to have tackled a problem, so modifying the framing in this way would not weaken the paper. In fact, it would strengthen it: As it currently stands, I could easily imagine readers objecting to this intro, whereas an intro that frames prior work more clearly would not give readers a reason to feel dissatisfied. I think that a straightforward way this modified framing could work would be: “There remains no consensus about how to process or represent mathematical expressions. Some prior works have encoded expressions based on context, but this is suboptimal because [list reasons]. We instead aim to encode expressions based on their meaning. Some prior work (e.g., Lample and Charton) has indirectly achieved this by training systems to perform mathematical tasks, which likely requires some sort of representation of the semantics, but the focus in such work is not representation but rather performance on the task. We instead focus directly on the problem of representation learning. A couple of prior works have also worked in this area - here is a description of how we differ from them: [describe differences from Allamanis et al. and Liu 2022, and any other relevant ones].” There is no need to frame it in this specific way I have listed, but addressing this concern in some way would be important for securing my recommendation.\n- RC2: Relating to W2 above, it would be nice (though not necessarily critical to securing my recommendation) to clarify the motivation for producing semantic mathematical representations. In addition to the motivation I listed under W2, another possible motivation that I can imagine is: “There are lots of domains (e.g., language and vision) for which we would like to generate representations that capture semantics. However, for those domains it is hard to evaluate the success of a proposed approach because we don’t have a clear understanding of the correct meaning. In contrast, we do understand the meaning of math very well; so by working with math, we can perform a controlled and principled analysis of the broader question of semantic representations, a question that is relevant to many domains.” In this regard, it might be worth citing the following work, which uses the same basic approach but for language rather than math (i.e., training an encoder-decoder model to produce semantically equivalent expressions): https://arxiv.org/abs/1809.10267\n- RC3: Relating to W4, it would be nice to have a description of how the proposed model differs from prior models that are compared to in experiments. This change would not be critical in securing my recommendation, but would strengthen the work.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I believe the manuscript can be repositioned (see claims & evidence).  
Decision: Accept (poster)