MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training
Authors: Jonathan Drechsel
Anja Reusch
Steffen Herbold
Venue: Accepted by TMLR ()

Abstract: Mathematical formulas are a fundamental and widely used component in various scientific fields, serving as a universal language for expressing complex concepts and relationships. While state-of-the-art transformer models excel in processing and understanding natural language, they encounter challenges with mathematical notation, which involves a complex structure and diverse representations. This study focuses on the development of specialized training datasets to enhance the encoding of mathematical content. We introduce Math Mutator (MAMUT), a framework capable of generating equivalent and falsified versions of a given mathematical formula in LaTeX notation, effectively capturing the mathematical variety in notation of the same concept. Based on MAMUT, we have generated four large mathematical datasets containing diverse notation, which can be used to train language models with enhanced mathematical embeddings. Experiments show that models trained on these datasets exhibit new SoTA performance on mathematical retrieval tasks. We publish our code, generated datasets, and pretrained mathematical models: https://github.com/aieng-lab/math-mutator.

Published in Transactions on Machine Learning Research (06/2025)
MAMUT: A Novel Framework for Modifying Mathemati- cal Formulas for the Generation of Specialized Datasets for Language Model Training
Jonathan Drechsel
brratten.drechscl@uurr
Facully of Compuler Science and Mathematics, University of Passau, Germany
Anja Reusch
anjj@Beammps..cchhsinn..cc
Taub Faculty for Computer Science, Technion Israel Institute of Technology, Israel
Steffen Herbold
sieffen.bertooddBBuii-ssssuu
Facully of Computer Science and Mathematics, University of Passau, Germany
Reviewed on OpenReview:
Abstract
Mathematical formulas are fundamental and widely used component in various scien- tific fields, serving a universal language for expressing complex concepts and relation- ships. While state-of-the-art transformer models excel in processing and understanding natural language, they encounter challenges with mathematical notation, which involves complex structure and diverse representations. This study focuses on the development of specialized training datasets to enhance the encoding of mathematical content. We intro- duce Math Mutator (MAMUT), a framework capable of generating equivalent and falsi- fied versions of given mathematical formula in ВТEX notation, effectively capturing the mathematical varicty in notation of the same concept. Based MAMUT, have gen- erated four large mathematical datasets containing diverse notation. Experiments show that models trained on these datasets exhibit new SoTA performance mathematical re- trieval tasks. We publish our code, generated datasets, and pretrained mathematical models: https:/ http://gttthbb.ooo com aicng-lall math-mutator
1 Introduction
Mathematical formulas are fundamental and widely used component in various scientific fields, serving as a universal language for expressing complex concepts and relationships. Their context-dependenn symbols, nested operations, and diverse notations pose distinct challenges for machine learning models due to their
symbolic and structural differences from natural language (Zanibbi al. 2020 Peng et al. 2021
Raw Datascts
AMPS, ARQMath (Section 2.2
NMFT (Section
\frac(d)((x) f(x) (lim Ato 0) \frac(((*h)-
Generation Formula Versions
VG (Section
5.1 PalseVG (Section
5.2
g' (y) Alim_{ \to 0) \left(g(hty) 8(y) (rghh)))
Generated Datasets
MF, MT, NMF, MFR (Section
Figure 1: MAMUT: Modifying formulas to generate large and diverse mathematical datasets.

---

Published in Transactions on Machine Learning Research (06/2025)
Dataset
Description
Example(s)
Mathematical
Mathematical formulas with high variance
x.2 = x2+N
Formulas (MF)
(a- a))
Mathematical Texts (MT)
Texts combining natural lan- guage and mathematical for- mulas
Identify En-o L) where Un+1 = (1+ yn) and
L = L+1. Let y > 2 and let f(y) = (1 3) Let
("()) be the n th iterate of f(V) Let L bc
Named Mathe-
High variance formulas of fa- mous named identities
Name: Pythagorcan Thm., Formula: c² b² + a² Name: Binomial Formula, Formula: (a + 2)2
matical Formulas (NMF)
2² + a +2 2
Mathematical
Pairs of formulas with labels indicating identical or different mathematical concepts
Formula
..2- .... n = n!, Formula 2: m! = I=== h, Label: Equivalent
Formula Retricval
(MFR)
Formula 1: a² + b2 c², Formula a² 2 e² Label: Not Equivalent
Table 1: Overview of generated datasets. The examples are shown as rendered ETEX
(Vawwani et al. 2017 in natural language tasks, they encounter challenges in comprehending mathematicall notation (Hendrycks el al. 2021: Gong et al.
Despite the success of transformer-basod language models
2022 Petersen et al. 2023 Dao & Le 2023 Shen et al. 2023 Reusch et al., 2024 Qiao et al. 2024
These challenges stem from the complex formula structure, diverse formula representations, and ambiguous
implicit semantics (Peng et al. 2021 For example,
~bty/2-1ac involves nested operations, while 2a
different notations, such as ;, r/v, y, and x v.y² can represent the same mathematical relationship, alongside the contextual meanings of symbols (e-g., as an index or imaginary unit) further complicate the understanding. These difficulties highlight the need for rich, specialized datasets to train models for mathematical content. However, existing datasets face scalability constraints due to expert curation or lack diversity in problem types and notation.
To address the need, we propose a framework, Math Mutator (MAMUT), for generating high-quality and diverse mathematical formulas to enhance the training and comprehension capabilities of mathematical language models. MAMUT allows for the creation of mathematically equivalent formulas (EquVG) and challenging non-equivalent ones that appear similar (FalseVG). This includes random alterations in variable and function identifiers and variations in ыТTX notation that leverage mathematical properties such commutativity and symmetry. Additionally, we extend this approach to text containing mathematical L&TEX notation, ensuring consistent changes in identifiers and notation styles across textual contexts. We apply MAMUT to generate four datasots (sce Figure [ and Table D designed for the training of mathematical language models, e.g., for further mathematical pretraining on equation completion tasks. We apply this mathematical pretraining to several models and evaluate them on mathematical retrieval tasks, showing that models trained MAMUT-enhanedh data outperform existing mathematical models.
2
Related Work
This section covers language models, datascts, and data augmentation techniques in mathematical contexts.
2.1
Mathematical Language Models
The success of Transformer-based models (Vaswani et al. 2017), such as Bidirectional Encoder Representa- tions from Transformers (BERT) (Devlin et al.] 2019), led to the development of domain-specific models, including SciBERT for scientific texts (Beltagy al.J 2019) and CodcBERT for programming (Feng et al. 2020). These models improve over general-purpose models by training on domain-specific data. Simi- larly, specialized models have been developed for mathematics, such as MathBERT (Shen et. al. 2021
2

---

Published in Transactions on Machine Learning Research (06/2025)
Query
Relevant Documents
Not Relevant Documents
(a + 6)2 a² + +abb 62
a² + 2ab + b² = (a + 6)2
(a 10+b) 6)2 + a² = 2ab + b²
(c+ d)² =2² + 2cd + de
(a+ = a² + 2ab + a²
(a + b)2 = a² + +b² + 2ab
(a + +)² = a² + b
a² + 13 c2
c² a² + 1/2
a² + 12 +
Pythagorean Theorem
a² 62 c²
Pythagorean Identity
> 1
Ek=1 h-
E n= x
f(x)) = lime (1xth)-1(a)
d d= gz)) = limd=00 ((o+e)-
lim f(x)= = 0
d
Table 2: Examples for MIR queries including relevant and not relevant documents. Note that Pythagorean
Idenbity (sin" + (x))(x) 1) is not relevant for the query Pythagorean Theorem (a² + b² =c).
MathBERTa Novottý & Stefánik 2022), and others (Peng et al. 2021 Reusch et al,, 2022 Liu et al, 2023 Shao et al. 2024). They typically employ additional mathematical tokens and were pretrained on mathematical datasets (see Section 2.2
A key application of mathematical language models is Mathematical Information Retrieval (MIR) (Dadure lt al.) 2024 Zanibbi et al.) 2025). where the goal is to retrieve relevant documents based on user's query, where both may contain mathematical content (see Table 2 for examples). Traditional MIR systems rely
keyword matching or simple embeddings (Kim et al. 2012 Greiner-Petter et al.) 2020 while more
sophisticated techniques leverage explicit mathematical knowledge, such operator trees or formula unifi- cation (Kristianto et al.) 2016 Mansouri et al.] 2019 2022b Aizawa & Kohlhase 2021 Peng et al. 2021). Transformer- based models offer new possibilities for MIR by addressing key challenges such as integrating natural and mathematical language, directly processing LΛTEX input, and handling diverse notations. As a
result, maal language models have been successfully adapted to MIR (Novatný Stefánik, 2022 Reusch et al., 2022 Zhong al. 2023).
Despite their promising performance, specialized mathematical models still face challenges in understanding mathematical notation (Gong et al. 2022 Shen et al., 2023). especially when it comes to handling variable names and recognizing mathematical equivalence beyond superficial textual similarities (Reusch et al. 2024). This motivates the creation of specialized datasots that reflect the unique roles of variables and aim to improve mathematical modeling.
2.2 Mathematical Datasets
The need for mathematical models has led to the development of various collections aimed at enhancing and evaluating language model capabilities in context of mathematics. Manually curated datasets like MATH (Hendrycks et al., 2021), ; GSM8K (Cobbe et al., 20021) and MathOdyssey (Fang et al. 2024) test problem- solving skills but are typically small, and reliant on expert input. Synthetically generated datasets like the Mathematics Dataset (Saxton et al.] 2019), AMPS (Hendryeks et al. 2021), and HARDMATH (Fan et al,, 2024) offer scalability but may lack diversity in problem topics, as they rely on generation rules, although they can produce a wide varicty of similar but distinct problems (with changing numbers, symbols, ...). which can be beneficial for models learning to generalize across formula represenlations. Datasets like NTCIR (Zanibbi et al. 2016) and ARQMath (Mansouri et al. 2022a),, sourced from the existing repositories arXiv, Wikipedia, and the Mathematical Stack Exchange, provide a broad range of real-world mathematical problems. However, they lack controlled variations of specific formulas, an important aspect for training MIR models to classify hhether two symbolic representations mathematically equivalent.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Natural Language
Mathematical Language
Purpose
General human communication, includ- ing opinions and feelings
Precise description of mathematical concepts
Vocabulary Large set of words (language depen- dent), sometimes with ambiguous mean- ing (e.g,, love, happy, data)
Small set of well-defined symbols (e.g., 2, +, IR, sin, V, oc, f) and terms (e-g. Eigenvalue, Derivative, Field) with precise meanings
Grammar
Rather flexible
Strict rules
Clarity
Often imprecise, open to interpretation
Single, unambiguous interpretation
Evolution
Evolves over time naturally, new words, phrases, and idioms emerge or disappear Linear structure in sentences and para- graphs using standard formats
Evolves slower, changes are introduced by mathematicians and are backward compatible Requires specialized formats (e.g., ITTEX) to represent complex notation in structured way
Writing
Style
Table 3: Comparison of natural and mathematical language.
2.3 Mathematical Data Augmentation Techniques
Recent advancements in mathematical data augmentation have introduced various innovative methods aimed al enhancing the diversity and depth of training materials. InfinityMath (Zhang et al. 2024) utilize GPT-4 (Achiam et al. 2023) to transform specific mathematical problems into generic templates. These templates can then generate multiple variations of the original problem, altering numerical values structural com- plexity, thereby increasing the dataset's variety. Similarly, Li et al. (2024) propose a method to formalize mathematical problems written in natural language, alter the difficulty by adjusting the problem's 's opera- tions, and then informalize these changes back into natural language using GPT-4, preserving mathematical integrity across different. levels of complexity. MathiGeni (Lu et al. 2024) augments step-by-step solutions by generating modified candidate solutions with a Llama model (Touvron al. (0023) with verified correct- ness, and then back-propagating these solutions to modified question. You et al. (2024) augment data by applying different strategies, including rephrasing and reorganization with LLM, and question alteration.
These approaches primarily focus on diversifying problem content rather than varying mathematical notation (c.g., (a + b)2 = a² + 2ab + 12 vs. (o + y)² = x² + y² + 2yx). In contrast, Reusch et al. (2024) explore variable renaming in training data to prevent models from taking shortcuts in problem-solving, such as relying only on variable overlap. Building on this idea, our study enhances mathematical formula diversity through substitutions and other techniques.
3 Natural and Mathematical Language
Mathematical language differs fundamentally from natural languages such as English, German, or Chi- nese. While natural language is used for general communication and often conveys subjective information, mathematical language serves a highly specialized purpose to prociscly describe mathematical topics, such as definitions, theorems, and proofs. It consists of both symbolic expressions (e.g., a2+ = o² and f sin()), dr:) and specialized terminology (e.g., derivative and eigenvalue). Despite their differences, natural and mathe- matical languages share some structural similarities. Both use symbols arranged in a syntax that conveys meaning, and both can be represented textual form. However, there are some key differences (aany et al. 2010 Scarlatos & Lan 2023) summarized Table 3 A crucial challenge for mathematical language models is symbol abstraction. In mathematical expressions, certain symbols act essentially wildcards and can be replaced without changing the mathematical meaning. These symbols are either voriables (e.g., T, a, A) generic functions (e.g., f, 9 ip) i.c., functions not tied to a specific mathematical object (c.g., Euler's Gamma function [(=)). For example, model should recognize that the first binomial formula,
(a + b)2 a² 2ab + 62
(1)
4

---

Published in Transactions on Machine Learning Research (06/2025)
Names
Factorial, Definition factorial n! 1 1.2.....R
Version 1
Version 2
n! = I
Version 3
Vn N: (n ( +)! (n 1) n!, 0!
Version 4
For any natural number n, we have n!! is defined as n! == II
Version 5
For any natural number n, n! can be obtained by multiplying all natural numbers from 1 to together.
Similar Formula
Binomial Coefficient Formula
False Version 1
n! 1- -2-3-4-- Vn E N (n + 1)! (n 1) n!, 0! 0 II - Σ, E, "natural" "real"
False Version 2
Falsifying Replacements
Table 4: Example entry for the definition of a factorial from the NMFT dataset (partially).
is equivalent to (c+ + 2ed +d, despite different variable names. In contrast, (a+ a² + a² uses the same variables but in mathematically non-derivable way. Likewise, the modified formula (a- 2 + b2 + 2ab appears different from Eq. (1. but is, in fact, mathematically equivalent.
Another important aspect the structure mathemattcal formulas. Consider the two formulas 2 and x². Assuming character-wise ыТтХ tokenization, both formulas use the same tokens but in a different order. A model should not treat x² as equivalent to 2 (but instcad to x z)). These structural variations highlight the need for a model that goes beyond simple token matching and actually understands mathematical meaning. Transformer language models (Vaswani et al. 2017) have shown their capabilities in modeling natural language, hence, it is worth exploring their potential to precisely capture mathematical language.
4 Named Mathematical Formula Templates (NMFT)
Previous datasets provide formulas and mathematical texts with significant variance across a wide range of mathematical topics. For the purpose of our proposed data augmentation methods introduced in the next section, is necessary to parse formulas inlo symbolic expressions. Real-world datasets contain formulas with various notations, some of them might be parsed incorrectly, or the parsing even fails completely. Therefore, created dataset consisting of only a few but high-quality parsable formulas. This datasct includes 71 well-known mathematical identities that are easily recognizable and associated with one multiple names. For example, a² + 122 = e² represents the Pythagorean theorem, while (a + b)2 a² + 2ab + 12 represents the first binomial formula (Eq. ( Since the mathematical formulas are associated with their names, we call this dataset Named Mathematical Formula Templates (NMFT), as the formulas serve as templates for deriving modified versions. An example entry can be found in Table g and Table lists all identities.
Each identity provides multiple representations, such Va, E R: (a+ a² 2ab 6² as another, more detailed version of the first binomial formula. Additionally, some representations are provided descriptive text, e.g., "In right-angled triangle with side lengths b, and c, where represents the length of the hypoteruse, the equation a² + b2 = holds true". Others paraphrase formulas textually, c.g., 'a² + b² is equal to ", reinforcing associations between the equals sign = and "equals". These textual versions intentionally exclude the name of the identity to make MIR tasks harder, where the name serves as the query. For each provided identity version, the variables and function symbols are explicitly given to assist the parsing and version generation. To enhance the generation of challenging falsified versions, similar-looking formulas are provided (e.g., the first binomial formula for the Pythagorean theorem, both ident ities contain
5

---

Published in Transactions on Machine Learning Research (06/2025)
multiple powers of two), hints to falsify any given representation by string replacement (e.g., removing "righl-angled" to lalsify the previous descriptive example of the Pythagorean theorem). The descriptive text versions have been partially generated by using GPTT-355 (Brown et al. 2020) and all manually verified for validity. Typically, we call entries of NMFT and of datasets generated from it formulas, but both, pure mathematical formulas and lextual descriptions of formulas are meant.
5 Math Mutator (MAMUT)
Our goal is to create high-quality, large, and diverse mathematical datasets to enhance mathematical mod- eling. We introduce Math Mutator (MAMUT), a framework consisting of two core algorithms designed to generate both equivalent and falsified versions of a given formula. The first algorithm, Equivalent Version Generation (EquVG), presented in Section 5.1 automatically generates various versions of a given formula, expanding the training data and enabling the model to learn math-specific language rules, such as treating variables placeholders that can be substituted without changing the validity of an expression. The second algorithm, Falsified Version Generation (FalseVG), introduced in Section 5.2 slightly modifies formulas to create mathematically non-equivalent versions of the original formula, offering challenging negative examples for MIR tasks.
5.1 EquVG: Variations of Mathematical Formulas
The key idea of this section can be summarized as follows: Given mathematical formula, our aim is to generate mathematically equivalent variations of this formula, called equivalent versions. For instance, consider the formula
(a ba+ a² +2 +2.00 b2.
(2)
In this context, we observe that all the following formulas describe the same mathematical relationship, namely the first binomial formula:
=-+a"==000 b2.
(3)
(a+ -a+=)==a-00 a-.a a
a- b+bb
(4)
a² + 22 (0-
(5)
22+ d.
2A+ X2 7 7..
By applying both additive and multiplicative commutativity, one can derive Eq. (3) from Eq. (2) In Eq. the exponentiation a² replaced by its definition a a. Since equality symmetric relation, equations remain valid after interchanging the sides, as done in Eq. (5). The final two equations can be obtained from
Eq. 12 by substituting variables. This section is dedicated to the automated generation of such equivalent versions. Note that for complete mathematical expression, it would be necessary to specify the range of values (e.g., of variables) for which the statement holds. For example, complete expression of Eq. (2) could be Va, b E IR (a+ 6)2 a² 2 a-aaa b + 62. However, in practical applications like MIR systems, the shortened version Eq. 22 may also be used, for the sake simplicity. Therefore, the somewhat less precise mathematical formulations in Eqs. 2 - are often sufficient.
The complete workflow of our proposed Equivalent Version Generation (EquVG) algorithm is depicted in Figure The input consists of formula written in LTTEX format. To implement transformations, seen in Eqs. 2
can identify two steps: the substitution of symbols and the modification of the mathematical notation. For both of these purposes, it is helpful to represent the formula not as string but as structured data format that captures the mathematical relationships and dependencies. This representation is achieved by parsing the ыАТЕX formulas into a symbolic expression format, essentially creating an operator tree. The symbolic representation categorizes elements into numbers, variables, functions, and other mathematical objects, establishing structured relationship between them. This organization enables the identification and substitution of variables (x, \alpha, ...) and generic functions (f, g, ...) to derive mathematically

---

Published in Transactions on Machine Learning Research (06/2025)
Input Formula
\drac(d)dx3 f(0) Alim_[h Ato 0) \frac(f(xth) - f(x))(h)
</> T Parsing
FQU/DERIV/FUNCIE, VAR(x)), VAR(x)). LIM(DIVISUBPPUNNCCC ADD(VAR(*), VAR(h))), FUNC(r, VAR()))) VAR(K),, VAR(h), INT(O))
c Symbolic Expression
2 Symbol
VAR(x) - VAR(y), FUNC()) - FUNC(K)
Substitution
EQUIDERIV/FUNCC& VAR()),, VAR())), LM(DIVSSUU(FUNCCC ADD(VAR(y), VAR(b))), FUNC(E, VAR()))) VAR(h)), VAR(h), INT(O)))
New Symbolic Expression
Randomized
FIT Printing
Generated Formula
g (v) = Alimh Ato e Aleft(a(hry) - g())
Figure 2: Visualization of the EquVG algorithm.
equivalent representation using different symbols. This substituted expression then converted back into ыАTК format during the printing process, which includes the desired modifications of mathematical notation, such writing instead of a². In the following, we will provide more detailed explanation of the three steps of EquVG: parsing, substituting, and printing.
</> ETEX Parsing Parsing a 1TPX formula into a symbolic expression presents certain challenges. For instance, if a letter precedes parentheses, it can be interpreted either as a multiplication (with omitted multiplication symbol, e.g., u(z + y) = v. (x + y)) or as a function call (v(x + y)). The symbol e could be either a variable or Euler's number e 2.718. Likewise, the symbol might function as a variable (c.g., as summation index in Ei=1 (2) or as the imaginary unit, sometimes expressed in 1Tрр as i (\mathrm(i)) to avoid ambiguity. It is crucial for our purposes to determine whether a symbol is substitutable variable constant. Otherwise, the imaginary unit might be incorrectly substituted by another symbol, resulting in non-equivalent expression. We have addressed this issue, partially by applying heuristics that consider the context (i = within the formula indicates a variable, while in indicates the imaginary unit, as in e!*). Furthermore, we introduce a safeguard to handle cases where the parser is uncertain about whether to treat a symbol as variable or the imaginary unit. In these cases, the symbol is represented in a way that prevents substitution while maintaining its appearance as the plain i, without enabling complex unit formatting options. Despite these measures, parsing can still fail in cases with unusual malformed notation.
The formula parsing can be conceptually expanded to include the parsing of text containing WTEX formulas. Such texts are referred to as mathematical terts in this study. The text parts remain unchanged during substitution and printing, only the formula parts are processed consistently by EquVG as shown in Figure 3 This allows to consistently change symbols in a mathematical text throughout all its formulas. Within a mathematical text, formulas are defined as text in between dollar signs (), as used TATEX documents to write inline mathematical formulas.
Symbol Substitution The symbolic expression format allows the substitution of symbols by simply replacing all occurrences of a given symbol within the expression. This is conceptually related to a-conversion in definitional equality (Nederpelt & Geuvers 2014). where renaming of bound variables (e.g., k in n == Ih k) preserves the meaning of an expression. Our framework generalizes this to also include free variables (e-g-; n. in the same factorial definition).
The generation of substitution (i.e., mapping of symbols) involves two steps. Firstly, a subset of all symbols in the expression is randomly selected. Secondly, a symbol is chosen for each selected symbol. The aim of the substitution process is to generate diverse formulas that are similar to formulas occurring in real-world seenarios. It is important to note that, intuitively, Eq. (6) with variables e and d appears more familiar for a binomial formula Ihan Eq. 7 which uses Greek and uppercase Latin letters (A and Z) that are not commonly used as variables in the context of binomial formulas. When variables are selected entirely at random from a uniform distribution over all Latin and Greek letters, unfamiliar symbol usage is more likely. This motivates the introduction of symbol groups, which categorizes similar variables or functions together. The defined symbol groups can be found in Table 11 along with description of a typical mathematical
7

---

Published in Transactions on Machine Learning Research (06/2025)
The derivative of a function Sf3, referred to as $xfrac(dX(d)F(()s, is definned as the limit of (x - ((x) as Shs approaches $05.
Input Formula
</> 16Tx Parsing
TEXT<" derivative of function ", FUNC(F), ", referred as ", DERIV(PUNC()) VAR(x)), ", is defined as the limit of DIVISUB(FUNCEE, ADD(VAR(x), VAR()))), FUNC(F, VAR(x))), VAR(h)), as ", VAR(h), " approaches ", INT(O), "!)
d Symbolic Expression
Symbol
VAR(x) - VAR(y), FUNC(F) - FUNC(g)
Substitution
The derivative of a function ", FFUNC(), ", referred to ", (ERI)) VAR(y)), ", is defined as the limit of DIV(SUB(PUNCC, ADD(VAR(y), VAR(h))), FUNC(g, VAR(y))), VAR(h)), as ", VAR(h), approaches INT(O). "!"
A New Symbolic Expression
+ Randomized
ITEX Printing
The derivative of a function $gt, referred to as $g (v)s, is defined as the limit of $Nleft(e(hty) - g(/)righht//s as Shž approaches ses.
Generated Formula
Figure 3:
Visualization of the EquVG algorithm for a mathematical text.
context for each group. For instance, we have the group of indices ki 1 and group of vectors {u, u, w). It is worth noting that variables can belong to multiple groups. Given symbol that should be substituted with a new symbol, all symbols from the relevant symbol group(s) are candidates. Additionally, the most common variable I is candidate in every variable group to reflect its common usage. To add variety, a random symbol may be added by chance to the set of candidates, selected from commonly used lowercase or uppercase Latin letters or lowercase Greek letters. Symbols that refer to constants in certain contexis, such as e, i, and π,, are excluded. Given the set of candidates, a random candidate is chosen. However, it is sometimes necessary (or least useful) to make the symbol selection dependent on multiple substitution symbols. For instance, consider the Fundamental Theorem of Calculus: f fx) da = F(b) F(a). Here, we have two generic functions, and F. Whenever a symbol has related variants in the formula, such as uppercase and lowercase forms or correspouding Greek letters (e.g., a, A, at), the algorithm automatically preserves these relationships by restricting possible substitutions accordingly. For instance, substituting f and F by g and G, respectively, yiclds an equivalent version J g(z) dz = G(b) - G(a). Again, this rather mathematically imprecise but aligns with the implicit. assumptions on mathematical notation found in real-world datasets. Another possible generated version is foj f(x) dr F(a2) - F(1)) where and are substituted by indexed variables a and 02 respectively. In cases where multiple variables of the same variable group appear in the same formula, the generation algorithm may randomly perform such indexed substitutions. The indexing enforces the model not only to attend to the variable itself but also its modifications, in this case, its index.
B Randomized IТTX Printing In the final step of EquVG, the symbolic expression is converted back into ETΣX format. To further increase the variety of generated formulas, the ETTEX printer makes randomized printing decisions. These variations can be categorized into two main sources: mathematical and ETTX notation. The parsed and printed formulas in Figure 3 are illustrating these differences. For example, the input text used the explicit notation for derivative, (fraccd)(dx)((t) while the printed substituted expression uses the shorthand notation g' (y). We developed list of equivalent mathematical notations, where the printer randomly seleets one of the available ones for printing. As another example in Figure 3. instead of the fraction notation with \frac, the printer used the forward slash / to denote division. Since addition commutative, hty is printed instead of yth. These examples represent mathematical variations, as they express a mathematical concept in an equivalent way. In contrast, the usage of Nleft and Aright commands represents ETEX variations, since these commands are not essential for mathematical reasons but only for differently rendered text. In addition to the already covered examples, the randomization of the UTEX printer includes the notation of
cqualities (x y vs. y = x),
inequalities (x 0 vs. 0 < x):
multiplication symbols (a \cdot b vs. * b vs. a Atimes vs. ab),
8

---

Published in Transactions on Machine Learning Research (06/2025)
divisions (2/n Acdot n^(-13 vs. \frac(2)(n) vs. \frac2n),
integer powers (a*3 a^2Ncdot vs. Acdot Acdot a),
inverse trigonometric functions (Xasin(x) vs. \arcsin(x) vs. \sin((-1](xx ((s1))))))(-1)),
higher order derivatives (f** (x) vs. (*((3)](x) \frac(("3>(dx<3) f(x)),
expected values (\mathbb(E)[X] vs. \operatorname(E][X] vs. E[X]),
matrix delerminants (A) vs. IAI),
binomial coefficients (\binom(n){k) vs. {n Achoose k)),
empty sets (Vemptyset vs. Avarnothing vs. V(V), and
natural logarithms (VIn(x) vs. Alog_e(x)).
As real-world data uses different styles of notations, language models should be capable of understanding all commonly used notations. This is similar to the notion of synonyms in natural language. The randomized ВТЕX printing provides an automation to diversify training data, such that models can learn the different notations. The combination parsing, substituting, and printing results as part of EquVG is a powerful tool to increase the training data size significantly. Additionally, research has shown that using training data with substituted query-document pairs for MIR helps the model to less focus on shallow features such variable overlapping (Reusch et al. 2024), confirming the usage of substitutions in EquVG.
5.2 FalseVG: Generating Challenging Negative Examples
We believe that a classification task determining whether two formulas describe the same mathematical concept helps the model to encode mathematics more effectively. To train models on such a task, we require both positive and negative formula pairs, similar to a MIR training. While positive pairs are often readily available in datasets, identifying mcaningful negative pairs is more challenging, as datasets rarely contain explicit negative examples.
A common approach to extract negative pairs is random sampling from datasets by pairing two random formulas. However, this may lead to simplified feature extractions. For instance, the model might learn to simply check for the presence of an important function, like whether both formulas contain the determinant function \det. Given a random negative document, a naive classifier that chocks a determinant is part of the formula would likely perform well, due to the rarity of the determinant function across most mathematical contexts. The language model may adapt to this behavior during training.
To prevent models from learning such easy shorteuts instead of the true semantic understanding, rescarchers have successfully used challenging negative examples in other domains (Cai & Liu 2020 Qiu et al. 2021 We introduce the Falsified Version Generation (FalseVG) algorithm to generate falsified versions of given formula, meaning a similar-looking but not mathematically equivalent formula. Since the formulas are already parsed into a symbolic expression for EquVG, we can simply use and modify this representation. We have developed and implemented eight modification strategies, which are described below. Table [T] provides illustrative examples of each strategy. Similar to EquVG, these strategics can also be applicd mathematical texts, by applying the strategies to the text's formulas.
= Equality Falsifying equality can be achieved by inserting removing a term one side of the equality. This can be done cither at the outermost level (e.g., changing sin(x:) = to sin(z) + = ...) or within a sub-expression (e.g., changing sin(x) = ... to sin(x + 1) = ..... When inserting a term, the algorithm selects either a subexpression from the entire formula, a random new variable, or a random number. Importantly, the algorithm avoids modifications that will not change the validity of an equality,
such adding zero or multiplying by one. This strategy enforces the model focus on the entire formula and term dependencies.
9

---

Published in Transactions on Machine Learning Research (06/2025)
Input Formula
\Frac(d)[dx) f(x) \limm[ Ato 8y \frac[f(xsh) f(x)](h3
</> 1TTEX Parsing
EQU/DERIV/FUNCTE, VAR(x)), VAR(x)), LINDDIISSUBEUUCCCE, ADD(VAR(x), VAR(h))), FUNC(F, VAR(x))), VAR(h)), VAR(b), INT(O))
Symbolic Expression
Equality
Remove the dividend VAR(h) of quotient DIV
E Strategy Selection and Application
Inequality
No change
A Variable
Replace VAR(h) by VAR(a) in limit LIM
Constant
Replace INT(0) by random number INT(1)
X Swap
Swap the order in subtraction SUB
EQU(DERIVIFUNCIE, VAR(x)), VAR(x)), LIM VAR(x)), FUNC(F, ADD(VAR(x), VAR(h)))), VAR(h), INT()))
Falsified Symbolic Expression
Symbol
VAR(x) + VAR(y), FUNC(F) FUNC(g)
Substitution
EQU(DERIV(FUNCIE, VAR(y)), VAR(y)), LIM(SUB(FUNCIE, VAR(y)), FUNC(g, ADD(VAR(y), VAR(h)))), VAR(a), INT(I))
Substituted Symbolic Expression
Randomized
IKΙІ Printing
Generated Formula
\limma \to g(y) g(h+y)
Figure 4: Visualization of the FalseVG algorithm.
S Inequality To falsify an inequalily, we simply invert the inequality symbol. Thus, the symbol < is replaced by > and vice versa. The same replacement holds for 2 and <. The not equals symbol * can be replaced by =; but not vice versa (as = indicates an equality). Similarly to the strategy equality, the model is forced to encode long-tere ter-term dependencies using this strategy.
x Swap The strategy swap involves altering umary and binary functions. Unary functions such as sine, square root, or logarithm get replaced by different random unary functions. In the case of binary non- commutative functions, we swap the order of the two arguments. These non-commutative functions are subtraction, division, and exponentiation (e.g., x² becomes 2*). These changes enforce the model to rely on the order of operands rather than just token occurrences in a random order.
A Variable The strategy variable essentially aims to split a single variable (e.g., a in (a+ a² 22a 6º) into two (e.g., into and in (c+6)) 6)2 a² + 2cb + 6²). Specifically, if a variable occurs least twice in the formula, it might be randomly replaced by another variable for proper and nonempty subset of its occurrences (i.e., at least one occurrence is replaced and at least one occurrence remains unchanged). This strategy enforces the model to check for a consistent use of symbols in the entire formula.
oo Constant The strategy constant focuses on numbers (1, 2, e, म, o0. as well as variables that are typically considered be constant within an expression, such as the upper limit n of an indexed sum like E 2. These constants are replacod by other constants enforcing the model to learn what tokens a certain formula should contain.
$ Distribute
The strategy distribute is inspired by the distributive law, fundamental mathematical rule relating two binary functions. A standard example for real numbers is that multiplication distributes over addition since x <-( = x holds for all real numbers x, y, z. This rule motivales this strategy, which applies a modified distributive law to non-distributive functions. Specifically, for a unary function f and a binary function @ in infix notation, the relation fax v))) f(x) f(y is (falscly) assumed. We use addition and multiplication as binary functions and the logarithm, factorial, power with fixed base, and trigonometric functions for the unary function. This readily results in examples where commonly known identities are
10

---

Published in Transactions on Machine Learning Research (06/2025)
falsified, e.g.; the falsified product of powers rule is 2 22 2 (instead of the correct .2 2)) The falsified sine additivity yields sin( = sin(2) sin (instead of sin(x+ v) = sin(z)coo(y)) + cos(()) sin
This strategy enforces the model to notice the presence of parantheses and to enhance its understanding of operator relationships, including precodence.
Manual While all previous strategies focused on modifying a formula by applying generally valid transformation rules to falsify it, this strategy relies on monual transformation or replacement rules. These rules refer to the specifically newly created NMFT dataset (see Section 4 The rules can be explicitly given falsified versions (e.g., Vn N: n! = 1-2-n), references to different but similar formulas (e.g., law of cosines for the Pythagorean theorem), or falsifying replacement rules. For example, formula replacement might change Vn E N to Vn E R if the quantified term only holds for natural but not for real numbers. These rules are also applicable to mathematical texts, where, for instance, natural can be replaced by real.
The simplest approach to generate a falsified formula is to use a random formula, mcaning an earlier generated equivalent version of a different formula. This approach is especially important to increase the models' robustness in real-world applications, where most of the input pairs are not inherently challenging.
Random
The complete FalseVG algorithm is summarized in Figure 4 It involves applying a random subset of the strategies to a parsed symbolic expression. Note that some strategies are not applicable to certain formulas, resulting in no changes. However, if at least one strategy succeeds, a falsified symbolic expression is generated. Finally, a random symbol substitution and randomized ETEX printing are performed to create the final formula as string, identically to EquVG.
6 Generated Datasets
This section presents four datasets generated using MAMUT employing EquVG and FalseVG. Our imple- mentation, built on SymPy (Meurer et al. 2017), is detailed in Appendix B.1. Table 5 summarizes key statistics of the generated datasets, including Hugging Face identifiers, while Appendix B.2 reports example
entries. All entries ensure uniqueness at the string level. Two of the generated datasets (NMF and MFR) are based on the specifically created NMFT dataset (see Section 4 while the other two datasets (MF and MT) are derived from two existing diverse sources that combine natural language with mathematical notation: Answer Retrieval for Questions on Math (ARQMath) (Mansouri et al. 2022a) and the Khan Academy prob- lems in Auxiliary Mathematics Problems and Solutions (AMPS) (Hendryeks et al. (021). ). While we focus on these sources, MAMUT is applicable to any mathematical corpus containing ыTTX notation. ARQMath, sourced from the Mathematics Stack Exchange, benefits from a user-rating system that ensures high-quality discussions and problem-solving content. The Khan Academy problems in AMPS provide structured exer- cises used for educational purposes. Example datasct entries are shown in Appendix A
Mathematical Formulas (MF) This dataset consists exclusively of mathematical formulas extredted from AMPS and ARQMath, enriched with variations by the EquVG algorithm. However, not all formulas from these raw datasets are included in the MF datasct. Only formulas are selected being suitable for a Masked Language Modeling (MLM) task (Devlin et al. 2019), where a masked token's 's value can be concluded by the remaining context of the formula. For example, a masked formula such as π > [MASK] has infinite algebraic solutions, like 3, 0, or any other value that can fill the masked position in a mathematical valid sense. Therefore, the formulas are restricted to equalities and implications to ensure meaningful inferences. sin(z)
Additionally, only generally valid formulas according to SymPy's formula solver (e.g., tan(x) = cos(s) as this equation holds for all E R) are considered as input formulas for EquVG to ensure high data quality. In case of an equation without general validity (e.g., x² == 2) but with existing solution(s) found by SymPy, the equation can be transformed into an implication (c.g., x² = 2 = x = - - or x = V2). A few examples of extracted formulas are (original ETTX formatting is preserved):
tan(x) = sin(x)/ (in(z))coo((z)
2
1 2
6
5 -
5 - -
=10 =0
11

---

Published in Transactions on Machine Learning Research (06/2025)
Name Hugging Face Identifier
Original (atasct(s)
Raw Entrics
Generated Versions
Ø v.p.f.
Max
v.p.f.
MF
ddrg/matt__oormllas
AMPS
30,985
958,735
30.9
101
ARQMath
55,894
2,257,820
43.3
101
Both
82,765
3,198,106
38.6
101
MT
adrg/math_tens
AMPS
62,099
2,542,015
40.9
101
ARQMath
690,333
4,480,369
6.5
96
Both
752,428
7,022,384
9.3
101
NMF
odrrg/namedd_math_ffrmuJ...
NMFT
71/ 522
23,707,392
333,906
400,000
MFR
adegg/maattu_formllaarrrrreva
NMF
71/ 522
23,702,560
334,092
400,000
Table 5: Summary of the generated datasets. The abbreviation e.p.f. stands for versions per formula. For MF, the Generated Versions values do not sum up from AMPS and ARQMath to Both due to duplicate removal. The raw values of NMF and MFR refer the number of mathematical identities and the total number of provided version templates of these identities, respectively.
3
2
1
13
13 13'
2m
(e = (-1)² 1,
V25 = 5,
(n + 1) x n (n 1} (n + 1)!
(n + 1) (n- 1)!
Mathematical Texts (MT) While the previous dataset MF focuses exclusively on mathematical formu- las, MT focuses on the relationship between mathematical formulas and natural language. Similarly MF, MT is generated using the AMPS and ARQMath datasets, along with applying EquVG, which consistently changes variable names across the text and prints the IΛTEX formulas in different ways. We only consider texts containing at least five formulas. Questions and answers of ARQMath are treated as separate lext, while the AMPS data is treated as single text where question and hints are concatenated. To ensure high data quality, only answers from ARQMath with at least five upvotes are includod. We generate up to 100 additional versions for each suitable input.
Named Mathematical Formulas (NMF) This dataset associates the name of mathematical identity with either its formula or a describing text. It is derived from NMFT by applying both, EquVG and FalseVG, resulting in diverse positive and negative pairs. This data could be used to train classifier that predicts whether formula is valid representation of an identity's 's name, using Next Sentence Prediction (NSP)-like task (Devlin et al. 2019). In typical NSP task, each positive pair is matched with random negative pair, which changes when the positive pair reused. To enhance training, we create an imbalanced dataset with four times more negative than positive pairs. This allows for training where positive pairs remain unchanged across epochs, while negative pairs vary between epochs (and remain challenging). With a maximum of four epochs, the model encounters unique negative pairs in each iteration. NMF originates from 71 mathematical identities, each with multiple base versions used to generate up to 400k versions per identity. About 60% of the NMF entries are textual descriptions, and the rest are pure mathematical formulas. For 20 of the 71 mathematical identities, fewer than 400k versions exist, as they offer fewer possibilities for generating versions, such as limited substitution options or fewer opportunities for creating randomized TΛTEX.
Mathematical Formula Retricval (MFR) This dataset consists of formula pairs, classified as either mathematical equivalent or not. It constructed by pairing each true formula version from NMF with an equivalent version and four falsified versions of that identity, all randomly sourced from NMF. This approach
12

---

Published in Transactions on Machine Learning Research (06/2025)
preserves the positive-to-negative pair ratio while ensuring that negative pairs remain challenging. MFR can be used to train a MIR system for querying relevant formulas based on a similar formula, like NSP task.
7 Experiments
In this section, we evaluate the effectiveness of MAMUT-generaatee data through mathematical pretrain- ing and fine-tuning based on the four generated datascts introduced in the previous section. Our goal is to demonstrate that MAMUT enhances mathematical encodings, even when applied to models already pretrained on mathematical corpora.
7.1 Setup
We consider the BERT- model (Devlin et al. 2019 as a general general-purpose purpose baseline and evaluate two math- specific variants initialized from BERT: MathBERT (Shen et al., 2021), trained on data from mathematical curricula, lextbooks and arXiv, and Math-PPretrainee--BBBIII (MPBERT) (Reusch al. 2022), trained on AMPS and ARQMath (see Table for for model checkpoints). We focus on BERT-base' models transformer encoders remain standard and effective choice for information retrieval tasks (Wang et al. 2024 Warner ct al. 2024). We further pretrain these base models on LLMMUT-cnhance datasets using two types of objectives: MLM on MF and MT, and NSP on NMF and MFR. Models pretrained on all four MAMUT- enhanced datasets are written with MAMUT as prefix (e.g., MAMUT-BERT) All models are subsequently fino-tuned on MIR tasks derived from NMF and MFR. These tasks are framed as query-based retrieval: given name or formula as query, the task to retrieve matching formulas from a candidate
(e.g., retrieving the formula. (a + b)2 a² + 2ab 122 from the queries Binomial Formula and e² + +2 c + d2 (c d)²). We sample 250 positive training examples per mathematical identity of NMFT and ten times many negatives, preserving the train-test split from pretraining. We evaluate using both binary classification metries (precision (P), recall (R), F1) and standard TR. ranking metries: Precicion at k (p@kk, Average Precision (AP), and nDCG, that are averaged over all test querics, with highor values indicating better performance (Manning) 2009 Radlinski & Craswell, 2010). To ensure robust results, we further average across five fine-tuning runs. To allow an unbiased comparison where no task or data similar the test data used during pretraining, we additionally pretrain models only the MLM tasks MF and MT, excluding NMF and MFR used for fine-tuning. We denote these models with suffix MLM (e.g,, MAMUT-BERT-MLLM,
7.2 Results
Table 6 summarizes the main results, while implementation details and additional results provided in Appendix C Results of the fully pretrained models, shown in gray for completeness, are included despite their unfair advantage from being trained on more data during protraining. Note that the fine-tuning test data was not seen during pre-tnainn and a new classification head was used during fine-tuning. Notably, the models pretrained only on MF and MT already outperform all baseline models across most metrics. Interestingly, even models that are already mathematically pretrained benefit from additional MAMUT-based training. This suggests that MAMUT introduces complementary patterns not present in prior mathematical corpora used for training. Remarkably, MPBERT mostly outperforms MathBERT, although the latter is often seen the SoTA mathematical BERT-base based model in recent studies (Scarlatos & Lan 2023 Duan et al 2024 Horowitz & Hathaway 2024 Mcadows et al. 2024). Overall, the results support the hypothesis that training on MAMUTT-enhnnces datasels improve mathermatonn capabilities of language models. Our best-performing
models are those based on MPBERT, further pretrained with MAMUT-enhane datasets. We publish the fully pretrained models on Hugging Face, see Table I7
8 Discussion
8.1
Comparison to Other Mathematical Datasets
While existing mathematical datasets such as ARQMath Manssuui et al. 2022a AMPS (Hendrycks al.,
2021), MATH (Hendrycks et al. 2021), and GSM8K (Cobbe et al. 2021 focus primarily on problem diver-
13

---

Published in Transactions on Machine Learning Research (06/2025)
NMF-FT
MFR-FT
Model
P
R
F1
p01
p&10
AP nDCG
P
R
F1 p01 pa10
AP nDCG
BERT
49.2
98.0
65.5
65.6
62.0
69.7
84.3
39.8
94.9
56.1
58.9
58.8
57.5
MathBERT
38.7
99.0
73.7
87.1
73.7
83.8
92.3
43.5
95.8
59.8
78.6
75.1
69.4
89.1
MPBERT
66.8
99.5
79.9
83.7
74.2
84.1
91.9
59.0
99.0
74.0
79.4
77.4
76.4
91.2
MAMUT-BERR--IM
74.4
98.9
85.0
89.7
78.6
89.2
94.4
64.4
99.7
78.2
83.1
82.2
80.5
92.9
MAMUTT-MahBERT -MLM
70.6
99.6
82.7
88.4
77-3
87.8
93.9
58.9
98.9
73.8
85.
85.2
81.6
93.8
MAMUT-MPBERT-ME
78.3 99.4
87.6
87.2
78.1
89.4
94.2
67.7
99.7
80.6
87.3
86.5
84.6
94.8
MAMUT-BERT
97.2 100.0
98.6
96.5
83.3
96.7
97.5
96.9
99.9
98.4
99.7
99.3
99.0
99.7
MAMUTTMMacbERRI
97.5 100.0
98.7
97.0
83.6
97.2
97.8
95.2
99.9
97.5
97.7
98.6
98.5
99.4
MAMUT-MPBERT 98.1 100.0 99.1 97.0
83.7
97.4
97.8
97.9
100.0
98.9
99.7
99.6
90.4
99.8
Table 6: Results for NMF-FT and MFR-FT. All values are reported as percentages. Since the models MAMUT-BERT, MAMUT-MathBERI and MAMUT-MathBERT include the training data of the NMF and MFR tasks in their pretraining, they have strong advantage and are therefore highlighted in gray, However, we note that these models did not. the test data of NMF MFR well, i.e., there is information leak leading to this performance. Still, focus analysis of the comparison of fine-tuned models that only used MLM as pretraining task to separale the impact of the data from that of the pretraining tasks. Bold highlights the best result per metric excluding the fully pretrained models.
sity, ranging across topics, complexity levels, and reasoning steps, MAMUT complements these datasets by targeting a different aspect of variation: diversity in notation and symbols. This focus on symbolic diversity addresses a critical yet underexplored dimension in math-oriented NLP tasks. By systematically modifying the notation and symbols of formulas, MAMUT enforces model's ability to gemeralize beyond simple pat- terns (e.g., the binomial formula contains a, b, and c), enabling better robustness in downstream tasks, shown by experiments. Moreover, due to the rule-based design, MAMUT tracks the transformations it applies, allowing for detailed, fine-grained evaluations (see Appendix C.4.1.
8.2 Data Quality and Reliability
Several measures are taken to ensure high data quality and reliability for all four generated datasets. For MF, we use only formulas judged generally valid by SymPy. MT generated from high-quality textual inputs: the AMPS problems from Khan Academy, reputable source of educational material, and the ARQMath dataset, which is based ensure quality, we include only ARQMath questions with at least five upvotes. The remaining datasets, NMF and MFR, are derived from the manually created NMFT of high-quality formulas. To avoid generating incorrect falsifications, care is laken to exclude transformations that preserve equivalence trivially (e.g.. adding zero or multiplying by one). The symbolic transformations performed by EquVG, including parsing, symbol substitution, and randomized printing, are based on thcorctically cquivalence- presoovvinr operations. Hence, the overall reliability of MAMUT MAMUT-enhanced datasets is closely tied to the correctness of input formulas and the accuracy of SymPy's parsing, modification, and printing routines. Among the transformation steps, parsing seems to be the most error-prone due to the inherent complexity and ambiguity of mathematical notation. This issue is further discussed in the following section.
commumity-moderattee mathematical StackExchange discussions. To
9 Limitations
While the experiments conducted in Section 7 demonstrate strong performance of MAMUT -egeurrteeee datasets, several limitations remain that could be addressed in future work.
First, although the current implementation of MAMUT already supports a wide range of randomized nota- tional variants, more equivalent transformations could be considered (e.g., 1 2 +... Em i). As
14

---

Published in Transactions on Machine Learning Research (06/2025)
result, models trained on MAMUT-enhanced datasets may generalize well to covered notations but struggle with those not supported yet.
Second, the current version of MAMUT focuses exclusively on IaTpX-based input and output. While E is the most prevalent format for mathematical notation in science, the underlying SymPy framework supports additional formats for parsing (e.g,, Mathematica and Maxima) and printing (e.g., MathMI. and various programming languages). This makes MAMUT easily extensible to other formats. Notably, the SymPy parsers require changes for using them in MAMUT. Extending support to these formats would mainly require adapting the SymPy printers to handle randomized output decisions, while FalseVG and the symbol substitution logic in EquVG remains unchanged. Notably, this setup naturally enables translating formulas from one format into another while generating equivalent and falsified versions.
Third, while our generation process entirely rule based and includes safeguards to avoid incorrect falsifi- cations, some edge cases may still result undesired behavior. Morcover, undotocted parsing errors (e.g.. incompletely parsed formula) may be another problem. A comprehensive error analysis of MAMUT outputs, especially of FalseVG, would help to quantify the reliability of the generated datasets.
10 Conclusion
Mathematical formulas are essential to communicate complex and abstract concepts in various scientific fields. To effectively encode the unique structure of mathematical language, specialized mathematical language models are required. We developed MAMUT, framework based SymPy (Meurer et al. 2017 that generates equivalent and falsified versions of LATEX. formulas through parsing, substituting, possibly falsifying, and printing again into 1ТFX format. MAMUT diversifies and expands datasets, as demonstrated by four generated large, high-quality datasets: MF, MT, NMF and MFR, all publicly available on Hugging Face (see Table 5). These datasets can be leveraged for further mathematical pretraining of language models utilizing tasks such as Masked Language Modeling (MLM) and Causal Language Modeling (CLM) to predict equation parts, a Next Sentence Prediction (NSP) variant that predicts if equations are equivalent, or contrastive learning between positive and negative samples learn equation embeddings. Our experiments show that pretraining with these datasets consistently improves mathematical performance for BERT models, even for those with prior matheratical training.
Acknowledgments
The authors gratefully acknowledge the computing time made available to them on the high-performance compuler at the NHR Center of TU Dresden. This center is jointly supported by the Federal Ministry of Education and Research and the state governments participating in the NHR This paper is based on work conducted during J.D.'s Master's thesis at Dresden University of Technology. A.R.
doctoral researcher at Dresden University of Technology during this time. A.R. was funded through the Azrieli international postdoctoral fellowship and the Ali Kaufman postdoctoral fellowship. We also thank Katja Noack for providing an initial version of the transformer pretraining code.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. anXiv preprint arXim:2303.0877 087744 2023.
Akiko Aizawa and Michael Kohlhase. Mathematical Information Retrieval, pp. 169-185. Springer Singapore, Singapore, 2021. ISBN 978-981-15-5554-1. doi: 10. 11111979777--11--555555555
12. URL https://doi.org
Ix Beltagy, Kyle To, and Arman Cohan. SciBERT: A pretrained language model for scientific text. Tn Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference
https:// www.:nnnr-rere.n.nmmmm wwwpe//eeeeeeeeeeeeeeeeeeeee partrers
15

---

Published in Transactions on Machine Learning Research (06/2025)
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNNLP) pp. 3615-3620, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.
1371. URL httpss//aaclaattoooooooo////////
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Danicl M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Newral Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829527
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 89 96, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10.1145/110233 1102663. URI. hnttps://doi.orgg 1145/1102311 T101:22251.11023555 1102363
Wenjie Cai and Qiong Liu. Image captioning with semantic-enhanced features and extremely hard negative examples. Neurocomputing, 413:31-40, 2020. TSSN 0925-2312. doi: http:://dboi.ogg//00000///eeee 2020.06.112. URL http:://wwwwe/oooooooo////////ta
Karl Cobbe, Vineet, Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Malthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint Xrxiv:2110 10. 14 168, 2021. doi: 10.48550/ar 211.111......68.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word masking for chinese BERT. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 0:3504-3514. 2021. doi: 10.1109 /ta.al.202..31233666 URL https:://boi.org/// 1109x/2aa0111 ptt.taalllpp..........m 31124365
Pankaj Dadure, Partha Pakray, and Sivaji Bandyopadhyay. Mathematical information retrieval: A review.
ACM Comput. Sure., 57(3), November 2024. ISSN 0360-0300. 1145/3699953)
doi: 10.1145/36699533. URL https:
Quy Dao and Ngoc-Bich Le. Investigating the effectiveness of ChatGPT in mathematical reasoning and problem solving: Evidence from the vietnamese national high school graduation examination. arXiv preprint 2iv: 06331, abs/23006 0630.333,, 2023. URL htttp:://arxiv. httpss///arxi..o/aa//////// wwwp/br//g/......///////// 06331
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL
Zhiyi Duan, Hengnian Gu, Yuan Kc, and Dongdai Zhou. Ebert: A lightweight expressionn-enhance large- scale pre-trained language model for mathematics education. Knowledge-Brac Based Systems, 300: 2024. ISSN 0950-7051. doi: https:/ http://ooooooo/o..1111110/ ccw.s/siieeee/rrrrcee/////////11 http://000100011100/222222..4..0 kow..ss..22...111...... URL
Jingxuan Fan, Sarah Martinson, Erik Y Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael Brenner. HARDMath: A benchmark dataset for challenging problems in applied mathematies. arXiv preprint arXiv:2410. 0.9909.,99..... 2024.
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. arXie preprint arXiv:2406. 18321, 2024.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pp. 1536-1547, Online, November 2020. Association for Computa- tional Linguistics. doi: 1.118633/11/202.fnddmm mondings-emnlp.112 URL http:://adlanthoooyyyor///222 Findings-emnlp.11
16

---

Published in Transactions on Machine Learning Research (06/2025)
Zheng Gong, Kun Zhou, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Continual pre-training of language models for math problem understanding with syntax-aware memory network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5923-5928 2022.
André Greiner-Petter Abdou Youssef, Terry Ruas, Bruce R Miller, Moritz Schubotz, Akiko Aizawa, and Bela Gipp. Math-word embedding in math search and semantic extraction. Scientometrics, 125:3017-3046,
2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NewrIPS, 2021.
Lucy Horowitz and Ryan Hathaway. Fine-tuning berts for definition extraction from mathematical text. arXiv preprint Xrx::22006 13827, 2024.
Bat-Sheva Ilany, Bruria Margolin, et al. Language and mathematics: Bridging between natural language and mathematical language in solving problems in mathematics. Creative Education, 1(03):138, 2010.
Kalervo Järvelin and Jaana Kekäläänen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422 446, 2002.
Shinil Kim, Seon Yang, and Youngjonng Ko. Mathematical equation retrieval using plain words as query. In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 2407 2410, 2012.
Giovanni Yoko Kristianto, Goran Topic, and Akiko Aizawa. MCAT math retrieval system for NTCIR-12 MathIR Task. In NTCIR, 2016.
Zenan Li, Zhi Zhou, Yuan Yao, Yu-Feng Li, Chun Cao, Fan Yang. Xian Zhang, and Xiaoxing Ma. Neuro- symbolic data generation for math reasoning, 2024. URL
Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, et al. Mathematical language models: A survey. arXiv preprint Xr::22311 07622, 2023.
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question lack-translation for enhancing mathematical reasoning of Ilms. arXiv preprint arXiv:2402. 16352, 2024.
Christopher D Manning. An introduction to information retrieval. Cambridge university press, 2009. URL https::/dss amu http://...auu.e..ee//eeee http://wwwwuuuttaaaaaaa///////// pdf 1eeesseceeee1ish.e lowsel
Behrooz Mansouri, Shaurya Rohatgi, Douglas W Oard, Jian Wu, Lee Giles, and Richard Zanibbi. Tangent- CFT: An embedding model for mathematical formulas. In Proceedings of the 2019 ACM SIGIR interna- lional conference on theory of information retrieval, pp. 11 18, 2019.
Behrooz Mansouri, Vit Novotný, Anurag Agarwal, Douglas W Oard, and Richard Zanibbi. Overview of arqmath-3 (2022): Third clef lab on answer retrieval for questions on math. In International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 286-310. Springer, 2022a.
Behrooz Mansouri, Douglas W. Oard, and Richard Zanibbi. Contextualized formula search using math abstract meaning representation. In Proceedings of the 31st ACM International Conference on In- formalion & Knowledge Management, CIKM '22, pp. 4329 4333, New York, NY, USA, 2022b. As- sociation for Computing Machinery. ISBN 9781450392365. doi: 10.115575511000.333777... URL https://doi.org: 1145/3511808 155553551888835555677
Jordan Mcadows, Marco Valentino, Damien Tency, and Andre Freitas. A symbolic framework for evaluating mathematical reasoning and generalisation with transformers. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association
17

---

Published in Transactions on Machine Learning Research (06/2025)
for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1505-1523, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ /v /2024.
naacl-long.84. URL https::/acllanthologg. wtwp://///222..11111oooooo 1ong.-11
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian F. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert. Cimrman, and Anthony Scopatz. SymPy: symbolie computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peens cs.103. URL https:
//doi.org/10, 7717/p 103
Rob Nederpelt and Herman Geuvers. Type theory and formal proof: an introduction. Cambridge University Press, 2014.
Vit. Nóvot and Michal Štefánik. Combining sparse and dense information retrieval. In Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (eds.), Proceedings of the Working Notes of CLEF 2022, pp. 104-118. CEUR-WS, 2022. URL
Shuai Peng, Ke Yuan, Liangeai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. ArXiu, abs/22105000377, 2021. URT. httpp://aarii..rr////////.....
Felix Petersen, Moritz Schubotz, Andre Greiner-Petter, and Bela Gipp. Neural machine translation for mathematical formulae, 2023. URL httos://arxivoorr/as/////3 16433
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song. Zhuoma Gong Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint (xXx:2407.01284, 2024.
Yao Qiu, Jinchao Zhang, Huiying Ren, and Jic Zhou. Challenging instances are worth learning: Generating valuable negative samples for response selection training. arXiv preprinl (Xxi:2119906533 2021. URL
https://arxxv..or//a////10.. 06538
Filip Radlinski and Nick Craswell. Comparing the sensitivity of information retrieval metrics. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pp. 667 674, 2010.
Anja Reusch, Maik Thiele, and Wolfgang Lehner. Transformer-encoderr and decoder models for questions on math. In Conference and Labs of the Evaluation Forum, 2022. URL http:::/ceurruussor//0013333111 paper-07.pdf
Anja Reusch, Julius Gonsior, Clandio Hartmann, and Wolfgang Lehner. Investigating the usage of formulae in mathematiaal answer retrieval. In European Conference on Information Retrieval, pp. 247-261. Springer, 2024.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXie preprint arXiv: 190p.0154.0, 2019.
Alexander Scarlatos and Andrew Lan. Tree- based representation and generation of natural and mathematical language. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3714 3730, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2D22 acl-long.205, URL
.rgg 202...cc111.... 205/
Zhihong Shao, Poiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint (ix::2402.03300 2024.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, Xintao Wu, Ben Graff, and Dongwon Lee. Mathbert: A pre-trained language model for general nlp tasks in mathematics education. arXiv preprint arXie:2106.00 07340, 2021.
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arihhmetic, 2023. URL hatps:://axxi..or//bb//// 14737
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL
http://arxiivoorr///////22 13971
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jiajia Wang, Jimmy Xiangji Huang, Xinhui Tu, Junmei Wang, Angela Jennifer Huang, Md Tahmid Rahman Laskar, and Amnan Bhuiyan. Utilizing bert for information retrieval: Survey, applications, resources, and challenges. ACM Computing Surveys, 56(7):1-33, 2024.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and lacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL
13663
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.ddd acwweb. org argg/anthooogy//ee 2020. emnlp-demos.6 6
Weihao You, Shuo Yin, Xudong Zhao, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath: Multi- perspective data augmentation for mathematical reasoning in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1032-2958, 2024.
Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topic, and Kenny Davila. Ntcir-12 mathir task overview. In NTCIR, 2016.
Richard Zanibbi, Douglas W Oard, Anurag Agarwal, and Behrooz Mansouri. Overview of ARQMath 2020: CLEF lab on answer retrieval for questions on math. In Experimental IR Meets Multilinguality, Multimodal- ity, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, Seplember 22 25, 2020, Proceedings 11, pp. 169 193. Springer, 2020.
Richard Zanibbi, Behrooz Mansouri, Anurag Agarwal, et al. Mathematical information retrieval: Search and question answering. Foundations and Trends@ in Information Retrieval, 19(1-2):1-19) 2025.
Bo- Wen Zhang, Yan Yan, Lin Li, and Guang Liu. Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CTKM '24, pp. 5405 5409. ACM, October 2024. doi: 10.1145/ 3627673.3679122 URL http://dx doi wr..10..//100 0ttpg//...115//327777.3333 3679122
Wei Zhong, Sheng- Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. One blade for one purpose: advancing math information retrieval using hybrid search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information. Retrieval, pp. 141-151, 2023.
19

---

Published in Transactions on Machine Learning Research (06/2025)
A Original Datasets
In Table i
WC present one version of cach mathematical identity of NMFT, while this entire raw dataset
is available on Hugging F as part of the NMF dataset files. Subsequently, Table S provides an example
entry of ARQMath (Mansouri et al. 2022a] from the Mathematical Stack Exchange, while Table shows an example of AMPS (Hendrycks et al. 2021
Name
Formula
Addition Theorem for Cosine Va, B E cos(a + 3) cos(a) cos (os(a) sin(a) sin
Addition Theorem for Sine
Va, B E R sin (a + (] = sin(a) cos + cos(a) sin (an(a))tanll
Addition Theorem for Tangent
Va, B E R tan(a + 3) I aan(a) tan(3)
Alternatigg Harmonic Series
Σ n== (-1)att
1 - 2
1
+ 3 1 + ln(
=
Basel Problem
E n² = +
+ 32
+ 22
Bayes' Theorem
P(A T( ((AA))
P(B)
Bernouilii Inequality
2 -1,Va > 1 + (1+z)* 21
Binomial Coefficient Formula
Vn, E E, 2 k: (7) セリ
m!
Binomial Distribution
P(X k) (')p* (1 (1-p)* p) n
Binomial Series
Vo, E C> 0: |o| < = (1 +1+))==00 = =h (2)
Binomial Theorem
va Ryn E N: (a + 6)th = Ex= am)
Chain Rule
d dr C= va,b,c,deR: c (ac (ac+bd)+ +fbc-add
=
. g(x))
Complex Number Division
atti
Complex Number Inverse
= =- कमक Iو
Complex Number Multiplication
va,b,c,dd E R (a + bi) (c di) (ac bel) + (ad + be)i
Complex Number Sum
va,b, d E R (a + bi) (c + di) = (a + c) + (b + d)i
Cosine Function Definition
Vr E R cos(x) E
(-1)" 2n
(2n)!
Covariance
Cov(X YJ F[(( F(( -F(])
De Morgan Law
vx -x A) - V -y
Derivative of Inverse Function
# (z)] -
Derivative of a Function
f'(x) = lim f(xth)-f(z) h
Determinant of 2x2 Matrix
det (:b) = a.ca b
Abb det def a..an det (1 b. det (af) + det (g
Determinant of 3x3 Matrix
Distributive Law of Sets Euler's Formula
AU AU n 0)= = (AUB)
Ya E C: ei = co)((a) + (a)
Euler's Formula for Polyhedra
V- F+
Eulor's Identity
et + +1=0
Eulor's Number
e e=lim, (1+1/n)"
Table The 71 mathematical identities of the NMFT dataset.
Z
20

---

Published in Transactions on Machine Learning Research (06/2025)
(Continued from previous page)
Name
Formula
Expected Value
E(X) = Ex x;P(X zi)
Exponential Function
Vr E R lim,, (1 + r/n)" E.
Factorial
vn e N. = ........
First Binomial Formula
R:bbR:((4) (a+ = a² + +2b + 6²
Fundamental Theorem of Calculus
[(r) dx = F(b) - F(a)
Gamma Function
V : [m)] = - x= = (n - 1)!
Gaussian Integral
S expl -x²)dz = V
Geometric Series
En r" -
Gregory-Leibniz Series
D En=o( (-1)"
'2n+1 2n+11 =
Harmonic Series
E 1 =0
n
Hölder Inequality
Vp, 1,
1 + = Vor, yy E R
q
E [.ik S (Ei= (mmp)) EE 119199
Integration by Parts
J [(() (1) dr = {(x)g(x) f f(x))((())) dx
Inverse of 2x2 Matrix
(22) ad-bo -c -
1
Law of Cosines
d= 2 + 1/2 - 2ab cos (*)
Law of Large Numbers
1im 1 Σ Jj = (X)
Law of Sines
sin(A)
sin(B) ain(C)
a
h
Law of Total Probability
P(A) = Σi= P(A
Logarithm Power Rule
VO E RR b> 0,b 1, C
0: :: (x))(=)))))))))))))))())))))ll r logg(x))
Logarithm Product Rule
V 0,
= logy(z) + logo
Logarithm Quotient Rule
V6 E R, 0: log, (I/y) = logs (2) - -looo (v)
Minkowski Inequality
Vp >p>1= E | + 31 Σ - | |)
(E 189e P)
Multiplication of 2x2 Matrix
A (2)), B (1%) + B
A
be+by of abtbl ce ef )
Normal Distribution
f(x) = 1 (x=-w) (2=0)
ov2n
Pascal's Rule
Vn, k E N. ((t)) = (1 (H)
Poisson Distribution
P(X = k) = -ッッ
E!
Power Rule
Vn E R, n + 0: ds (a*) nz=
Principle of
|AU ||| + |B - IAn B
Product Rule
A [((()) [[([z)))(()) = v' (z) u(x) u(z)) v2
Pythagorean Identity
Va E R sin²(a) + cas
Pythagorean Theorem
0² =
Quadratic Formula
Va, b. E R, 0 +
F1,2 -b+vl=4ar
Quotient Rule
Vb E R,
0: log, (z/v) log, logs
Riemann Zeta Function
Vz E C, Re( 2) > 1: {(x) Σm n
Table 7: The 71 mathematical identities of the NMFT datasel.
21

---

Published in Transactions on Machine Learning Research (06/2025)
(Continued from previous page)
Name
Formula
Rule de I'Hôpital
f(ar) lima- ((x) lim g'(x)
f((x)
Second Binomial Formula
Va, E R: (a - b)² = =a² -2 b b²
Sine Function Definition
Vo R sin(r) = En-o(-1)"0/0 + 1)1x2
Stirling Approximation
vn E N. n! 2 (7))
f(x) = Σ ( (a) (I a)
Taylor Series
n!
Third Binomial Formula
Va, b E R (a + b) b) a² b2
Variance
Var[X] = E [X }
Wallis Product
I 4n =
4n2
Young Inoquality
vp. > 1, 1/p + 1/9 = 1, b 0= ab <
+ 69
p 4
pq Formula
V/, R :
pz + +
x1,2
12 v
9
Table 7: The 71 mathematical identitics used in NMFT.
B MAMUT
Table 10 shows examples of the strategies for generating falsified formulas of FalseVG. Table 11 reports the used symbol groups for the symbol substitution of EquVG.
B.1 Implementation
As discussed in Section 6. MAMUT relies on the EquVG and FalseVG algorithms, which generate equivalent falsified versions of given formula. We implemented these algorithms using the Python library SymPy 1.12, which is open-source symbolic mathematics library with computer algebra system features (Meurer et al. 2017p. This library includes ЕТΙX parser for converting expressions into an internal SymPy rep- resentation, which can then be printed back into ETEX.. The SymPy formula representation is a symbolic expression, as required for EquVG and FalseVG, and supports the substitution of variables and generic func- tions. However, the built-in SymPy parser had limitations handling various mathematical notations. The parsing capability has been expanded during this work, including the parsing of matrices, sets, derivatives, and various operators (=, U, n, E[X], Var[X],...] Additionally, the SymPy ETTEX parsing was expanded to support a wider range of mathematical expressions through the implementation of adaptive hybrid approach. This approach introduces a SymPy-like expression that enables safe string- based substitutions. As discussed earlier, a naive string replacement is inadequate for mathematical symbol substitution. For instance, if we replace x in Nexp(x), it would also unintentionally replace the occurrence of x within \exp. To address this issue, our implementation of the safe string-based substitution detects such situations resulting in failure to avoid invalid expressions during the generation of versions ensuring high data quality. This SymPy-like expression also utilizes a predefined list of known symbols and LTTX commands that, when present in the input, are excluded from substituting. For example, the \exp command is included in this list, allowing \exp[x) to be substituted using our safe string-based based approach. This method, while being less powerful than the classical SymPy expressions, exlends substitution support to a wide range of math- ematical notations that can not be parsed the classical parser. Hence, the hybrid combination of the classical SymPy expression with randomized printing and the string-basod substitution, supporting wider range of operators, aligns perfectly with our need create diverse, high-quality mathematical dataset with substituted symbols.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Title
Derivative sigmoid function a(x) = Ite
Question
In my AT textbook there is this paragraph, without any explanation. The sigmoid function is defined as follows: "o(x) 14e-s- This function is casy to differentiate because da(x) o(z)- (1 az)).. It has been a long time since I've taken differential d(z)
equations, could anyone tell me how they got from the first equalion to the second?
Answer 1
Consider f(x)
1t Then, on the one hand, the chain rule gives f'(x)
0 (x)
a (a) xz) and on the other hand, f()) - f(z)= =
(xx)
-(x)-1 Equate the two expressions, and voilà!
(w) (w)
Answer 2
Let's denote the sigmoid function as o(x) e
The derivative of the sigmoid is
d
(() (1x -x))) Here's a detailed dcrivation:
dr
d (() dx -
d
dr
d
(1+e
dx
-1+ (---)
(1+e-2)²
1
1
1
(1+
1
1
1te
1t
1
i+e
= =(r) (1 - a())
(1-((z))
Table 8: Example entry of the ARQMath dataset with preserved LTEX formatting (post ID 78575, answer IDs 78578 and 1225116).
112 3p 54
Problem
Simplify the following expression:
p 9
Answer/
First factor the polynomial in the numerator. p² 1p-3p-pp 54 = (p+)). So we can rewrite
Hints
(p- 9)(p + 6)
the expression as: y p
We can divide the numerator and denominator by (p 9) condition that 9. Therefore y = 6:p
Table 9: Example entry of the AMPS dataset (file hmmps/khan/554/16777000 jsoo)..
23

---

Published in Transactions on Machine Learning Research (06/2025)
Original Formula
Falsified Formula
Description
Equality
a² ++ b==²²c a² + = a² c²
Subtracted 1 from right side
Removed b²
a² + 1 = =6?
Inserted + in exponent of b
S Inequality
>
x zS!
Inverted > to S
ab < a² +
ab > 2 + b²
Inverted 3 10 >
2
2
x # 0
x=0
Inverted of to
X Swap
a² 12 c
a² + 2 6²
Swapped and 2 in 6² Swapped order of arguments
F(a)
- F(b)
F(b) F(a)
x
In = In(x) In(y)
ln
= sin In(y)
Replaced In by sin in In(x)
sin(a)
sin(3)
log(a)
(in
Replaced sin by log in sin(a)
b
A Variable
n!! = 1.2 ........ n
k! = 1.2 ......
Replaced n by l in m!
Σ 1
> k2
Replaced by only in 1?
i=1
OO Constant
eLT = =1
3
Replaced by 3
g 1
Replaced by 1
ie
Replaced by
42 -1
Replaced by 42
2
n²
Σ 12 6
Replaced oa by
12 6
i=1
Distribute
sin(x) (2) sin(v)
sin(x + y)
Applied sine additivity
n!!
n!
Applied faculty multiplicity
(n k-)!
(I: (n - k:)
n!
Applied faculty multiplicity
hl (n! kd
Manual
Vn E N n! a² + 1 =
Vn R2 nn! ...
Replaced N by R Similar formula
a² 1/2 x²
2hc (a)
In any right- angled triangle...
In any right-angled square
Replaced "triangle" by "square"
Random
a² 2+ In any right- angled triangle...
sin + cas)(a) =
Random formula
The derivative of a function ʃ ...
Random text
Table 10: Examples of the strategies for generating falsified formulas (FalseVG).
24

---

Published in Transactions on Machine Learning Research (06/2025)
Symbol Groups
Typical Context
Example
a, a,b, e, i, m, p,q,T,8,t U,
Parameters
ar² + bæ + 0
Indices
C₁ = Σn Ain
Counts
(x) = n!
kl(n-k)
Variables
Parameters, Points
x² + pz + =-=00
Vectors
u
3, A, E.
Unknowns
x+ 2y + 32 =4
Matrices, Sets
A AU = (AUB)
R,R,S,T,UR W,W,K 7,8,,,8,0,44 0. T,0,
Random Variables
X=Y-Z
Angles
a 3 - 180
Scalars
A ( + + 1 = 0
Functions
f,g.h,u,v U,G.H.U.V T,
Generic Functions
[un]' ulu uv S" f(x)dx = P
Antiderivatives
Permutations
(
Table 11: Defined symbol groups for the symbol substitution of MAMUT.
In addition, our SymPy parser implementation is adaptive. Even if an input formula can not be parsed classi- cally, the classical parsing still succeeds for parts of it. As a result, formulas are split at delimiter symbols such as : or Rightarrow. Using these extended parsing capabilities, the input \forall x, y: x'cdot y=yAcdot x is parsed into two sub-cxpressions: Aforall x, y, which can not be parsed classically with the used imple- mentation, and x\cdot y=y\cdot x, which is parsed into classical SymPy expression. Both sub-expressions support the substitution of x and y. This results in, for instance, \forall \alpha, kalpha'times a=a\ times \alpha, where randomized printing was incorporated for the right subexpression. Similarly, support for parsing entire texts containing formulas enclosed within dollar symbols, denoting the LTTEX mathematical inline mode, is integrated into the SymPy parser. To create randomized IATEX formulas from the parsed SymPy expressions, the SymPy LIEX printer has been enhanced to support randomized decisions. The printing process is guided by randomized sctting which define all the randomized decisions the printer could make. The modified SymPy code is accessible in forked repository on GitHut providing a simple interface for generating equivalent and falsified versions of formula. Additionally, the generation code for the generated datasets based on AMPS, ARQMath, and NMFT is publicly available including the logic for base formula filtering, extraction, and validation. We filter the input formulas considered for MF us- ing two SymPy methods.
formula is retained either sympy. solve() sympy.solve() finds at least one solution (e.g.. x = 2/2 for x² = 2), which can then be used as an implication (c.g.. x² = + I = -V2 or I = 22), or sympy. simpy.simliff() evaluates as True (e.g., 1 1 2 = 3 and tan(x) cos(z)
sin(ar)
B.2 Generated Datasets
To illustrate the behavior of MAMUT and the data extraction process, we provide artificial examples for each generated dataset based on the previously shown raw data: MF in Table 12 MT Table [3 NMF Table 14 and MFR in Table Please note that not all examples are verified as part of the actual generated datasets, but are selected to illustrate the diversity of MAMUT.
https://github... cow/aieng hawpmngyyya/yyyayaaaaaaaaa
https://githuu con/aieng-.aeeeeooooooeo.coeii.a Lab/math mutator
25

---

Published in Transactions on Machine Learning Research (06/2025)
Formula
a(x)-1
[()
[(m)
T(V) +1 -() (-1++(u))
[(j * ( [ = 1/v
(x)++(-0)) (-1))AAA))
1/1(z)
(-1/v
-1/m(0)+ (a()) +-1))
(m(x)
((x) (-11)/A())=111 X(x
p² p- 11-3p-54- 54 (p-
(0+9-(-1)) (6))))))))))' -
------+---((1) (-1)=(9-(-1)) + (-1) +))
54- p² p
p* - p*3 54(-1)
Table 12: Example entries for MF (based on Table and Table 9
Text
In my AI textbook there is this paragraph, without any explanation.
The sigmoid function is defined as do (a) (x) (1 - (x)).. 8 It has d(n) been a long time since I've taken differential equations, so could anyone tell me how they got from the first equation to the second?
follows: "a(x) 14e-x- This function is casy to differentiate because
In my AI textbook there is this paragraph, without any explanation. The sigmoid function is defined as follows: "r(y) = 1/(c + 1). This function is casy to difforentiate because r(y)(=T(g) + 1) = r r'y). It has been a long time since I've laken differential equations, so could anyone tell me how they got from the first equation the second?
Consider f(x) 1 1 = Then, the one hand, the chain rule gives f'
d
d'()) (xx)
(xx)
de
oxx)
and on the other hand, ('(x) a (1+e = - = 1 f(x) = 1 o(z) expressions, and voilà!
s(al-1 a(x)
Equate the two
Consider u(y) = 1/o(y) 1 v Then, on the one hand, the chain rule gives
d ay
(y) dy aly)
d [ dy (y), and on the other hand, n' u'(y)= d
- = 1 u(y) = 1 1 a(y)
Equate the two expressions, and voilà!
22 3p 54
Simplify the following expression:
First factor the polynomial in the numerator. p²
3p - 54 = (p - (p + 6). So we can rewrite the expression as: (p 9) + 6) We can divide the numerator and denominator by (p 9) condition that p # 9. Therefore - 9
6.
Simplify the following expression: - First factor the polynomial in the numerator. 8 ø-9
8 54 (s 9) (6 s). So we rewrite the expression as: و (s 9) (6 s). We divide the numerator and denominator by -9 + condition that s o 9. Therefore + 6; s o
Table 13: Example entries of MT (based on Table and Table 9).
26

---

Published in Transactions on Machine Learning Research (06/2025)
Name
Formula
Label
Factorial
d! = 1.11-...441 .
Definition of a factorial
Vn E : -11
Definition of a factorial
Vn :: +):= (n (n) n! A =
Definition of a factorial Factorial
For any natural number & we have k! is defined as k k= IT For any natural number n, n! can be obtained by multiplying all nat- ural numbers from 1 Y logether.
Definition of factorial
Vn,j E N, 2j: ()) jI-(n-J} ml 1. .....=
Factorial
Factorial
Vm 2 1: ml = m (m (-1))1,0!=
Definition of a factorial
1 + 4...s=== ===
Definition of a factorial
= 3) 18.4 5/-
X
Factorial
n! Σm_
Factorial
The sum of two complex numbers 91 +i- z and g2 + defined 91 + 92 + i (h + u 2
is
Definition of factorial
0! 1 ........
Table 14: Example entries of NMF (based on Table 4
Formula 1
Formula
Label
The value of (1+1/T)² approaches the constant
As approaches infinity, the expression (1 +
as 7 tends to infinity.
1 1/ converges to the value of e x 2.718.
By utilizing
the infinite series we for all real numbers 2.
For all real mumbers x the expression sin(2) defined as the infinite sum E 2n+1.
Do 1+22 (-1)
can define sin(z)
[1+2n)]
(2 1)! (-1)-
The limit as I approaches infinity of the expres-
Vo E C. = -
29/21 x
sion (1+ y) converges to the exponential function ed for any real number y-
I x² /31 +...
For all real positive g with g 1 and for all real positive s, y, have log, (sy) log, (s) + (z).
For all real bases such that 0 b and b o and for all real positive 2, y, the equality lzgı(2/v) = log, (2) log. holds.
The derivative of a composite function f (9(2))
The derivative of composite function x f((())) with respect to is given by
with respect to = is given by क ([g (z))
g=).
do(u) ((g(u))((2,g(()))
Vm 2 1: ml = m (m + (-1))
Va E :: (a (1! (a+1) a!, 0r =1 (a b) -12 a
Let and b be real numbers. In this case, (c+ () c) is equal to o² 2-28 62
Table 15: Example entries of MFR.
27

---

Published in Transactions on Machine Learning Research (06/2025)
Model
Hugging Face Identifier
Reference
BERT
bert-base-cased
Devlin et al.
2019
MathBERT
tbs17/MathBERT
Shen et al. (0021)
MPBERT
AnRee//nath.pretrainee__ee
Reusch et al.
(002
Table 16: Baseline models used for comparison in experiments.
B.3
Analysis of NMF
For better understanding of the version generation algorithms, EquVG and FalseVG, we delve into
detailed analysis of the generated NMF dataset, visualized in Figure 5
Figure ba shows the distribution of strategies over the mathematical identities of the NMF dataset. The figure illustrates the proportions how many falsified versions of a mathematical identity utilized a particular strategy. Since multiple strategies might be applied to generate a single falsified version, the proportions do not sum up 100% per identity. Approximately half of the time, only a single stralegy applied. In general, the different strategies obviously have different proportions across the mathematical identities. The most common strategics Variable and Swap because variables and swappable expressions (o.g.: sin(x + y) > sin(x) + sin(y)) occur in almost all formulas, often even multiple times. About 20% of the strategies are intentionally completely random to avoid introducing bias towards challenging negative examples. In real-world MIR applications, random pairs are more common than challenging ones. Some strategics can not be applied to certain identities, particularly the strategy inequality, which not applicable to most identities. The reason why some identities containing no inequalities still have nonzero proportion for the strategy Inequality is that this strategy can be applied even after random manual formula (containing incquality) is chosen
falsified version.
Another analysis can be deducted from Figure 5b which shows the distribution of whether variable, function or any of them has been replaced in a generated version of a mathematical identity in the NMF dataset. Since only ten identities contain generic function symbols, substitutions of functions can only be applied to those identities regularly. Again, we recognize proportions slightly above zero for many identities not containing generic functions due to function substitutions after applying the strategics Random or Manual. The overall proportion of substituted formulas is 52.3%, but when considering only equivalent versions, this proportion rises to 81.3%. For falsified versions, the substitution proportion is about 45.1%.
C
Experiments
This section provides implementation details and further results for the experiments conducted in Section 7 The model checkpoints used for comparison and starting point for further mathematical pretraining are reported in Table [6] Note that we also tried the MathBERT variant with custom mathematical vocabulary (tbs17/MathBERT-cusst but found that it performs weaker than MathBERT, and therefore excluded it from this study.
C.1 Pretraining
This section provides details regarding the mathematical pretraining tasks and the implementation. We publish our code for mathematical pretraining, and our fully trained mathematical models (sce Table 17
C.1.1 Tasks
In this section, we introduce two domain-adapted MTM and two domain-adapted NSP tasks used for the training of mathematical BERT models based on MAMUT-enhanced data. MLM and NSP have proven
httpps://gi thub. con/aieng-
28

---

Published in Transactions on Machine Learning Research (06/2025)
Equality
Inequality
Swap
Variable
Random
Constant
Distribute
Manual
0.7
0.6
0.4
0.3
0.1
0.0
Addition Thearem fur Cosine
Addition Teooemm for Sime Addition Thegemm for langent
Alleraaingg Harmwyye tie mrbslem Binomia
Basel Problem
Bemoull ei Ingquallty
Coefficiet Formela
Binumal Distr brtthtn
B Serees
Binomigl The en
Chain Kule
Complex Comp plex Mallig Inverse
wisten
Complex N Number Multpliiuti Num Complex Number Cosine Function De linition
ellcttionn
Covariance
De Morgan Law Derivative of Inverse ffunctio Dervativ of a aeiemmnant
Functien
3x3 Matrii
Distributig Law of Sels Eulers Formuls for leers lyyeyrrr
Eulers Fo Formull
Expecte led Value
Function
[actorinl
Formila
us
Gamma Caussan Functton
Sagant Integral
Series
Grogor Farmmen eniiiic Series
H L---em Seriis
Holder Inequal aequallly
Parts
Inverse of 252 Matrix
Law of Cos cosines
Law Large Numbers
faw Lmmeers
Law of Total Tobalo Sines Cogarthmm Power Sule
Loganthn Paoduct RU Logarittun
Quehent Ka lc
13 requality
Normal Distri burrontt
Pawcal 's Rale Poisson
Dostribuu bastrbution yton
Fower Kule
Produtt Rale
Pythago gerean dentlit
Fy (ithrggsree Quadaati Jesorest
Formul
Riemann Quotinett Rule
Mule de Hoptal
Second Rinomtal Forrhpla
Sine Function tio Definttinn Stirling Approyinn siindiio dien
Third Dinomial grr Series
Varlance
Wallss Procuge
Young Insquality pa
Boom
Number
Exponential
First Dinonial
heggem of
Integration
of Inclus Inclusion--
Multipliaal Multppication
Fundamental
Mathematical Identity
(a) Proportions of strategies used for generating falsified versions NMF dataset per mathematical identity.
Variable Substitution
Function Substitution
Symbol Substitution
0.6
Proportion
0.4
Addition Theoeen Coane
Additiin Teeeem for langent Altemating cormont te Sepes
Basel Problen-
Baves Tharrem
Binomial Bemounin qualityy
Eanomial Iistiibttionn
Rinomia Semss
Binomial Chai Rule
Theorem
Complex Vlabber Muitip liation
Comples complex Number Sum
Cosine Funclion Delinition
Covarian Morgan Law
Derivative of Inverse Deriveave aevveeeee of a Function Desernina mant of 202 Matris
Detenninan nant of 333 Matrix Distributive Law of Set
Ealers Formula Enkers fermmi
sulers ldentity
Etalems Numbeer ee
Ex Expey ed Value
sunction
facteral
Calculus
Camma Fanction
JosssinnIngge Injeraal
menteetn Serres
Barmee Serres
Hölder squulity
Integration by Parts
Inverse of 2x2 Matrix
Law of Cosines
Law of Large Number beee
Law of Total rebabillty
Logar gor fower Rulo
Lngartilon Product Rulo
Rule
Inquuat
Normal Distriuttonn
Pescal's s Rule
Distribution
EeEsch Rule
Principle Inclusinn Exclusint
Product Pythagorean ldentlly
Quadratic Ihsore
Fermule
Qhoteett Rule
Rule de senctionn
Second Bunomin) Forthulé Siste unition Definition Stirling Apprnimmatin Tavior Series
Third Binomial Furmula
Wallis Product
Young auguality pg Formuls
Quolient
Gregory Leibmi
2x2
Exponential
First Binomia
Pundamental Theremm
Gaussan
Multip Minkoway
Polsson
Ily "ythagurean
Comples
Mathematical Identity
(b) Proportion of generated (equivalent and falsified) versions with least one variable, function, or symbol substi- tution (variable or function).
Figure 5: Analysis of NMF. The mean across all identities is printed as a solid line.
29

---

Published in Transactions on Machine Learning Research (06/2025)
Model
Hugging Face Identifier
Source Model
MAMUT-BERT
bert-base-cased
MAMUT-MachBERT ddrg/MathBERTT-maaul
tbs17/MathBERT
MAMUT-MPBERT ddrg//mathuppeetrained..ec hetprraiiindebbe___maaaaa Ankeu//athhpprettaiiee__cr
Table 17: Hugging Face identifiers of our published mathematical models.
BERT
Adding Mathematical Tokens
MF-PT
n! Acdot [MASK] \cdot \dots \cdot [MASK]
For any [MASK] number $n$, we have that $[MASK]!S [MASK] defined $1 [MASK] Acdot Adots Acdot n$.
MT-PT
NMF-PT
Factorial [SEP] n.! = 1 \cdot 2 \cdot \dots \cdot n
MFR-PT
n! \i=1d_ i [SEP] n! = 1 Acdot 2 \cdot \dots \cdot n
Figure 6: Visualization of the additional mathematical pretraining with all protraining tasks (i.c., the training of MAMUT-BERT). MLM-like pretraining tasks colored in blue, NSP-like in red.
to be effective in literature (Devlin et al. 2019: Feng et al. 2020 Reusch et al. 2022). Each of the four MAMUTT-generated datasets (MF, MT, NMF, and MFR) create its dedicated pretraining task (denoted by <dataset>-PT), see Figure 6 for an overview. the following, we briefly discuss the mathematical peculiaritics.
MF-PT This pretraining task adapts the standard MLM by utilizing mathematical formulas instead of natural language text. It applies the masked language modeling, where 15% of the tokens are masked following the proposed setting of Devlin et al. (2019) for BERT: 80% of selected tokens replaced with the special [MASK] token, 10% are replaced with random token, and 10% remain unchanged. This task intends to train the model basic mathematical knowledge and understanding. Importantly, the formulas in MF have been carefully filtcred such that only generally valid formulas are contained, which ensures that a masked token can be logically inferred by context (see Section G for details).
MT-PT While MF-PT is designed Iearn intra-formula dependencics, MT-PT focuses on understanding relationships between formulas and natural language. That is why this pretraining task uses mathematical texts input. To enforce masking of the mathematical parts in the text, special masking algorithm is applied, which involves two additional masking probabilities:
haan--formuua--rrbbbbblt This probability determines the proportion of formula tokens being masked (ie., tokens inside dollar tokens indicating the mathematical ыЛЕE inline mode). We set min- formula- probabllprobability to 20%, meaning 20% of all formula tokens are masked.
min-mathh-urrds mon--attatuunn---rbbbibtttttt This probability controls the masking of mathematical words, such sum, minus, equals, less, or function. We collected 219 of such words manually We use mbm- 30%, meaning about 30% of the tokens belonging to mathematical words within the sequence are masked. These words are either masked completely or not at all, effectively applying a whole word masking (Cui al. 2021 to these mathematical words.
www--worr---pbbbbbi
words.py
30

---

Published in Transactions on Machine Learning Research (06/2025)
bin ##om { n
k
}
##racc
n
!
n
!
##do ##t
k
Example for different tokenization with the original BERT base tokenizer and their enriched versions. Red boxes mark split up tokens the original tokenizer, which are tokenized as a single token the enriched mathematical tokenizer.
Figure 7:
If the total number of masked tokens is below 15,, additional tokens selected at random to ensure that 15% of tokens are masked in total.
NMF-PT This NSP-like task associates a name of a mathematical identity with its formula or a describing text. Since for each positive sample four times more negative examples are generated (soe Section 6 an unseen negative example can be used every epoch (as we train less than four epochs). Thus, within each epoch, a balanced number of positive and negative pairs are provided. The positive pairs are the same over the epochs, only the negative pairs change between the epochs.
MFR-PT Similar to the previous task, two separated sequences are used as model input in NSP-like style. However, in this task, two formulas are used instead of a name and formula, similar to the the second task of the ARQMath competition, where a formula is used as a query to find similar formulas (Mansouri ctt al. 2022a). This data is generated from NMF, by replacing the formula name with an equivalent version of the formula (from another positive sample of the mathematical identity within the dataset). This approach preserves the proportion between positive and negative pairs, and the negative samples are split over the epochs for NMF-PT.
C.1.2 Pretraining Setup
We combine multiple of these four mathematical pretraining tasks to further pretrain existing BERT based models. A visualization such a mathematical pretraining with all four tasks is shown in Figure 6 includ- ing illustrative example samples. We apply our pretraining tasks not consecutively (i.c., one task is fully completed before starting the next one), but in mixed manner, where the tasks alternate after each batch, For example, when training four tasks in this style on 8 GPUs, there are two dedicated GPUs for each task. Experiments show that the mixed training style euhances the overall training results, as the model does not forget previously learned information. Each task is trained for 250k optimization steps.
To enhance mathematical encodings, mathematics-ppccific tokens are added to the BERT model, whose randomly initialized embeddings are learned during the mathematical pretraining. This approach similar as seen by other mathematical models (Shen et al. 2021 Reusch al. 2022 Novotný Stefánik 2022). Before further mathematically pretraining, we enriched the vocabularies of BERT with the 300 most common ETEX tokens found in MT that were not originally part of the tokenizer vocabulary. We defined LMTEX tokens as either word starting with a backward slash (e.g., Afrac) a TATEX environment identifier, i.e., text inside \begin(...) } and Nend(...} (e.g., pmatrix). The influence of the enriched BERT tokenizer regarding the tokenization of an example formula is shown in Figure 7 The word embeddings for these 300 newly added LTTX tokens were initialized randomly and learned during the mathematical pretraining. We use the same initial weights across all mathematical pretrained BERT models. The token enrichment should help the model to better understand the word embeddings of frequently occurring terms in the context of mathematical ыТІX texts. Note that while this enrichment helps in capturing the correct semantic structure of mathematical content more effectively (Peng et al. 2021b, the tokenization may still not always perfectly capture mathematical semantics. For instance, in both base models, the tokenization of the formula ab, representing multiplication of a and with omitted multiplication symbol, results in a single loken. Notice that these mathematical tokens have been only added with BERT as the starting model for mathematical pretraining. MPBERT already added 501 mathematical tokens as part of its mathematical pretraining RRuushh al. 2022). MathBERT available in two variants, one using the standard BERT vocabulary and another with customized vocabulary (Shen et al. 2021 However, the variant without special tokens
31

---

Published in Transactions on Machine Learning Research (06/2025)
performed better our downstream tasks. Therefore, we used the version with the standard vocabulary, without any vocabulary enrichment.
C.1.3 Implementation Details
For all our experiments, we employed programs written in Python 3.10. and utilize the libraries transformers 4.25.1 (Wolf et. al. 2020) and datasets 2.10.1 from Hugging Face for tokenization and model training. For pretraining, a custom implementation of training loop has been used relying on the PyTorch library (torch 1.13. 1.3.1+cul17) to enable parallelized training multiple GPUs. We initialize a separate new prediction head for each specific objective, following a similar approach to the implementation of the transformers library. All of our models introduced the next section were pretrained on eight A100 GPUs. For all pretrained models, we maintained a batch size of 16 per GPU, 200 warm-up steps, a maximum input length of 512 tokens, learning rate of 10-5, and trained for four epochs. Training one model took about 12 hours per applied task.
C.2 Fine- Tuning
We reuse NMF-PT and MFR-PT with a revised interpretation. During pretraining, the model's objective is to determine whether a given input, a name or formula, can be associated with formula, essentially creating binary classification task. In the downstream task, we adapt this objective use a name formula query retrieve relevant formulas as ranked MIR task. Despite this reinterpretation, the training process remains the same, although only a reduced version of the dataset is used compared to pretraining. We denote these tasks as NMF-FT and MFR-FT, respectively.
To apply these downstream tasks, we created reduced versions of the datasets NMF and MFR that were used for pretraining. For each mathematical identity, we keep 250 positive examples and ten times more negative examples. Importantly, we maintain the train-test split of the pretraining, ensuring that samples in the downstream task's test set have never been seen during training. Importantly, the pretraining test data has not influenced any training decisions, such early stopping, model selection, any other procodure that could introduce information leakage. Specifically, the pretraining lest set was further divided into two splits: one for validation and one for testing. All splits are sampled in a stratified manner to preserve having ten times more negative examples than positive ones.
For each fine-tuning task, we train all tested models for ten epochs, with negative entries changing every epoch during the training in the standard setting (see Section C.4.2 for an exception), as done for the pretraining tasks on this data. After the ten epochs, the best model selected based on the validation Fi-score, which is computed after every epoch. To enhance the robustness and reliability of our results, we train the models five times with different random seeds and average the values to obtain more stable results. All random decisions between runs are made deterministic, e.g,, the same partition of the identities is applied within every training run. Similarly to the pretraining, we create a new IR-head, identical to the classical NSP-head for all fine-ture fine-tnnig tasks (i.e., we do not re-use the NSP/NME //FRR heads learned during pretraining). We publish our evaluation codo
C.3 Metrics
We report several standard classification and ranking metrics (Manning 2009
Radlinski & Craswell 2010
Accuracy: Proportion of correctly retrieved formulas among all retrievals.
Precision: Proportion of retrieved formulas that are relevant.
Recall: Proportion of relevant formulas that are retrieved.
Fi-Score: Harmonic mean of precision and recall: 2 precision recall precision + rocall
hhttps://gi thub. httpa//nnn-a///raafff--aaateeeee
32

---

Published in Transactions on Machine Learning Research (06/2025)
Precicion at lo (p@k): Precision among the top & retrieved documents
Average Precision (AP): Average of påk values for all retricved relevant documents, i.c.
AP Ek=r p@kk relevant(k))
Ek=1 relevant(k)
Here, denotes the number of retrieved documents and relevant(k) is 1 if the k-th ranked document is relevant, otherwise O. The denominator is equal to the total number of relevant documents.
Normalized Discounted Cumulative Gain (nDCG): A score comparing the ranked list to ideal ranking (Järvelin & Kekäläinen 2002). It builds on Discounted Cumulative Gain (DCG), which measures how well relevant documents are ranked by giving higher scores to relevant documents that appear earlier in the list. The formula for DCG (Burges et al. 2005
relevance loga 1)
DCG =
where relovance(k) the graded relevance of the document at position k. For this work, we use a binary relevance score. nDCG is the normalized DCG value computed by dividing throngh the maximum possible DCG (i.e., the ideal ranking).
For NMF, we use all unique names as queries, and for MFR, we randomly pick one true formula per mathematical identity to serve as the query for all paired formulas of that identity in the test set. In both cases, we compute the ranking metrics over all formulas associated with the same name identity (including the challenging false examples). The final score is obtained by averaging the metrics across these evaluations. For binary metrics (accuracy, precision, recall, F1), we consider any predicted probability above 50% to indicate positive classification.
C.4 NMF Retrieval Task
We present additional results based on the NMF fine-tuning task discussed in Section 7 The ability to conduct such fine-grained analysis highlights the strength of MAMUT, which tracks all applied transforma- tions, allowing meta data to be used for analyzing their impact. We release the NMF-FT dataset with this additional meta data
C.4.1 GrainGr NMF Evaluation
We perform a fine- fineggrineed performance analysis of NMF-FT based on three criteria: the number of falsifying strategies applied (Table 18 whether symbol substitutions were applied (Table 19h. and whether the input was given as a mathematical formula or as a textual description (Table 20
Table 18 reveals how model accuracy varies with the number of falsifying strategies used for the test sam- ples. Strategies with zero strategies correspond to the positive labeled entries. As expected, applying more strategies generally makes falsified samples easier to detect, resulting in higher accuracy. This suggests that future applications of MAMUT could focus more applying fewer falsifying strategies to increase the task difficulty.
Table breaks down accuracy by whether symbol substitutions were applied, divided into variables, func- tions, or both. As expected, applying substitutions slightly reduces accuracy, suggesting that these changes introduce a moderate challenge. Function substitutions appear to be slightly more challenging than vari- able substitutions. However, the smaller number of training samples with function substitution could also contribute to this discrepancy.
Table 20 highlights the performance gap between models when processing mathematical formulas versus their textual descriptions. In genera, models perform better when provided with textual descriptions. Importantly,
formulas_ft
33

---

Published in Transactions on Machine Learning Research (06/2025)
Number of Strategies
Accuracy
Number of Test Samples
0
99.39
2,626
90.01
2,626
2
97.44
8,352
99.23
7,075
99.79
3,176
99.98
785
Table 18: Accuracy of NMF-FT across varying number of applied falsifying strategies. Results are averaged across all models (Table 0 Accuracy scores are reported as percentages. Note that 0 applied strategies corresponds to positive examples.
Variant
Accuracy
Number of Test Samples
X
X
Variable Substitution
95.61
97.16
10,280
18,616
Function Substitution
94.43
96.71
1,279
27,617
Symbol Substitution
95.60
97.22
10,802
18,094
Table 19: Impact of symbol substitutions on NMF-FT accuracy. Results are averaged across all models (Table 6 Substitution types are variable, function, and both (symbol). Accuracy is reported seprarately for cases with (V) and without (x) substitution. Accuracies are reported as percentages.
Model
Accuracy
Difference
Text
Formula
BERT
91.23
89.41
1.81
MathBERT
93.98
89.41
1.15
MPBERT
95.79
94.81
0.98
MAMUT-BERT-MIM
97.11
96.57
0.54
MAMUT-MachBERT-MNI
96.32
95.98
0.34
MAMUT-MPBERT-MLL
97.62
97.13
0.49
MAMUT-BERT
99.76
99.68
0.08
MAMUT-MathBERT
99.78
99.73
0.05
MAMUT-MPBERT
99.84
99.80
0.04
Table 20: Accuracy of NMF-FT models when evaluated on textual (N 18, 764) versus formula (N
10, 132) inputs. Results averaged across all models (Table 6 percentages.
All accuracy scores are reported
34

---

Published in Transactions on Machine Learning Research (06/2025)
Model
Precision
Recall
p@1
p@10
AP
nDCG
MPBERT
66.8
99.5
79.9
83.7
74.2
84.1
91.9
MPPBERRTraandon-falles
14.0
99.7
24.6
17.1
16.3
19.0
53.7
MPPEERT-cnstant-flles
59.5
99.2
74.4
76.2
70.3
78.9
89.3
Table 21: Results for the NMF finc-tuning task for different fine-tuning settings for MPBERT. All scores are reported percentages.
our models pretrained on MAMUT-enhanced data exhibit a smaller gap between text and formula inputs, reflecting their pretraining on both, textual (MT) and formula (MF) MLM tasks.
We also considered analyzing accuracy by applied falsifying strategy types. However, observed that strategies with more training samples lend to show higher accuracies, making it difficult to identify which strategies are more challenging.
C.4.2 Ablation Study on Challenging False Examples
To assess the specific contributions of MAMUT in falsifying formulas, we also evaluate two modified fine- tuning versions of MPBERT. The standard setup involves challenging false examples generated by MAMUT, dynamically changing with each epoch. To assess the impact of MAMUT's falsification, we evaluate two controlled variants of MPBERT: one using randomly sampled (i.e., non-challenging) false examples from other identities that change each epoch false examples across all epochs PPPPEERT-constatt--ee unchanged. Results are reported in Table 21 alongside the baseline MPBERT. The table confirms our intuition: both the difficulty of false examples and the dynamic sampling strategy contribute positively to model performance. While PEPBERRT-anddo--fff nearly all documents, it performs very poorly on all other metrics, emphasizing the importance of challenging negatives during training.
and another using a fixed set of challenging Note that the validation and test sets remain
achieves the highest recall, likely due to retrieving
C.4.3 Mathematical Identity Analysis
Up to this point, we have only considered metrics across all mathematical identities in NMF. Now, we will discuss how the models perform on formulas basod on specific mathematical identity in NMF. Figure g shows the Fi-score of NMF-FT for various models dependent on the original mathematical identity. In this figure, the identities are sorted by the average F1-score calculated from all shown models. Consequentlyy, the most challenging identities are located on the left-hand side, while the easier ones are on the right. Notably, formulas containing many substitutable symbols are among the most challenging identitics, such Derivative of a Function (z) = lim;-+0 [(a+h)-((()) Complex Number Sum ((a bi) + (c di) = (a + c) + (b + d)i),
h (ab
Determinant of 3x3 Matrix (det de = a- det (R b. b-del (11) + c- del (g {)),, and Rule de l'Hôpital h
(lima-sa g(z)
[()) lim f(()) (z)) while formulas with few substitutable symbols are among the easiest identities,
such as Alternating Harmonic Series (1 (1-1+1 - In())), Covariance (Cov [X, YJ E((X - EX E[Y])]), De Morgan Law (Vz,y -(x Ay) - V-y), and Eulers Formula (Va E C e cos( a: + +i)) Inequalities are rather challenging, while addition theorems appear to be casier.
When comparing the results of different models, we observe that the best models are able to classily most identities almost perfectly but do struggle for certain identities (e.g., Binomial Formulas), resulting in drops up to several percentage points Fi-score. In general, each model appears to have its own challenging identities, since the Fi-scores of a particular model do form monotonic line across the mathematical identities in Figure
35

---

Published in Transactions on Machine Learning Research (06/2025)
BERT
MachBERT
MPBERT
MAMUT BERT MLM
MAMUT MuhBERT MLL
MAMUT MPBERT MLM
MAMUT-BERR BERT
MAMUT MahBERT
MAMUT MMBERT
1.00
0.95
0.90
Seare
0.80
0.75
0.60
0.55
0.45
Thire Holder Fommlla
Derivative of Incqualyy
Yours po Inequallyy Detemmine of 38.3 Marni Comple Nambee Sum Binoenaa Fonmun
Wallis tule
Quadratic Formula
Rule de HHlptte
Second Binomia Digision
Comp Fomnull
Minkows &i Tnnggu Sertes
Sárling Banomaa Thui.
Anreosimaain mation
Complex Enles Identit
Multpple pplicsttion
Multipicatin grataai by Pant
asstim Fower Rule
Luuiit, Tan, Calculis Binomia) Coeffeiign Fornula
lioesse oi 2:2 Matrix
Fumecion mefiniti
Gregony Leibaz Series
Txnes ressiidd Value
Bisel Prob Problem
Law of Sines
Coussian Integral
Deteminsg Dehaifoon 2x2 Maina
Bayer Theewer
Noraal Distn biuutoo
Law of Cosines
Power Rale
Beroon Incquale
Logariting Qwiteent Rule
Law Total robubillty
Comple lex Sumber laversa-
Expa Expuiential Fume-ton
Hurmont Serti
Factorin
Flalers Fommlaa for polyhelli Distribeet
Law of Se5s
Additioe Tim. for Cuiinn
Gesmentin Ealerg Series
Chain Rale
Logarithm Prodic Rule
Pascal Kulc
Derivative Pytloggonent julentyy
Bioonia fimeiion
Pythagor Theree
Preduc: i Rule
Poisson Disti bullon-
Canina Functio innion
Numnbers
Kiomann Function
Binomia Distri busien
De Morgan Law
Addttion Theirem for Sing
Co verrance luggent
Logarthin 2x2
Entlrss
Altemating darmonic
En
Integrai
Addineo Thm.
Law
Mathematical Idcutity
Challenging
Easy
Figure 8: F1-Score of NMF-FT for various models split up by mathematical identities. The identities are
sorted by the average Fi-score across all modcls. to be updated with final models
Model
Precision Recall
p@1
p@10
AP
nDCG
BERT
7.4
28.1
6.7
4.3
8.2
12.3
45.2
MathBERT
9.4
28.2
12.9
4.6
6.7
11.1
44.0
MPBERT
9.4
31.8
12.6
2.0
7.9
12.0
45.2
MAMUT-BERI-MLM
13.0
45.0
19.8
9.5
14.6
17.7
50.6
MAMMTTMathBERTT MLM
9.1
31.4
10.1
0.6
4.5
10.2
42.5
MAMUTT-MPPERT-MIM
13.2
50.3
20.7
12.9
15.9
20.5
52.6
MAMUT-BERT
33.7
94.5
49.6
74.8
66.7
75.0
88.0
MAMUT-MathBERT
14.6
56.6
23.0
26.5
20.6
23.7
55.4
MAMUT-MPBERT
37.7
99.5
54.6
81.4
71.6
80.1
91.2
Table 22: Results for NMF-Split. All scores reported percentages. Since the models MAMUT-BERT, MAMUT-MathBERT and MAMUT-MathBERT include the training data of the NMF and MFR tasks in their pretraining, they have a strong advantage and are therefore highlighted in gray. However, we note that these models did not see the test data of NMF or MFR as well, i.e., there is no information leak leading to this performance. Still, we focus our analysis the comparison of fine-tuned models that only used MLM as pretraining task to scparate the impact of the data from that of the pretraining tasks. Bold highlights the best result. per metric excluding the fully pretrained models.
36

---

Published in Transactions on Machine Learning Research (06/2025)
C.4.4 NMF-Split Fine- Tuning
We evaluate the models on a modified NMF task, denoted as NMF-Split. In this task, we further reduce the NMF-FT data, such that each mathematical identity appears exactly one of the data splits. For example, all versions of the Pythagorean Theorem are included in the training data but not in its validation and test data. This setup allows to assess the mathematical knowledge of the model. During fine-tuning, the model is only instructed on which lype of objective to learn, and it must combine this learned objective with knowledge gained during pretraining for evaluation. For each of the five runs used for averaging the results, we apply a different partitioning of mathematical identities, resulting in 49 identities remaining in the training data, and eleven identities in both the test and validation set. During fine-tuning, we freeze the model parameters of BERT such that only the classification head is learned and we evaluate on actually learned core model knowledge.
The results shown in Table This task is clearly much more challenging than NMF-FT. Since the test set contains ten times more negative positive examples, the BERT model even performs worse than naive always true classifier in terms of precision. Notably, the p@1 scores are consistently low across all models. However, MAMUT-variants of BERT and MPBERT demonstrate a clear improvement across all metrics. This suggests that these models remember mathematical identities better and are more effective at applying this knowledge to previously unseen formulas.
37


Reviews:
[{"review_id": "rKhkTYIoPC", "reviewer": "Reviewer_dURn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- Fills a key gap in symbolic mathematical data for LLMs.\n- Well-structured augmentation using symbolic parsing, semantic grouping, and diverse LaTeX printing.\n- with >30 versions per formula on average. Supports multiple downstream tasks**: MLM, NSP, contrastive learning, MIR.\n- *Code and data available.\n\n**Weakness**\n- Does not benchmark improvement in large scale LLM performance.\n- Lack of quality evaluation especially human evaluation on the generated datasets\n- Complex LaTeX formulas might fail to parse or generate valid variants.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed MAMUT (Math Mutator): a framework developed to create diverse and specialized datasets for training language models on mathematical content. The core idea is to produce: 1. Mathematically Equivalent Variants (EquVG) – through symbolic transformations such as commutativity, associativity, variable/function substitution, and LaTeX formatting variants. 2. Falsified Variants (FalseVG) – formulas that look similar but are mathematically incorrect, used as hard negatives for classification or retrieval tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- benchmark improvement in large scale LLM performance.\n- Quality evaluation especially human evaluation on the generated datasets", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sb3RBWpmMY", "reviewer": "Reviewer_AzAm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Focusing on a special kind of text data, math formula, and analyzed its key difference from raw texts. This kind of data is quite important in reasoning and scientific domains.\n2. By proposing rule-based framework in detail, the proposed framework MAMUT can cover a large protion of math formulas, so that it can generate data of enough quantity.\n\nWeakness:\n1. The target of this work is to increase diversity of math formula and further use it as text to help training LLMs. But weakness of LLM in processing math formulas can somehow solved by processing codes, such as Latex. Therefore, the necessary of this work is kind of questionable.\n2. This work focuses on mathematical formulas, while formulas are also common in fields like physics. Current formula corpus seems to be weak as they are from a single field.\n3. Author provides quantities and some statistical results about the generated dataset, but its effectiveness is not examined by experiments, making this work incomplete.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to enhance quantity and quality of mathematical fomulas. It proposes a framework to modify existing math formulas considering several property and keys of this kind of data. Based on the proposed framework, quantity of math formulas are increased for hundreds times.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Authors are expected to present experiments to show effectiveness of proposed data framework in training LLMs such ChatGPT, Claude, Qwen, InternLM, and etc. \nBesides the normal LLMs for plain texts, LLMs for code that can process LaTex also should be included in comparison.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fmtQe5rtLw", "reviewer": "Reviewer_krLM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n+ S1: Aiming at the limitations of existing datasets in terms of scalability and diversity, the author generates different LaTeX representations by randomly changing variable and function identifiers and utilizing mathematical properties. This is an interesting piece of work.\n+ S2: The author has open-sourced most of the code, and the appendix materials are quite comprehensive.\n+ S3: The proposed method is clear, and easy-to-understand examples are provided at the same time.\n\n**Weaknesses:**\n+ W1: It seems that the performance of existing models (such as MathBERT) on the proposed BMK has not been given.\n+ W2: Regarding how to evaluate the quality of the generated datasets, I think the proposed method is similar to using existing large language models for rollout and then conducting rule matching and extraction. High-quality answers are the key to improving my evaluation score.\n+ W3: Equivalent version generation and error version generation are used to enhance the performance of the dataset. However, these operations seem to be exhaustible? But in reality, mathematical operations are difficult to exhaust.\n+ W4: How can we ensure that when generating equivalent operations, the model's understanding of mathematical formulas is based on deep semantics rather than shallow similarity features?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The author proposes the first systematic mathematical formula mutation framework, MAMUT, which supports symbol substitution, syntactic rewriting, and error injection, significantly enhancing data diversity. Additionally, based on this framework, four large-scale datasets are proposed to help improve the model's ability to understand and represent mathematical formulas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Increase the number of model comparison experiments on the dataset. This is a change that I particularly value.\n2. **Recommend adding more discussions on the quality assessment of the dataset**. Meaningful discussions will be key to recommending acceptance.\n3. The manuscript is an interesting piece of work. The authors have put in a substantial amount of effort in creating the dataset. However, more comparisons with other existing mathematical benchmarks (such as MathVista) are needed to enhance the expressiveness of the work.\n4. The figures could incorporate more visual elements. This will enhance the readers' understanding.", "broader_impact_concerns": "The work does not raise concerns regarding ethical implications. I believe this is a meaningful contribution to the community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UzsIEfpaNi", "reviewer": "Reviewer_RUmQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This work targets a practical yet underexplored task. This could enhance multiple tasks, including retrieval, AI4SCI, reasoning, etc. Identifying such a niche area that can have broad implications is commendable and shows the potential value of the research.\n2. The proposed framework is technically sound: Parsing a LaTeX formula into an operator tree, mutating the operator tree, optionally verifying its validity by SymPy, and finally back-translating it into LaTeX (w/ equivalence-preserving randomization). This technical rigor indicates a well-thought-out approach to the problem at hand.\n3. The curated datasets are both large in size and broad in domain (manually constructed + adapted from existing large-scale math datasets).\n\n### Weaknesses\n1. **Predefined mutation rules are relatively limited**:\n\t1. EquVG is limited to symbol substitution. However, in real-world scenarios, many more potential equivalent transformations could be considered, e.g., $1 + 2 + \\dots + n$ and $\\sum_{i=1}^n i$ (the elimination of ellipsis);\n\t2. Rules of LaTeX printing randomization are also limited, e.g., recursive application of the same function $f$ can be both denoted as $f^{(3)}(x)$ and $f(f(f(x)))$.\n\n2. **Potential validity issues**. This work doesn't lay adequate emphasis on ensuring the validity of formulas generated by EquVG. The authors claim \"the (general) validity of each used expression is verified using SymPy to ensure high data quality\" in Sec. 6. However, SymPy check is insufficient for more complex scenarios. For example, variables of different domains/types might result in different behaviors:  $\\lim_{n \\rightarrow \\infty} \\sin(n\\pi) = 0$ (human tend to assume $n \\in \\mathbb N$). Upon replacing $n$ to $x$, $\\lim_{x \\rightarrow \\infty} \\sin(x\\pi)$ does not exist if $x\\in \\mathbb R$.\n\n3. **Potential false negatives**. It appears that only the MF dataset, which is augmented by EquVG, has been verified using SymPy. The results of FalseVG have not been verified, which may lead to false negatives in the dataset.\n\n4. **Insufficient baselines**. This submission claims that the MAMUT framework can augment mathematical data and subsequently enhance mathematical embedding. However, there is a lack of supporting experiments for this claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Focused on a novel, practical yet underexplored task: generating equivalent variants and inequivalent hard negatives given a mathematical formula to better train models' encoding of mathematical content.\n2. Proposed a structured framework, **MAMUT**, to address the aforementioned task: parsing a LaTeX formula into a symbolic expression, mutating it, and outputting its LaTeX version. Mutations include:\n\t1. *EquVG*: Generating mathematically equivalent variants by symbol substitution based on predefined symbol groups;\n\t2. *Randomized LaTeX Printing*: Generating mathematically equivalent variants by randomizing LaTeX code based on predefined rules;\n\t3. *FalseVG*: Generating similar-looking but inequivalent variants by predefined modification strategies.\n3. Manually curated NMFT dataset (composed of 71 named mathematical formulas) and subsequently constructed four larger-scale datasets, MF, MT, NMF, and MFR, for model training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Typos.\n\t- Sec. 5, Redundant language \"slightly modifies formulas slightly to\"\n\t- Table 5. Inconsistency between \"Generated Versions\" in title and \"Generated Entries\" in caption.\n2. (Critical for recommendation) **Add supporting experiments**, such as MIR comparison between a model trained using the constructed datasets and relevant baselines.\n3. Add more details about SymPy implementation to the main text, especially verifying the validity and falsity of generated expressions.\n4. (Additional comment, not a requested change) It seems that \"symbol substitution\" in EquVG bears a resemblance to $\\alpha$-conversion in Definitional equality[1]. Hopefully, the task of \"generating equivalent/falsified mathematical formula\" could benefit from a formal verification perspective.\n\n[1] https://rocq-prover.org/doc/v8.18/refman/proofs/writing-proofs/equality.html", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pq4uQjyCnF", "reviewer": "Reviewer_CfKK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-The framework addresses a significant gap in mathematical language processing by automating the generation of diverse and challenging datasets.\n\n-The use of symbolic expressions (via SymPy) to manipulate formulas ensures mathematical correctness and scalability.\n\n-The datasets are well-documented, extensive, and target various tasks, making them highly useful for advancing mathematical language models.\n\nWeaknesses:\n\n-While the paper provides a statistical analysis of the datasets, there is no empirical evaluation of how these datasets improve the performance of mathematical language models.\n\n-The paper lacks benchmarks or comparisons with existing mathematical datasets (e.g., MATH, ARQMath).\n\n-The datasets are primarily tailored for LaTeX-based mathematical corpora. It is unclear how well MAMUT could generalize to other representations of mathematical notation (e.g., MathML).\n\n-The paper does not provide an error analysis of the generated data, particularly for falsified examples. An analysis of failure cases would help identify potential biases or shortcomings in the framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "his paper introduces Math Mutator (MAMUT), a novel framework designed to generate diverse, high-quality datasets of mathematical formulas and texts by leveraging symbolic representations, substitutions, and falsifications of mathematical expressions in LaTeX. MAMUT addresses the challenges of limited diversity and scalability in existing mathematical datasets, which are crucial for training mathematical language models. Two algorithms, EquVG (Equivalent Version Generation) and FalseVG (Falsified Version Generation), for generating mathematically valid and invalid variations of formulas. Methods for parsing and manipulating LaTeX formulas into symbolic expressions using SymPy, enabling substitutions, falsifications, and randomized LaTeX printing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include experiments demonstrating how the generated datasets improve the performance of mathematical language models on tasks like MIR, equation completion, or formula embedding.\n\n- Compare the diversity and quality of MAMUT-generated datasets with existing datasets, such as ARQMath, MATH, or GSM8K.\n\n- Include a case study that fine-tunes an existing mathematical language model (e.g., MathBERT, LLMs like Qwen2 or LLaMA) on the MAMUT datasets and evaluates its performance on a relevant task.\n\n- Provide an error analysis of the generated datasets, especially for falsified examples, to assess the quality and reliability of FalseVG.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)