Pretrained Language Models are Symbolic Mathematics Solvers too!

1 Introduction . Deep learning is a ubiquitous choice in solving statistical pattern recognition problems of regression and classification . With a large training data set and compute power , they have proven to be very effective and achieve state-of-the-art performance in a wide range of tasks in natural language processing , computer vision , speech recognition , sentiment analysis , etc ( Lu et al. , 2021 ) . Though deep learning triumphs in the statistical domain ( Bengio et al. , 2003 ) , there is an active interest in extending deep networks in symbolic computation ( Lample & Charton , 2019 ; Davis , 2019 ; Allamanis et al. , 2017 ; Zaremba et al. , 2014 ; Loos et al. , 2017 ) . There are mainly two motivations for this : ( i ) performing symbolic mathematical tasks , such as symbolic integration and solving differential equations , in deep net architectures , and ( ii ) applying neural networks in the domain automated theorem proving , computer algebra systems , and natural language understanding ( NLU ) that requires a symbolic knowledge system . The key capability of symbolic computation is that symbols maintain their identity as they do multiple roles while deep neural networks exploit shared representation and composition . This paper uses a pretrained language model to solve symbolic mathematics tasks , particularly symbolic integration and differential equations . We show our pretrained transformer architecture on language translation is expressive enough to solve large class symbolic mathematics such as function integration and differential equations , which have traditionally been approached using logic and exhaustive search . Moreover , our pretrained model is sample efficient and compute efficient–i.e. , it requires fewer epochs to converge to good accuracy . The first major work of solving symbolic mathematics with transformer architecture is by Lample & Charton ( 2019 ) . They use the transformer model that is mainly used for NLP tasks to solve the symbolic computation . They first re-frame the mathematical equations as text sequences and then solving those equations as a sequence-to-sequence translation . Their transformer model catches pattern in the mathematical expressions , e.g. , the expressions of the form sin−1 ( x ) will have its primitive as 1√1−x2 . We extend the work of Lample & Charton ( 2019 ) and train their symbolic math dataset by fine-tuning pretrained translation models to solve the downstream task of symbolic mathematics . The pretrained language model will transfer the syntactic and semantic structure of the present in the language , mathematical expressions represented as trees . The inherent limitation between the manyto-one map between mathematical expression and tree encoding is partially regularized by the pre-training with the language translation . For example , the same mathematical expressions 7 + 3× ( 5 + 2 ) and 3× ( 5 + 2 ) + 7 are represented are encoded as different trees . We regularize ( penalize ) this freedom of expression of encoding a mathematical expression by multiple trees by pretraining our transformer model with language translation . The sentence in a language has an order as specified by the famous quote by J. R. Firt “ You shall know a word by the company it keeps. ” . Unlike language , where the meaning of a word is given by its neighbors , the value of a mathematical sub-expression ( mathematical word ) is not influenced by its neighboring expressions . In their training data set generation for function integration , mathematical expressions F and G are generated and the corresponding derivatives f and g are computed . The training data set are the tuples ( f , F ) , ( g , G ) and a new integration function data set Fg is generated ( assuming ( fG , ∫ fG ) is in the training set ) through IBP ( Integration By Parts ) 1 method as : ∫ Fg = FG− ∫ fG . Their vanilla transformer model during training learns to build the correlation between ∫ Fg and fG for solving symbolic mathematics . We differ from their model by ( i ) forcing our transformer model to develop conditional probability between randomly generated functions PΘ ( f |G ) and PΘ ( g|F ) as follows : P ( fG ) = P ( f |G ) P ( G ) P ( Fg ) = P ( g|F ) P ( F ) where PΘ is our pretrained transformer model and Θ is the learned parameter ( weights and biases ) . By re-framing the problem to a conditional probability model , we bypass the distributions of randomly generated functions P ( F ) and P ( G ) . Our method also shows marginal robustness to different types of data set generation method , as shown in table 4 . ( ii ) Our model ’ s predictions improve even when there is a difference of length between input and output sequence . This is because of the phenomena of heavy-tailed distribution , where the model can generate rare small or large output expressions ( Sornette , 2006 ; Martin & Mahoney , 2018 ) . Our model is less sensitive with large difference of length between input and output mathematical expressions ( i.e. , the problem and the solution sequence . ) as explained in Section 3 . The paper is organized as follows : In Section 2 we discuss the prediction of our pretrained transformer model in the language conditional probability and optimization , Section 3 discusses our proposed of heavy-tailed self-regularization under the mild condition of our pretraining , Section 4 discusses experimental setting and methodology , architecture , datasets , and the evaluation metric , and Section 5 poses the following research questions and answers them : 1 . Does this pretrained model help us to use less data for fine-tuning ? 2 . Does the result of this fine-tuning depend on the languages used for pretraining ? 3 . How robust this fine-tuned model is with respect to distribution shift of test data in comparison with fine-tuning data ? Section 6 , discusses literature review , and finally , Section 7 concludes the paper . 1More details about the datasets are explained in section 4.2 . 2 Problem Formulation . Mathematical expressions can be depicted as binary-unary trees , with operators in the form of internal nodes , operands in the form of children , and numbers , constants , and variables in the form of leaves ( Lample & Charton , 2019 ) . These trees can be transformed into a sequence of math letters by traversing them in some specific order . In this paper , a tree of symbolic mathematical expressions is scanned by the prefix traversal to produce a sequence corresponding to the mathematical expression . We formulate our symbolic mathematics as a Seq2Seq translation problem with a large scale pretrained mBART ( and Marian-MT ) transformer . The pretrained transformer is retrained with random expressions data set for the case function integration and differential equation . The training dataset for both tasks is a tuple of mathematical expressions in the format of ( fproblem , fsolution ) . Our pretrained transformer model PΘ solves the symbolic mathematics task by minimizing the prediction loss e as follows : minimize Θ 1 n n∑ i=1 e ( PΘ ( fproblem ) , fsolution ) ( 1 ) where Θ is the learned parameter and n is the number of samples . 3 Theory . Pretraining of the mBART transformer PΘ is done by Seq2Seq translation between the source language English and target language of Romanian . The model parameters Θ as expressed in Equation 1 are learned with gradient descent during the translation task . Fine-tuning the model on symbolic data set allows the model to transfer the knowledge of language translation . Our model inputs the mathematical expressions and predicts the output mathematical expressions of the shortest length . The model searches its big hypothesis space and finds the optimum hypothesis that outputs the shortest mathematical sequence . Searching the big hypothesis space incurs huge optimization cost and also the optimization surface is non-convex . Our model generalizes using the phenomena of self-regularization , which is a complex interpolation process between two parameter spaces . The parameter space Θ of our transformer model lie in a very high-dimensional space and is poorly understood . We model our parameter space as a Normed space . The learned parameter space Θtranslation = ( Rd , ||.||Θtranslation ) is a d-dimensional normed space where d is very large . The learned parameter space comes after the training from English to Romanian translation . Fine-tuning the model with mathematical expressions is a perturbation of the parameter space Θtranslation to a new parameter space Θfine−tuning . The fine-tuning normed space can similarly be defined as Θfine−tuning = ( Rd , ||.||Θfine−tuning ) . Our transformer model during the fine-tuning does a complex interpolation between normed spaces from Θtranslation to Θfine−tuning with a bounded variance ( Andoni et al. , 2018a ; b ) . During the fine-tuning of our model with a new test sample Q ⊆ Θfine−tuning of symbolic expression , our model does an approximate nearest neighbor approximation search and find closet point in P ⊆ Θtranslation in the translation metric space . For given c ≥ 1 and r > 0 our transformer model finds point P such that distance dX ( q , p ) ≤ r with c−Approximate Near Neighbor Search ( c-ANN ) . The large size of the transformer parameter space , it is randomly partitioned into translation space Θtranslation and fine-tuning space Θfine−tuning . Each of the partition of translation space and fine-tuning space is prepossessed as a data structure of Celltranslation and Cellfine−tuning . The Celltranslation ’ s are then represented as a binary tree as shown in1 . Given a query data structure Cellfine−tuning during the symbolic mathematics tasks , our transformer model searches its Cellfine−tuning binary tree to find the approximate match . The normed space restrictions of Θtranslation and Θfine−tuning guarantees the c-Approximate Near Neighbor Search ( c-ANN ) in polynomial time . The self-regularization process comes from the underlying high-dimensional geometric structure of Θtranslation and Θfine−tuning . 4 Experimental Setting . We evaluate a diverse set of symbolic mathematical data sets as introduced in Lample & Charton ( 2019 ) . The tasks studied in these datasets include symbolic integration and solving differential equations of order one and two . Mainly , we are interested in whether pretrained language models are genetically capable of solving these tasks with fewer data . Moreover , whether the language that they have been pretrained on impacts their result after transfer learning . In section 5 , we will do this empirical study by asking structured research questions . 4.1 Architecture . We use the Marian model ( Junczys-Dowmunt et al. , 2018 ) and the mBART model ( Liu et al. , 2020 ) , pre-trained on different translation tasks by the NLP group at the University of Helsinki and Facebook , using the Marian model and the mBART model of the famous NLP framework , Hugging-Face ( Wolf et al. , 2019 ) . Both models follow the famous transformer architecture introduced in Vaswani et al . ( 2017 ) . The Hugging-Face mBART model has an embedding size of 1024 , with 16 attention heads and 12 layers . The Marian-MT model we use ( only ) in section 5.2 , has an embedding size of 512 , with 8 attention heads and 6 layers . The Marian Model and the mBART model have approximately 74 and 610 a million parameters . The Parameter counts may vary depending on the vocab size of the language they have been pretrained on . We also train the model used in Lample & Charton ( 2019 ) with the same parameters as the mBart model ( i.e. , with an embedding size of 1024 , 12 layers and 16 attention heads . ) .

Summary: This work investigate the problem of whether pretraining on language task such as machine translation could help with solving symbolic mathematics problems. Specifically they focused on solving symbolic integration and differential equations using the dataset similar to [Lample & Charleston, 2019]. The authors argued that finetuning a pretrained transformer model could get similar or better accuracy with fewer epochs. They then investigated the effect of the choice of language pairs and how the model works under domain shifts.