Pretrained Language Models are Symbolic Mathematics Solvers too!

1 Introduction . Deep learning is a ubiquitous choice in solving statistical pattern recognition problems of regression and classification . With a large training data set and compute power , they have proven to be very effective and achieve state-of-the-art performance in a wide range of tasks in natural language processing , computer vision , speech recognition , sentiment analysis , etc ( Lu et al. , 2021 ) . Though deep learning triumphs in the statistical domain ( Bengio et al. , 2003 ) , there is an active interest in extending deep networks in symbolic computation ( Lample & Charton , 2019 ; Davis , 2019 ; Allamanis et al. , 2017 ; Zaremba et al. , 2014 ; Loos et al. , 2017 ) . There are mainly two motivations for this : ( i ) performing symbolic mathematical tasks , such as symbolic integration and solving differential equations , in deep net architectures , and ( ii ) applying neural networks in the domain automated theorem proving , computer algebra systems , and natural language understanding ( NLU ) that requires a symbolic knowledge system . The key capability of symbolic computation is that symbols maintain their identity as they do multiple roles while deep neural networks exploit shared representation and composition . This paper uses a pretrained language model to solve symbolic mathematics tasks , particularly symbolic integration and differential equations . We show our pretrained transformer architecture on language translation is expressive enough to solve large class symbolic mathematics such as function integration and differential equations , which have traditionally been approached using logic and exhaustive search . Moreover , our pretrained model is sample efficient and compute efficient–i.e. , it requires fewer epochs to converge to good accuracy . The first major work of solving symbolic mathematics with transformer architecture is by Lample & Charton ( 2019 ) . They use the transformer model that is mainly used for NLP tasks to solve the symbolic computation . They first re-frame the mathematical equations as text sequences and then solving those equations as a sequence-to-sequence translation . Their transformer model catches pattern in the mathematical expressions , e.g. , the expressions of the form sin−1 ( x ) will have its primitive as 1√1−x2 . We extend the work of Lample & Charton ( 2019 ) and train their symbolic math dataset by fine-tuning pretrained translation models to solve the downstream task of symbolic mathematics . The pretrained language model will transfer the syntactic and semantic structure of the present in the language , mathematical expressions represented as trees . The inherent limitation between the manyto-one map between mathematical expression and tree encoding is partially regularized by the pre-training with the language translation . For example , the same mathematical expressions 7 + 3× ( 5 + 2 ) and 3× ( 5 + 2 ) + 7 are represented are encoded as different trees . We regularize ( penalize ) this freedom of expression of encoding a mathematical expression by multiple trees by pretraining our transformer model with language translation . The sentence in a language has an order as specified by the famous quote by J. R. Firt “ You shall know a word by the company it keeps. ” . Unlike language , where the meaning of a word is given by its neighbors , the value of a mathematical sub-expression ( mathematical word ) is not influenced by its neighboring expressions . In their training data set generation for function integration , mathematical expressions F and G are generated and the corresponding derivatives f and g are computed . The training data set are the tuples ( f , F ) , ( g , G ) and a new integration function data set Fg is generated ( assuming ( fG , ∫ fG ) is in the training set ) through IBP ( Integration By Parts ) 1 method as : ∫ Fg = FG− ∫ fG . Their vanilla transformer model during training learns to build the correlation between ∫ Fg and fG for solving symbolic mathematics . We differ from their model by ( i ) forcing our transformer model to develop conditional probability between randomly generated functions PΘ ( f |G ) and PΘ ( g|F ) as follows : P ( fG ) = P ( f |G ) P ( G ) P ( Fg ) = P ( g|F ) P ( F ) where PΘ is our pretrained transformer model and Θ is the learned parameter ( weights and biases ) . By re-framing the problem to a conditional probability model , we bypass the distributions of randomly generated functions P ( F ) and P ( G ) . Our method also shows marginal robustness to different types of data set generation method , as shown in table 4 . ( ii ) Our model ’ s predictions improve even when there is a difference of length between input and output sequence . This is because of the phenomena of heavy-tailed distribution , where the model can generate rare small or large output expressions ( Sornette , 2006 ; Martin & Mahoney , 2018 ) . Our model is less sensitive with large difference of length between input and output mathematical expressions ( i.e. , the problem and the solution sequence . ) as explained in Section 3 . The paper is organized as follows : In Section 2 we discuss the prediction of our pretrained transformer model in the language conditional probability and optimization , Section 3 discusses our proposed of heavy-tailed self-regularization under the mild condition of our pretraining , Section 4 discusses experimental setting and methodology , architecture , datasets , and the evaluation metric , and Section 5 poses the following research questions and answers them : 1 . Does this pretrained model help us to use less data for fine-tuning ? 2 . Does the result of this fine-tuning depend on the languages used for pretraining ? 3 . How robust this fine-tuned model is with respect to distribution shift of test data in comparison with fine-tuning data ? Section 6 , discusses literature review , and finally , Section 7 concludes the paper . 1More details about the datasets are explained in section 4.2 . 2 Problem Formulation . Mathematical expressions can be depicted as binary-unary trees , with operators in the form of internal nodes , operands in the form of children , and numbers , constants , and variables in the form of leaves ( Lample & Charton , 2019 ) . These trees can be transformed into a sequence of math letters by traversing them in some specific order . In this paper , a tree of symbolic mathematical expressions is scanned by the prefix traversal to produce a sequence corresponding to the mathematical expression . We formulate our symbolic mathematics as a Seq2Seq translation problem with a large scale pretrained mBART ( and Marian-MT ) transformer . The pretrained transformer is retrained with random expressions data set for the case function integration and differential equation . The training dataset for both tasks is a tuple of mathematical expressions in the format of ( fproblem , fsolution ) . Our pretrained transformer model PΘ solves the symbolic mathematics task by minimizing the prediction loss e as follows : minimize Θ 1 n n∑ i=1 e ( PΘ ( fproblem ) , fsolution ) ( 1 ) where Θ is the learned parameter and n is the number of samples . 3 Theory . Pretraining of the mBART transformer PΘ is done by Seq2Seq translation between the source language English and target language of Romanian . The model parameters Θ as expressed in Equation 1 are learned with gradient descent during the translation task . Fine-tuning the model on symbolic data set allows the model to transfer the knowledge of language translation . Our model inputs the mathematical expressions and predicts the output mathematical expressions of the shortest length . The model searches its big hypothesis space and finds the optimum hypothesis that outputs the shortest mathematical sequence . Searching the big hypothesis space incurs huge optimization cost and also the optimization surface is non-convex . Our model generalizes using the phenomena of self-regularization , which is a complex interpolation process between two parameter spaces . The parameter space Θ of our transformer model lie in a very high-dimensional space and is poorly understood . We model our parameter space as a Normed space . The learned parameter space Θtranslation = ( Rd , ||.||Θtranslation ) is a d-dimensional normed space where d is very large . The learned parameter space comes after the training from English to Romanian translation . Fine-tuning the model with mathematical expressions is a perturbation of the parameter space Θtranslation to a new parameter space Θfine−tuning . The fine-tuning normed space can similarly be defined as Θfine−tuning = ( Rd , ||.||Θfine−tuning ) . Our transformer model during the fine-tuning does a complex interpolation between normed spaces from Θtranslation to Θfine−tuning with a bounded variance ( Andoni et al. , 2018a ; b ) . During the fine-tuning of our model with a new test sample Q ⊆ Θfine−tuning of symbolic expression , our model does an approximate nearest neighbor approximation search and find closet point in P ⊆ Θtranslation in the translation metric space . For given c ≥ 1 and r > 0 our transformer model finds point P such that distance dX ( q , p ) ≤ r with c−Approximate Near Neighbor Search ( c-ANN ) . The large size of the transformer parameter space , it is randomly partitioned into translation space Θtranslation and fine-tuning space Θfine−tuning . Each of the partition of translation space and fine-tuning space is prepossessed as a data structure of Celltranslation and Cellfine−tuning . The Celltranslation ’ s are then represented as a binary tree as shown in1 . Given a query data structure Cellfine−tuning during the symbolic mathematics tasks , our transformer model searches its Cellfine−tuning binary tree to find the approximate match . The normed space restrictions of Θtranslation and Θfine−tuning guarantees the c-Approximate Near Neighbor Search ( c-ANN ) in polynomial time . The self-regularization process comes from the underlying high-dimensional geometric structure of Θtranslation and Θfine−tuning . 4 Experimental Setting . We evaluate a diverse set of symbolic mathematical data sets as introduced in Lample & Charton ( 2019 ) . The tasks studied in these datasets include symbolic integration and solving differential equations of order one and two . Mainly , we are interested in whether pretrained language models are genetically capable of solving these tasks with fewer data . Moreover , whether the language that they have been pretrained on impacts their result after transfer learning . In section 5 , we will do this empirical study by asking structured research questions . 4.1 Architecture . We use the Marian model ( Junczys-Dowmunt et al. , 2018 ) and the mBART model ( Liu et al. , 2020 ) , pre-trained on different translation tasks by the NLP group at the University of Helsinki and Facebook , using the Marian model and the mBART model of the famous NLP framework , Hugging-Face ( Wolf et al. , 2019 ) . Both models follow the famous transformer architecture introduced in Vaswani et al . ( 2017 ) . The Hugging-Face mBART model has an embedding size of 1024 , with 16 attention heads and 12 layers . The Marian-MT model we use ( only ) in section 5.2 , has an embedding size of 512 , with 8 attention heads and 6 layers . The Marian Model and the mBART model have approximately 74 and 610 a million parameters . The Parameter counts may vary depending on the vocab size of the language they have been pretrained on . We also train the model used in Lample & Charton ( 2019 ) with the same parameters as the mBart model ( i.e. , with an embedding size of 1024 , 12 layers and 16 attention heads . ) .

Summary: The authors use transformers pre-trained on machine translation tasks on the integration and differential equation datasets proposed by  Lample and Charton (Deep Learning for Symbolic Mathematics, ICLR 2020). They show that pre-training on "pure language" tasks help models, in terms of learning speed, accuracy, and, to a lesser extent, out of distribution generalization. Pre-training on different language pairs, on the other hand, has little impact on performance.