Pretrained Language Models are Symbolic Mathematics Solvers too!
Authors: Kimia Noorbakhsh
Modar Sulaiman
Mahdi Sharifi
KALLOL ROY
Pooyan Jamshidi
Venue: ICLR 2022 Submitted (2022)

Abstract: Solving symbolic mathematics has always been of in the arena of human ingenuity that needs compositional reasoning and recurrence. However, recent studies have shown that large scale language models such as transformers are universal and surprisingly can be trained as  a sequence-to-sequence task to solve complex mathematical equations. These large transformer models need humongous amounts of training data to generalize to unseen symbolic mathematics problems. In this paper, we present a sample efficient way of solving the symbolic tasks by first pretraining the transformer model with language translation and then fine-tuning the pretrained transformer model to solve the  downstream task of symbolic mathematics. We achieve comparable accuracy on the integration task with our pretrained model while using around $1.5$ orders of magnitude less number of training samples with respect to the state-of-the-art deep learning for symbolic mathematics. The test accuracy on differential equation tasks is considerably lower comparing with integration as they need higher order recursions that are not present in language translations. We pretrain our model with different pairs of language translations. Our results show language bias in solving symbolic mathematics tasks. Finally, we study the robustness of the fine-tuned model on symbolic math tasks against distribution shift, and our approach generalizes better in distribution shift scenarios for the function integration.

Under review as a conference paper at ICLR 2022
PRETRAINED LANGUAGE MODELS ARE SYMBOLIC MATHEMATICS SOLVERS TOO!
A Anonymous authors
Paper under double-blind review
ABSTRACT
Solving symbolic mathematics has always been of in the arena of human ingenuity that needs compositional reasoning and recurrence. However, re- cent studies have shown that large scale language models such as transform- ers are universal and surprisingly can be trained as sequence-lo-sequeee task to solve complex mathematical equations. These large transformer models need humongous amounts of training data to generalize to unseen symbolic mathematics problems. In this paper, we present a sample effi- cient way of solving the symbolic tasks by first pretraining the transformer model with language translation and then fine-tuning the pretrained trans- former model solve the downstream task of symbolic mathematics. We achieve comparable accuracy on the integration task with our pretrained model while using around 1.5 orders of magnitude less number of train- ing samples with respect to the state-of-the-art deep learning for symbolic mathematics. The lest. accuracy on differential equation tasks is consider- ably lower comparing with integration as they need higher order recursions that are not present in language translations. We pretrain our model with different pairs of language translations. Our results show language bias in solving symbolic mathematics tasks. Finally, we study the robustness of the fine-tuned model on symbolic math tasks against distribution shift, and our approach generalizes better in distribution shift scenarios for the function integration.
INTRODUCTION
Deep learning is ubiquitons choice in solving statistical pattern recognition problems of regression and classification. With a large training data sot and compute power, they have proven to be very effective and achieve state-of-the-art performance in a wide range of tasks in natural language processing, computer vision, speech recognition, sentiment analysis, etc (Lu et al. 2021). Though deep learning triumphs in the statistical domain (Bengio let al.] 2003). ) there is an active interest in extending deep networks in symbolic computation (Lample Charton 2019: Davis 2019 Allamanis et al., 2017 Zaremba et. al. 2014 Loos (tt al. 2017). There are mainly two motivations for this: (i) performing symbolic mathe- matical tasks, such as symbolic integration and solving differential equations, in deep net architectures, and (ii) applying neural networks in the domain automated theorem prov- ing, computer algebra systems, and natural language understanding (NLU) that requires symbolic knowledge system. The key capability of symbolic computation is that symbols maintain their identity as they do multiple roles while deep neural networks exploit shared representation and composition.
This paper uses a pretrained language model to solve symbolic mathematics tasks, partic- ularly symbolic integration and differential equations. We show our pretrained transformer architecture on language translation is expressive enough to solve large class symbolic math- ematics such as function integration and differential equations, which have traditionally been approached using logic and exhaustive search. Moreover, our pretrained model is sample efficient and compute efficient-i.e., it requires fewer epochs to converge to good accuracy. The first major work of solving symbolic mathematics with transformer architecture is by Lample & Charton (2019). They the transformer model that is mainly used for NL.P

---

Under review as a conference paper at ICLR 2022
tasks to solve the symbolic computation. They first re-frame the mathematical equations as text sequences and then solving those equations as a sequence-to-sequene translation.
Their transformer model catches pattern in the mathematical expressions, e.g., the expres- sions of the form sin (xx) will have its primitive as VI-z² We extend the work of ofmmle &
Charton (0199) and train their symbolic math dataset by fine-tuning pretrained translation models to solve the downstream task of symbolic mal hematics... The pretrained language model will transfer the syntactic and semantic structure of the present in the language, mathematical expressions represented as trees. The inherent limitation between the many- to-one map between mathematical expression and tree encoding is partially regularized by the pre-training with the language translation. For example, the same mathematical ex- pressions 7+3 (5+ + 2) and 3 x (5 1 2) + 7 are represented are encoded as different trees. We regularize (penalize) this freedom of expression of encoding mathematical expression by multiple trees by pretraining our transformer model with language translation. The sen- Ience in language has an order as specified by the famous quote by J. R. Firt "You shall know a word by the company it keeps.". Unlike language, where the meaning of a word is given by its neighbors, the value of mathematical sub-expression (mathematical word) is not influenced by its neighboring expressions. In their training data set generation for func- tion integration, mathematical expressions F and G are generated and the corresponding derivatives and are computed. The training data set are the tuples F), (g, G) and new integration function data set Fg is generated (assuming (fG. fG) is in the training set) through IBP (Integration By Parts)
method
f -IG-G-JJG..
Their vanilla transformer model during training learns build the correlation between Fg and fG for solving symbolic mathematics. We differ from their model by (i) forcing our transformer model to develop conditional probability between randomly generated functions Po(S)G) and (o( as follows:
P(JG) = P(JG))(() P(Fg) P(g)F)
where Pe is our pretrained transformer model and θ is the learned parameter (weights and biases). By re-framing the problem to a conditional probability model, we bypass the distributions of randomly generated functions P(F) and P(G). Our method also shows marginal robustness to different types of data set generation method, as shown in table A (ii) Our model's predictions improve even when there is difference of length between input and output. sequence. This is because of the phenomena of heavy-tailed distribution, where the model can generate rare small or large output expressions (Sornette) 2006 Martin & Mahoney 2018h. Our model is less sensitive with large difference of length between input and output mathematical expressions (i.e., the problem and the solution sequence.) as explained in Section 3
The paper is organized as follows: In Section 2 we discuss the prediction of our pretrained transformer model in the language conditional probability and optimization, Section 3 dis- cusses our proposed of heavy-tailed reguaaagulii under the mild condition of our pre- training, Section discusses experimental setting and methodology, architecture, datasets, and the evaluation metric, and Section 5 poses the following research questions and answers them:
1. Does this pretrained model help us to use less data for fine-tuning?
2. Does the result of this fine-tuning depend on the languages used for pretraining?
3. How robust this fine-tuned model is with respect to distribution shift of test data in comparison with fine-tuning data?
Section 6 discusses literature review, and finally, Section concludes the paper.
Moore details about the datasets are explained in section 4.2
2

---

Under review as a conference paper at ICLR 2022
2 PROBLEM FORMULATION
Mathernttical expressions can be depicted as binary-unary trees, with operators in the form of internal nodes, operands in the form of children, and numbers, constants, and variables in the form of leaves (Lample & Charton 2019). These trees can be transformed into sequence of math letters by traversing them in some specific order. In this paper, a tree of symbolic mathematical expressions is scanned by the prefix traversal to produce sequenee corresponding to the mathematical expression. We formulate our symbolic hematiis as Seq2Seq translation problem with a large scale pretrained mBART (and Marian-MT) transformer. The pretrained transformer is retrained with random expressions data set for the case function integration and differential equation.
The training dataset for both tasks is a tuple of mathematical expressions in the format of (Sproblem : Ssoluiion). Our pretrained transformer model Pe solves the symbolic mathematicn task by minimizing the prodiction loss e as follows:
minimize- 1
Σ
(p(Pe(Urraseem) fsolutiin))
(1)
where θ is the learned parameter and n is the nunber of samples.
3 THEORY
Pretraining of the mBART translormer Pe is done by Seq2Seq translation between the source language English and target language of Romanian. The model parameters θ as expressed in Equation [ are learned with gradient descent during the translation task. Fine-tuning the model on symbolic data set allows the model to transfer the knowledge of language translation. Our model inputs the mathematical expressions and predicts the out- put mathematical expressions of the shortest length. The model searches its big hypothesis space and finds the optimuum hypothesis that outputs the shortest mathematical sequence. Searching the big hypothesis space incurs huge optimization cost and also the optimization surface is non-convex. Our model generalizes using the phenomena of relf-regularization, which is complex interpolation process between two parameter spaces. The parameter space θ of our transformer model lie in very high-dinensional space and is poorly under- stood. We model our parameter space as a Normed space. The learned parameter space Orvanslation = (Rdd, (irentttttt))))) is a d-dimensional normed space where d is very large. The learned parameter space comes after the training from English to Romanian translation. Fine-tuning the model with mathematical expressions is perturbation of the parameter space Otranslation to a new parameter space -tuningn The fine-tuning normed space can similarly be defined as fine-tuning = (Rd 11-11e (l Our transformer model during the fine-tuning does complex interpolation between normed spaces from Otranstation to O fine-tuning with a bounded variance (Andoni et al. 2018a b).
During the finc-tuning of our model with new test sample Q C O fine-tuning of symbolic expression, our model does an approximate nearest neighbor approximation search and find closet point in P Otranstation in the translation metric space. For given e and our transformer model finds point P such that distance dx (4,p) 3) S r with e- c-Approximate Near Neighbor Search (c-ANN). The large size of the transformer parameter space, it is randomly partitioned into translation space Orranstation and fine-tunin space θ fine- fine-tuning Each of the partition of translation space and fine-tuning space is prepossessed as a data structure of Celltranstation and Cell fine-tanng The Celleranslation 's are then represented as a binary tree as shown in[ Given a query data structure Cell iine- during the symbolic mat hematicss tasks, our transformer model searches its Cell tine-nning binary tree to find the approximate match. The normed space restrictions of θ Oransslation and θ fine-taning guarantees the c-Approximate Near Neighbor Search (c-ANN) in polynomial time. The self-regularization process comes from the underlying high-dimensional geometric structure of Otranstation and ffnce Fine-tuning-
search
3

---

Under review as a conference paper at ICLR 2022
Random Partition of Translation Normed Space O tronslation
Random Partition of Fine- Fine-tuning Space
Cellirattinio
Cellyaanceen" puecun
Lnc runiing
Cell
Cell
ypancbation
Approximate Nearest Neighbor matching
Binary Encoding Translation Space
Figure 1: Interpolation between translation space and fine-tuning space.
EXPERIMENTAL SETTING
We evaluate diverse set of symbolic mathematical data sets as introduced in Lample & Charton 2019h. The tasks studied in these datasets include symbolic integration and solving differential equations of order one and two. Mainly, we are interested whether pretrained language models are genetically capable of solving these tasks with fewer data. Moreover, whether the language that they have been pretrained on impacts their result after transfer learning. In section will do this empirical study by asking structured research questions.
4.1 ARCHITECTURE
We use the Marian model (Junczys-Done Downmutt et al. 2018 and the mBART model Liu lt al. 2020),, pre-trained on different translation tasks by the NLP group at the University of Helsinki and Facebook, using the Marian model and the mBART model of the famous NLP framework, Hugging-Face (Wolf et al.. 2019).
Both models follow the famous transformer architecture introduced in Vaswani et al. 2017 The Hugging-Face mBART model has an embedding size of 1024, with 16 attention heads and 12 layers. The Marian-MT model we use (only) in section 5.2 has an embedding size of 512, with attention heads and layers. The Marian Model and the mBART model have approximately 74 and 610 million parameters. The Parameter counts may vary depending on the vocab size of the language they have been pretrained on. We also train the model used in Lample & Charton (2019) with the same parameters the mBart model (i.c., with an embodding size 1024, 12 layers and 16 attention heads.).
4.2 DATASETS
Thanks to Lample & Charton 2019), there is a good dataset resource for Symbolic Mathe- matics available publicly. In all the experiments this paper, we use the same datasets as Lample & Charton 2019 or generate new datasets using the same generation methods.
For the mathematical integration task, there are three generation methods. Forward (FWD), Backward (BWD), and Integration by Parts (IBP). The forward approach, generates random functions and calculates their integrals with an external symbolic mathematical framework. The backward approach, on the other hand, generates random function and then computes its derivative and add the pair to the dataset with a backward manner. Both backward and forward approaches have some issues. The forward approach is only capable of creating samples that can only be integrated by a mathematical framework and also the samples generated by this approach have short problems with long solutions. The backward approach normally generates samples that the integral is shorter than the equation itself. In contrast to the other two methods, the IBP approach uses the integration by parts formula to generate

---

Under review as a conference paper at ICLR 2022
27.8
ICC
$2.00
1001
190
100
Aonary 50c
23.8
71.8
The Antuur
The Ammur
25.0
10
17.02
006
Bilk
150
3ik
10
1202
DEK
151
Boling Dataut Size
Thkking Tutasr St
Thankig Tuocoe
Thakerg Deroe
Thiking Thtaae Skee
(a) FWD
(b) BWD
(c) TBP
(d) ODE1
(e)
ODE2
Figure 2: The accuracy of our mBART language model and the LC model when trained on different training sample sizes. Panels (a), (b), and (c) are for the integration task.
Accuracy of our models (in percentage (%)) and the Lample & Charton (2019 's model integration and differential equation solving. The number of training samples used train the models in all tasks million. Results tested on test data sets of size 5000 samples.
Table 1:
Integration (FWD) Integration (BWD) Integration (IBP)
ODE 1
ODE 2
Our Model LC's Model
87.4
92.2
86.2
62.2
17.9
79.4
83.4
87.4
71.8
39.9
samples that do not need an external computer algebra framework, but in terms of the equation lengths, it is similar to the FWD approach (generates short problems and long solutions.) [Lample& & Charton 2019). The dalasets for the finst order differential equations are referred as ODEI and the second order differential equations referred as ODE2.
Detailed information about datasets can be found at Lample Charton 20119)
4.3 METRIC
In all of our experiments, report the Accuracy, which is defined as follows:
Accuracy: As discussed in the section 4.3 of Lample & Charton (2019), we can easily calculate the accuracy of our predictions by comparing the generated equation and the reference equation. The generated equation by the models might not be in the same format as the reference equation; therefore, we simplify the difference between the predicted and the reference equation to check whether it is 0 or not. It is also necessary to mention that all of the results in section are reported with the evaluations of beam size 1.
EXPERIMENTAL EVALUATION
In this section, examine the results showing transfer from language translation to solving symbolic math equations and altempt to understand better why this happens and which factors enable this transfer. The following subsections include research questions, how we design experiments to answer them, the discussions the results, and their implications. Note that refer to Lample & Charton 2019 's model results with the keyword LC in our tables and visualizations.
We train models with the Adam optimizer (Kingma & Ba 2015), with learning rate of 10-4. We run all of our experiments with the mBART and the Marian-MT model only for 15 epochs, while train the LC model long the model converges (which usually takes around 100 epochs.).
5.1 DOES THIS PRETRAINED MODEL HELP US TO USE LESS DATA FOR TRAINING?
As studied in Lample & Charton 2019), train transformer architecture on the symbolic math data, we need vast amount of training data for each task to achieve the highest
The experiments with the mBART model were performed on a machine equipped with one RTX A6000 NVIDIA GPU and 48GB memory. The experiments with the Marian-MT model were performed machine equipped with one NVIDIA Tesla V100 GPU and 512GB memory.
5

---

Under review as a conference paper at ICLR 2022
accuracies (in the order of 40 million to 80 million training samples for each task.). We investigate if fine-tuning the pretrained models on language translation tasks on the symbolic math data can help use considerably fewer data in the fine-tuning stage.
In this section, we will use the pretrained mBART (Liu et al., 2020) model for the English to Romanian translation task [ and finc-tune it on our math data (sce section 4.2) We report the accuracy of our models on the integration and differential equation solving in table 1 In this table, we use the same training dataset for both our mBART model and the LC model. We train our mBART model only for 15 epochs for all 5 tasks (FWD, BWD, IBP, ODEI, and ODE2), but we continue the training of the LC model until convergence (which takes around 100 epochs for each task.). We can see in the table [ that our model outperformed in the integration task, with a considerable gap from the L.C model. But it cannot properly perform on the differential equation task, especially the second-order differential equations.
We extend this exploration by running the same experiment for different orders of magnitude of training data (i.e., 10K, 100K, and 1M). We report the test accuracy (see section 4.3) of each experiment for both models (mBART and LC) in figure 2 Our model has higher accuracy in comparison to LC in all tasks and with different traiinn sample sizes, except that in the differential equations the accuracy growth of our model suddenly gets lower than
the LC model when using the 1 million samples for training.
We achieve comparable accuracy on the integration task with our pretrained model while using around 1.5 orders of magnitude less number of training samples than the statc-of-the- art model in Lample & Charton (2019) (i.c, we use million training samples against the 40-80 million samples that Lample & Charton (2019) used for training their model.). As we have discussed previously in the section 3 the mBART language model has already been pretrained by the language translation. During this pretraining, our mBART model searches for that hypothesis that outputs the shortest translated sequence (the shortest Romanian sequence for a given input of English sequence). During the fine-tuning, it uses the same hypothesis learned previously to search for mathematical expressions that has minimum length. Also, because our mBART language model very large, is doing an internal look-up and search for the solutions depth-wise in the mathematical expression tree. The model is thus effectively searching greedily than the LC model.
Note that the accuracies reported for the LC model in table D as well as in tables 3 and are by training this model also with 1 million training samples (the high accuracies (over 95%) reported in Lample & Charton 2019 are achieved by sample sizes of range 20 - 40 million training samples).
5.2
ARE THE RESULTS OF SUCH FINE-TUNING, LANGUAGE DEPENDENT?
We investigate whether different languages used to train our pretrained models impact the results of this transfer learning. We wish to see whether the quality of the results in section 5.1 might have been dependent on the specific source- source-target language in our language model, ie., the learned representations. In other words, the specific language could have been confounder. Therefore, to remove this confound, we fine-tune our symbolic math data on
different pretrained language translation tasks containing various source-target languages.
To be able to perform more experiments on multiple languages (due to the computational costs), we fix our training sample size to 100K samples per task, and we use the pretrained Ma model of Hugging-Fart Face (Wolf et al. 2019 which has already been pretrained on many language translation tasks, and is available online + Since the accuracy of the models based on what we saw in section 5.1 are consistent, we only report the accuracies for the 100K sample dataset. Accuracies will not be optimal, but they are sufficient to answer our question. We test all the experiments on test datasets of size 1000. The results are shown in table 2 As we can see in this table, for each task, different pretrained language has the highest accuracy (indicated in bold case.). For example, in the FWD task the French to
3 'The pretrained mBART model is available at httpp///uuuggiggfeeccc////bbbbb/
en-ro
The pretrained Marian-MT models are available at https///nnggingiooocooooooehiih
6

---

Under review as a conference paper at ICLR 2022
Table 2: Evaluation of accuracy of our Marian-MT model (in percentage (%) on the in- tegration and differential equation solving for different pretrained languages. The highest accuracy is indicated by bold case in cach column (task). We see that the language has no specific impact on the results of this fine-tuning.
Language
Integration (FWD) Integration (BWD)
Integration (IBP)
ODE
ODE 2
English Romanian
38.8
67.8
51.5
23.4
1.8
English- Greek
69.5
48.6
17.3
English- Arabic
43.9
71.3
53.5
16.4
2.7
English- French
47.7
71.4
52.5
18.9
2.9
English Spanish
143.5
70.4
51.8
18.7
3.3
Greek English
39.1
69.1
47.9
16.2
2.2
Arabic English
43.3
69.3
50.7
22.5
2.3
French English
50.5
71.2
52.7
19.7
2.3
Spanish English
40.4
69.9
51.7
20.2
2.0
English model had the highest accuracy and so on. Therefore, table that the results of this fine-tuning approach are not language dependent and our hypothesis that language is confounder for our results is not true.
is also important note that this Marian-MT model has an embedding size of 512, which is twice less than the mBART model (and the LC model) use in section 5.1 But because our goal this section is to study the impact of languages and there are many pretrained models available of Marian-MT, choose to use this model in our language study
5.3 How ROBUST THIS FINE-TUNED MODEL IS WITH THE DISTRIBUTION SHIFT?
As also studied in Lample & Charton 2019 it is important to see whether these transformer models are biased towards the distribution of their training data or not. In order evaluate this concept, we define two different kinds of distribution shift as follows:
The first one is only for the integration task and is similar to section 4.7 described Lample & Charton (2019). Meaning that we will investigale how robust our models trained in 5.1 are when we change their testing distribution. We report the evaluation metrics trained and tested on a different combination of training datasets in table 8
The second kind of distribution shift that we are interested in is due to the modality of the test dataset. This type of distribution shift was not studied by Lample & Charton (2019) and is a new lype of distribution shifts we introduce in this paper. Each training sample we
all tasks (in sections 5.1 and 5.2 has combi- nation of all different types of equations such as polynomial, trgonometric, and logarithmic expressions. We want to see whether a model trained on this type of dataset can generalize to solve type-dominant functions (i.e, functions containing only polynomial equations or containing only trigonometric equations and so on.). Therefore, we generate different types of test data, varying in the kind of equation they represent, such as trigonometric equations, polynomial equations, and loga- rithmic equations. We lest. the ability of our models trained in 5. to see which kinds of equations they can solve better, helping us to understand the impact of linguistic data better. The results are reported in table
Table indicates that our mBART model more robust with respect to the generation distribution shift (i.e,, FWD, BWD and IBP method for integration task.) and can achieve comparable performance in comparison to the pure transformer model (LC) model.
"nvestigatin the effect of embedding size more systematically to the results is considered as future work.

---

Under review as a conference paper at ICLR 2022
To evaluate the robustness of our approach in terms of different equation types, created three different test datasets for each task. The first dataset is polynomial dominant, meaning that the samples of dataset were created mostly by polynomials without using trigonometric and logarithmic functions. The second and third datasets are trigonometric dominant and logarithmic dominant, respectively. This means that the trigonometric dominant dataset was created using mostly trigonometric functions, and the logarithmic dataset was generated using mostly logarithm and exponential functions. T'alle[iidicater that our mBART model is not able to generalize to type dominant equations as well as the LC model can (except in the FWD and BWD approaches of the integration task.). The highest accuracies of both models are in their generalization to solve trigonometric expressions, and the lowest results are in pure polynomial ones. This agrees with our theory (see section 3), because the mBART model tries to find the shortest sequence and the higher order polynomial equations are less compressible. Also, higher order polynomials need accurate precision (F64) for their representation. On the other hand, trigonometric and the logarithmic equations can be compressod into shorter expressions (for example, si)) + cos"(x) 1. or ehe = cos i sin x)), and ;thercfore, the performance these sets of type-dominant test samples are better.
Table 3: Accuracy of the models (in percentage (%)) on function integration. Results are tested on test data sets of size 5000 samples. The models trained on the 1 million sample size training data, dicussed in section 5.1
Integration by parts
Training data
Ours(mBART)
L.C
Ours(mBART)
LC
(ur((mBART)
LC
FWD
87.35
79.12
7.30
6.90
71.20
71.10
BWD
12.82
9.28
92.24
83.40
24.02
17.00
IBP
30.46
28.70
35.00
20.50
86,26
87.44
Table Accuracy our models (in percentage (%)) on the integration and differential equation solving for different pretrained languages. Results are reported on test datasets of different types (polynomial, trigonometric and logarithmic.), and of size 5000.
Testee Type
Metries
Intogration (FWD)
Integration (BWD)
Integration (IBP)
ODE [order
ODE (urder 2]
Ous
60.6
67.8
70.7
39.1
8.9
Polymomiuls
54.7
60.0
60.6
Ours
91.9
87.0
78.0
48.3
10.6
Trigonometric
LC
92.4
85.8
91.8
74.4
Ouns
90.9
75.1
72.4
35.9
6.8
Logarithmic
LC
87.9
73.3
87.96
75.6
72.0
RELATED WORK AND DISCUSSION
TRANSFORMERS IN DIFFERENT MODALITIES
Attention (ahdanau et al. 2014 is powerful mechanism led to recent achievements in developing strong DNN models NLP like the transformer architocture (Vaswani et al. 2017 Attention mechanism has also been used in other tasks such as visual explanation Fukui et al. 2019), video captioning (Yan et al. 2019), healthcare (Choi et al. 2016), object detection [Chorowski al. 2015). and speech recognition [ et al., (0020). The transformer architecture introduced in VVaswani et al.] 2017) is an autoencoder that encodes the input data and then decodes them to the target domain. and just uses self-attention mechanism. It is a breakthrough in NLP and is the base for many language models including bidirectional encoder representations from transformers, BERT, (Devlin et al.) 2019). generative pretrained transformer, GPT-3, (Brown et al. 2020 Text-to- Text Transter Transformer, T5, (Raffel et al.) 2020 and Google's Meena 2020). It has also been successfully used
does not use recurrent modulus
[Adiwardana et al.)
baseline in other tasks
(Chen et al., 2021), image 2019), and visual question answering [Tan & Bansal 2019). Furthermore, Yun et al. (2019) showed that transformers
such as object detection (Carion et al.) 2020), image generation
colorization (Kumar et al.
2021), video understanding (Sun et al.
can universally approximate sequence sequence functions. Therefore, the transformer is
8

---

Under review as a conference paper at ICLR 2022
good choice for transfer learning not only because of their prosperity across different tasks, but also because of its architecture which makes it possible to use the hardware parallelism to train much more big models with much more training data.
6.2 SYMBOLIC COMPUTATION RELATED WORKS
The research on algebraic manipulation systems through computer is quite mature. The early work of solving symbolic integration were the heuristics programs written in LISP. They were named SIN (Symbolic INtegrator), SAINT, and SOIDIER (SOLUtion of Ordi- nary Differential Equations ROUTINE) (Moses] 1967). The obvious motivation during those programs, is the use of symbolic systems as an adjunct to numerical integration programs which involves parameters. SAINT program of symbolic integration shows the capability of a freshman calculus student. Thus, an unmodified SAINT pe.g.,am was of limited use in a practical algebraic system. More powerful programs follow, e.g., MATLAB project by MITRE Corporation, which solves inlegration of rational functions as good as sophomore college stuetts. Though the capabilities of these programs are quite impressive, they mainly use tree search and matching algebraic expressions (pattern matching) as their workhorse. The program started showing its inherent limitation for those expressions which are not in- tegrable in closed form, c.g., f'e de or J = dr. Though there were some attempts of using Edge heuristics to solve those wild integrals, they were mainly unsuccessful. The era of deep neural networks ushers a new hope of solving the symbolic lasks by representing (encoding)
the algebraic expressions in feature space (Lample & Charton 2019 Arabshahi et al. 2018
Allamanis et al., 2017 Zaremba et al.] 1014|ooo et al.) 2017 Trask et al., 2018 Lukasz
Kaiser & Sutskever 2016 Zaremba. & Sutskever 2015 Valipour et al.) 2021 Ling et al.
2017
Polu & Sutskever 2020). So instead of pattern matching on the raw mathematical ex-
pressions done in the pre-deep learning cra programs, these deep models solve the algebraic systems in the feature space. These works on representing the symbolic expressions in a continuous and differential space using deep net architectures show the fundamental differ- ence in the philosophy from the early SIN, SAINT, and SOIDIER pograms. The advantages of using deep net architectures are remarkable in terms of solving the algebraic systems approximately e.g., for those integrals which have no closed form solutions, and the aver- age time complexity to solve. The deep models even started to show creativity on solving complex mathematical expressions, e.g., representing a mathematical expression in multiple ways. Very recently the research community started using language base transformer neural networks to solve symbolic computations (Lample & Charton 2019 Hendrycks et al. 2021). The mathematical expressions are encoded as a sequence and a transformer is trained for pepuence-to-squeeee translation task. The dot product attention module in the trans- former architecture solves symbolic tasks efficiently. [aatonn el al. (2019) takes a different route and created large symbolic mathematics data set. All these research directions point towards the direction of solving mathematics is no more in the genre of human creativity, but a data problem. The unreasonable effectiveness of symbolic mathematics data and large neural architectures show the inevitable future of machine generated mathematical prover and symbolic mathematics.
7 CONCLUSION
Considering success of the transformer architecture in many tasks (Lu et al. 2021). including both language and symbolic mathematics, we proposed transfer learning from a pretrained language model with the transformer architecture for the downstream task of solving sym- bolic mathematical problems such as integration and differential equations. Using multiple experimental evaluation, we showed that these models could achieve competitive perfor- mance (specially in the integration tasks) with transformers fully trained on the symbolic math task without being pretrained on linguistic data. We showed that the language that the transformer model has been pretrained does not have significant impact in this transfer learning. We also evaluated that a model fine-tuned using our approach generalizes better in distribution shift scenarios for integration tasks.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
We have provided all the code and data needed to ro-produce our results as the supple- mentary materials. The Zip file contains README.pdl file explaining how to run the experiments. The code for the LC model is run by the configurable scripts available on the lample & Charton (2019 's GitHub repository
REFERENCES
Daniel Adiwardana, Minh- Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoory Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXio:201.009977, 2020.
Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. Learning continuous semantic representations of symbolic expressions, 2017.
Alexandr Andoni, Assaf Naor, Aleksandar Nikolov, llya Razenshteyn, and Erik Waingarten. Data-dependent hashing via nonlinear spectral gaps. In Proceedings of the 50th Annual
ACM SIGACT Symposium on Theory of Compuling, STOC 2018, pp. 787 800, New York, NY, USA, 2018a. Association for Computing Machinery. ISBN 9781450355599. doi: 10.1145 10.1145/3188742 /311888887433 318811886. URL https://doi oooooogg///// 10. /145/3187/45555 3188846
Alexandr Andoni, Assaf Naor, Aleksandar Nikolov, Ilya Razenshteyn, and Erik Waingarten. Hölder homeomorphissms and approximate nearest neighbors. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pp.
IEEE, 2018b.
Forough Arabshahi, Sameer Singh, and Auimashree Anandkumar. Towards solving differ- ential equations through neural programming. In ICML Workshop on Neural Abstract Machines and Program Induction (NAMPI), 2018.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arxiv: 040.04444.4.0000 73, 2014.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural proba- bilistic language model. The journal of machine learning research, 3:1137-1155, 2003.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. ar preprint arXic:2005.14 14165, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213 229. Springer, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pallern Recognition, pp. 12299 12310, 2021.
Edward Choi, Mohammad Taha Bahadori, Joshua A Kulas, Andy Schuetz, Walter F Stew- art, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. arXie preprint vrXin:1608.0577 2016.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. arXiv preprint arXio:1506. 07503, 2015.
Ernest Davis. The use of deep learning for symbolic integration: A review of (lample and charton, 2019), 2019.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pro-training of deep bidirectional transformers for language understanding, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10705- 10714, 2019.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, UIrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pp. 116- 121, Melbourne, Australia, July 2018. Association for Computational Linguistics. URL httt:://ww... acw.eb...bbb...................b org/ watthology/P18440000000
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (ods.). 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv. org/abs/ ht////1../111..... 1412. 6980
Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. Colorization transformer. arXiv preprint 2rX2::2102.04
2021.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. CoRR, abs/19122 0bs//912.01112, 2019. URL http::/arxixi http:///aaxiv..ooraa//////////// org/ 01412
Wei Li, Kai Liu, Lizhe Zhang, and Fci Cheng. Object detection basod on an adaptive attention mechanism. Scienlific Reports, 10(1):1 13, 2020.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation Learning to solve and explain algebraic word problems, 2017.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. CoRR, Abs/2001.08210, 2020. URL https://arzi argg/bs//000/ 08210
Sarah Loos, Geoffrey Irving, Christian Szegedy, and Cezary Kaliszyk. Deep network guided proof search, 2017.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXio preprint "Xxx:21033.0224 2021.
Charles H. Martin and Michael W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning, 2018
Joel Moses. Symbolic integration. 1967.
Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020.
Colin Raffel Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-67, 2020. URL httt:://mmml wttpppapess//1//20--00---
html
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathemat- ical reasoning abilities of neural models, 2019.
Didier Sornette. Critical phenomena in natural sciences: chaos, fractals, selforganization and disorder: concepts and tools. Springer Science & Business Media, 2006.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the TEEF/CVE International Conference on Computer Vision pp. 7464-7473, 7473, 2019.
11

---

Under review as a conference paper at ICLR 2022
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.0774000 2019.
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units, 2018.
Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. Symbolicgpt: A generative transformer model for symbolic regression, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6005 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Hugging- face's transformers: tftte-of-the-ar natural language processing, CoRR, abs/ abs/1910.03371, 2019. URL http://arxiv org/abs/ 1910. 03771
Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. Stat: Spatial-temporal attention mechanism for video caption- ing. IEEE transactions multimedia, 22(1):229 241, 2019.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-tossequeeee functions? arXie preprint arXiv: 10077, 2019.
Wojeiech Zaremba and Ilya Sutskever. Learning execute, 2015.
Wojciech Zaremba, Karol Kurach, and Rob Fergus. Learning to discover efficient mathe- matical identities, 2014.
Lukasz Kaiser and Nya. Sutskever. Neural gpus learn algorithms, 2016.
12


Reviews:
[{"review_id": "wNR_MapawZd", "reviewer": "Reviewer_oP2C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main critique is that the paper omits closely related work - in particular GPT-f by Polu & Sutskever, which has already demonstrated gains on mathematical tasks after pretraining on natural language. I do not see that this paper adds a significant insight.\n\nThe claim stated in the title of the paper is not exactly true either. The pretrained models require a significant number of training steps in “fine-tuning” on datasets for symbolic mathematics to achieve a quality that is comparable to training them only on symbolic mathematics data.\n\nThe quality of the write-up is mixed. In particular the section called “Theory” is not ready to be published. For example:\n- second line of section “Theory”: Roman -> Romanian?\n- The next sentences read: “The learned parameter, Θ which is a matrix of neural weights, encodes the knowledge between any two sequences. Our model tries to predict the sequence of the shortest length by the principle of Occam’s razor.”\n- The entire section is one long paragraph and I could not follow what it is saying. It appears to be a mix of intuition about what Transformer models are doing and a vague description of an approach?\n\nAlso in the introduction there are passages that need to be fixed:\n- “and thus our model predict with better when the length …” -> “and thus our model’s predictions improve when the length …”\n- “Section??” -> “Section 3”\n\n", "summary_of_the_paper": "This paper improves studies if pretraining transformers on natural language improves their accuracy on symbolic mathematics tasks. The work follows the well-known paper by Lample & Charton and use the same distribution of (synthetic) data.", "summary_of_the_review": "I am not convinced of the novelty of this work and the quality of the write-up is not up to ICLR standards.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "huEMwr30meP", "reviewer": "Reviewer_2CuZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. This work provided some empirical results on whether pretraining in machine translation could help with solving symbolic integration and solving differential equations. \n\n2. The analysis on the choice of language pairs in pretraining and the generalization under distribution shift helps withe the understanding of the effect of pretraining. \n\nWeakness:\n\n1. The title \"Pretrained Language Models are Symbolic Mathematics Solvers too!\" is misleading. It suggests the pretrained language model is able to solve symbolic math problems reasonably well (without further training), but it actually can not until it is trained / fine tuned on a symbolic math dataset and the effect from pretraining is just faster training. Plus, it has already been shown in [Lu et al, 2021] that pretraining in language can help other modalities, so it is not so surprising that it can help in learning to solve symbolic math problem. \n\n2. The presentation of results is confusing. The paper claims that \"We achieve comparable accuracy on the integration task with our pretrained model while using around 1.5 orders of magnitude less number of training samples than the state-of-theart model in Lample & Charton (2019)\", but it is unclear to me which result it is referring to. Both table 1 and table 3 states \"The number of training samples used to train the models in all tasks is 1 million\" and \"the models are trained on the 1 million sample size training data\". Figure 2 is explicitly shows the training dataset size and it doesn't seem like the mBART can outperform LC with 1 order of magnitude less data. Since this is one of the main claim of this paper, it would help to make the supporting evidence more clear and highlighted. \n\n3. The result of \"LC\" in this paper seems much lower than the results presented in [Lample & Charleston, 2019], for example, Section 4.4 in their paper shows accuracy over 95% in FWD, BWD and IBP. What is the reason for the difference? \n\n4. Section 3 (\"Theory\") is unclear. Some statements like \"our model differs from the statistical machine learning theory, and uses a phenomena of self-regularization to find the optimum hypothesis Θ.\" are quite confusing. It might be helpful to use a concrete example or some equations to help explain the theory that the authors want to convey.\n", "summary_of_the_paper": "This work investigate the problem of whether pretraining on language task such as machine translation could help with solving symbolic mathematics problems. Specifically they focused on solving symbolic integration and differential equations using the dataset similar to [Lample & Charleston, 2019]. The authors argued that finetuning a pretrained transformer model could get similar or better accuracy with fewer epochs. They then investigated the effect of the choice of language pairs and how the model works under domain shifts. \n", "summary_of_the_review": "This paper provides some empirical study of whether pretraining in machine translation could help with solving symbolic integration and differential equations, and showed promising results. However, the effect that pretraining on language tasks could help other modalities has already been show in previous works so that novelty is limited. Furthermore the current draft contains several flaws: the title and some claims are not well supported; some of the writing, for example, Section 3, is confusing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aaZCAQ6IwEE", "reviewer": "Reviewer_4ER6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper demonstrates that transformer pre-trained on NLP tasks can be used on non-linguistic, symbolic, problems. Given the amount of data needed to train large transformers, and the availability of models pre-trained on language problems, this is an important subject. There is some prior work on the subject, notably for theorem proving (https://arxiv.org/abs/2009.03393), but I know of no systematic study like this one. My main concern with the paper is the way the comparisons are handled. In particular, the models compared have very different sizes, which makes the effect of pre-training difficult to measure. Also, the difference in accuracy between the models complicates the sample size and out of distribution accuracy comparisons. These flaws in the experimental protocol weaken the authors' conclusions. Correcting them would greatly reinforce the paper.\n\nHere is a series of observations and suggestions that might help improve the paper. The most important once (on experimental comparisons, are points 5-8): \n\n1- page 2 : \"trees are order invariant\", this is not true : for instance, the left and right nodes of operators '-' and '/' cannot be exchanged. A more correct statement would be that the correspondance between mathematical expressions and the objects they represent is a many to one, but the same is true for natural language. Therefore, it is unlikely that pre-training on NLP tasks can eliminate this redundancy. \n\n2- page 2 : the claim \"Thus, our method is more robust than Lample & Charton (2019) as it is invariant to the data set generation method\" seems incorrect: table 4 shows that whereas your model fares a little better than L&C when tested on a different generator than the one used at training, it is by no means invariant to the generation method (see also point 8).\n\n3- page 3 : section 3, target language is \"romanian\" (instead of roman)? \n\n4- page 4: Can you explain why the read path in figure 1 represents an expression of shortest length? The expression represented here is 3x^2 + e^x - (1-x^2), and is certainly not the shortest representation of this mathematical object: something like the tree representation of 4x^2 - 1 + exp(x) would probably be the shortest (and even 4x^2 + exp(x) if this is a solution to the integration problem, as you can set the constant to 0 to save tokens). \n\n5- page 5-6 : Table 1 and figure 2: the mBart model has 12 layers and 16 attention heads, the LC model 6 layers and 8 attention heads. Therefore it is difficult to know whether the increase in accuracy over the FWD and BWD samples are due to the pre-training or to the larger model. Could the comparisons be done with a larger LC model (or a smaller mBart), to minimize the effect of model size?\n\n6- You mention that mBart is biased toward shorter solutions. Can this bias be measured, for instance by comparing over a fixed test set the length of the solutions found by LC and mBart? Also, is it possible to measure if this bias has an impact on the accuracy of mBart? Could it be deactivated and the model tested without it? It is not clear whether this \"short solution bias\" can improve accuracy. In fact, it might even account for the bad performance over the more difficult ODE sets (by introducing a bias that complicates the model's work). \n\n7- page 6: Your results suggest that mBart learns faster than LC (this is after all the goal of pre-training). However, the sample size comparison,1M for mBart vs 40-80M for LC, is not correct. 40-80M is the size of the data sets L&C created, not the number of samples needed to train their model to a certain accuracy level. Besides, the difference in size and accuracy between mBart and LC complicate the assessment. I think a correct metric would be to compare the number of samples needed by LC to achieve the same accuracies as mBart after 100k, 500k and 1M examples (judging from figure 2, this would probably suggest that mBart need half the samples LC use). This would not change your conclusion, but result in better supported claims. \nAs before, the comparison would be reinforced if the two models had the same number of layers and attention heads. \n\n8- page 7: As it stands, table 3 does not really support your claim about the distribution shift. On the FWD trained set, mBart accuracy is 10% better than LC on the FWD test set, but only 5% on the BWD test set (and almost 0 on IBP).  For the BWD trained set, the conclusions go in the other way. Table 4, as you observe, is in favor of LC (especially if the higher accuracy of mBart is taken into account), but this may be due to the short solution bias (as you note). I believe a more precise analysis is necessary in order to back up your claim.\n\n9- the GPT-f paper by Polu and Sutskever (https://arxiv.org/abs/2009.03393) is prior work, and should be in the related works. Together with language data (webcrawl in their case) they tried pretraining from Github code, and math articles from MathOverflow and Arxiv (their WebMath dataset) and report slight improvements in final accuracy. It might be interesting to experiment with such different pre-training tasks, to see whether \"more topical\" pre-training data (i.e. computer science and maths, instead of common text) helps. \n", "summary_of_the_paper": "The authors use transformers pre-trained on machine translation tasks on the integration and differential equation datasets proposed by  Lample and Charton (Deep Learning for Symbolic Mathematics, ICLR 2020). They show that pre-training on \"pure language\" tasks help models, in terms of learning speed, accuracy, and, to a lesser extent, out of distribution generalization. Pre-training on different language pairs, on the other hand, has little impact on performance.", "summary_of_the_review": "Overall, the paper is interesting and addresses an important subject. However, the results could be made more convincing by making mBart and LC more comparable. This includes : \n- comparing models with similar number of parameters and features\n- testing the impact of the bias towards shorter expressions\n- better measurement of the number of samples needed to train in both cases\n- a better analysis of the distribution shift comparison\n\nHence my note of 6, which I would raise if these issues can be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2quVtFDUFaR", "reviewer": "Reviewer_3m2T", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper indeed seems to show you can sometimes get performance and efficiency improvements over a baseline transformer trained from scratch by using a pretrained transformer.\n\nWeaknesses:\n- This paper is unclear and poorly written. There were many sentences I didn't understand (to a very unusual extent). \n- The evaluation setup seems strange to me. If they want to investigate the effect of language pretraining (of some sort) on symbolic mathematics, why focus exclusively on translation? For a number of experiments, they also focus exclusively on English to Romanian; while they assess other languages as well later on, they find that the specific pair of languages can make a big difference, so I'm not sure how much to trust the results with just English and Romanian.\n- Presumably they could have easily used more standard pretrained language models (e.g. GPT-2 or T5) but don't do so for some reason. This would make the evaluation more compelling.\n- The paper also makes a number of claims that seem unsupported or at least strange and confusing. For example, they say \"Therefore, our mBART transformer model is lazy and during the fine-tuning method neural weights along the breadth are almost frozen and only the weights along criteria-satisfied path are updated. This method is also called stunting during its training\" I don't know what this means. There were *many* other similarly confusing sentences.\n- Their approach, which is *extremely* simple and just amounts to fine-tuning a pretrained model, also doesn't even consistently do better. For example, it seems to degrade performance on ODEs by quite a bit.", "summary_of_the_paper": "This paper builds on Lample & Charton (2019) by assessing the ability of large pre-trained language models to solve symbolic mathematics problems. They show that transformers trained on language-to-language translation tasks can improve the efficiency final accuracy of learning symbolic mathematics in some settings.", "summary_of_the_review": "This paper is poorly written and makes confusing + unsupported claims, the evaluation isn't thorough at all, and their approach doesn't even seem to help consistently very much anyway.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers presents a method for solving symbolic mathematic tasks. It first pretrains a transformer model with language translation, and then fine-tunes the pretrained model to the downstream mathematic tasks. It contains interesting points but our reviewers have serious concerns which are not fully resolved in the rebuttal. For the integration task, the proposed method achieves good results comparing with Lample & Charleston 2019 (LC) with much less training data. However, as the authors also noted (see the rebuttal), the higher accuracies in LC are achieved with more data. If the authors could also at least show how much data the proposed method needs to achieve the best result in LC, it will be very helpful for understanding the value of this work. In addition, the proposed method did not show similar improvements on the ODE task. So it is hard to see how this proposed method can be generally useful. Our reviewers also have big concerns on writing. Many sentences are really confusing.
Decision: Reject