Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper improves studies if pretraining transformers on natural language improves their accuracy on symbolic mathematics tasks. The work follows the well-known paper by Lample & Charton and use the same distribution of (synthetic) data.
main_review: My main critique is that the paper omits closely related work - in particular GPT-f by Polu & Sutskever, which has already demonstrated gains on mathematical tasks after pretraining on natural language. I do not see that this paper adds a significant insight.

The claim stated in the title of the paper is not exactly true either. The pretrained models require a significant number of training steps in “fine-tuning” on datasets for symbolic mathematics to achieve a quality that is comparable to training them only on symbolic mathematics data.

The quality of the write-up is mixed. In particular the section called “Theory” is not ready to be published. For example:
- second line of section “Theory”: Roman -> Romanian?
- The next sentences read: “The learned parameter, Θ which is a matrix of neural weights, encodes the knowledge between any two sequences. Our model tries to predict the sequence of the shortest length by the principle of Occam’s razor.”
- The entire section is one long paragraph and I could not follow what it is saying. It appears to be a mix of intuition about what Transformer models are doing and a vague description of an approach?

Also in the introduction there are passages that need to be fixed:
- “and thus our model predict with better when the length …” -> “and thus our model’s predictions improve when the length …”
- “Section??” -> “Section 3”


summary_of_the_review: I am not convinced of the novelty of this work and the quality of the write-up is not up to ICLR standards.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work investigate the problem of whether pretraining on language task such as machine translation could help with solving symbolic mathematics problems. Specifically they focused on solving symbolic integration and differential equations using the dataset similar to [Lample & Charleston, 2019]. The authors argued that finetuning a pretrained transformer model could get similar or better accuracy with fewer epochs. They then investigated the effect of the choice of language pairs and how the model works under domain shifts. 

main_review: Strength:

1. This work provided some empirical results on whether pretraining in machine translation could help with solving symbolic integration and solving differential equations. 

2. The analysis on the choice of language pairs in pretraining and the generalization under distribution shift helps withe the understanding of the effect of pretraining. 

Weakness:

1. The title "Pretrained Language Models are Symbolic Mathematics Solvers too!" is misleading. It suggests the pretrained language model is able to solve symbolic math problems reasonably well (without further training), but it actually can not until it is trained / fine tuned on a symbolic math dataset and the effect from pretraining is just faster training. Plus, it has already been shown in [Lu et al, 2021] that pretraining in language can help other modalities, so it is not so surprising that it can help in learning to solve symbolic math problem. 

2. The presentation of results is confusing. The paper claims that "We achieve comparable accuracy on the integration task with our pretrained model while using around 1.5 orders of magnitude less number of training samples than the state-of-theart model in Lample & Charton (2019)", but it is unclear to me which result it is referring to. Both table 1 and table 3 states "The number of training samples used to train the models in all tasks is 1 million" and "the models are trained on the 1 million sample size training data". Figure 2 is explicitly shows the training dataset size and it doesn't seem like the mBART can outperform LC with 1 order of magnitude less data. Since this is one of the main claim of this paper, it would help to make the supporting evidence more clear and highlighted. 

3. The result of "LC" in this paper seems much lower than the results presented in [Lample & Charleston, 2019], for example, Section 4.4 in their paper shows accuracy over 95% in FWD, BWD and IBP. What is the reason for the difference? 

4. Section 3 ("Theory") is unclear. Some statements like "our model differs from the statistical machine learning theory, and uses a phenomena of self-regularization to find the optimum hypothesis Θ." are quite confusing. It might be helpful to use a concrete example or some equations to help explain the theory that the authors want to convey.

summary_of_the_review: This paper provides some empirical study of whether pretraining in machine translation could help with solving symbolic integration and differential equations, and showed promising results. However, the effect that pretraining on language tasks could help other modalities has already been show in previous works so that novelty is limited. Furthermore the current draft contains several flaws: the title and some claims are not well supported; some of the writing, for example, Section 3, is confusing.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper builds on Lample & Charton (2019) by assessing the ability of large pre-trained language models to solve symbolic mathematics problems. They show that transformers trained on language-to-language translation tasks can improve the efficiency final accuracy of learning symbolic mathematics in some settings.
main_review: Strengths:
- This paper indeed seems to show you can sometimes get performance and efficiency improvements over a baseline transformer trained from scratch by using a pretrained transformer.

Weaknesses:
- This paper is unclear and poorly written. There were many sentences I didn't understand (to a very unusual extent). 
- The evaluation setup seems strange to me. If they want to investigate the effect of language pretraining (of some sort) on symbolic mathematics, why focus exclusively on translation? For a number of experiments, they also focus exclusively on English to Romanian; while they assess other languages as well later on, they find that the specific pair of languages can make a big difference, so I'm not sure how much to trust the results with just English and Romanian.
- Presumably they could have easily used more standard pretrained language models (e.g. GPT-2 or T5) but don't do so for some reason. This would make the evaluation more compelling.
- The paper also makes a number of claims that seem unsupported or at least strange and confusing. For example, they say "Therefore, our mBART transformer model is lazy and during the fine-tuning method neural weights along the breadth are almost frozen and only the weights along criteria-satisfied path are updated. This method is also called stunting during its training" I don't know what this means. There were *many* other similarly confusing sentences.
- Their approach, which is *extremely* simple and just amounts to fine-tuning a pretrained model, also doesn't even consistently do better. For example, it seems to degrade performance on ODEs by quite a bit.
summary_of_the_review: This paper is poorly written and makes confusing + unsupported claims, the evaluation isn't thorough at all, and their approach doesn't even seem to help consistently very much anyway.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors use transformers pre-trained on machine translation tasks on the integration and differential equation datasets proposed by  Lample and Charton (Deep Learning for Symbolic Mathematics, ICLR 2020). They show that pre-training on "pure language" tasks help models, in terms of learning speed, accuracy, and, to a lesser extent, out of distribution generalization. Pre-training on different language pairs, on the other hand, has little impact on performance.
main_review: This paper demonstrates that transformer pre-trained on NLP tasks can be used on non-linguistic, symbolic, problems. Given the amount of data needed to train large transformers, and the availability of models pre-trained on language problems, this is an important subject. There is some prior work on the subject, notably for theorem proving (https://arxiv.org/abs/2009.03393), but I know of no systematic study like this one. My main concern with the paper is the way the comparisons are handled. In particular, the models compared have very different sizes, which makes the effect of pre-training difficult to measure. Also, the difference in accuracy between the models complicates the sample size and out of distribution accuracy comparisons. These flaws in the experimental protocol weaken the authors' conclusions. Correcting them would greatly reinforce the paper.

Here is a series of observations and suggestions that might help improve the paper. The most important once (on experimental comparisons, are points 5-8): 

1- page 2 : "trees are order invariant", this is not true : for instance, the left and right nodes of operators '-' and '/' cannot be exchanged. A more correct statement would be that the correspondance between mathematical expressions and the objects they represent is a many to one, but the same is true for natural language. Therefore, it is unlikely that pre-training on NLP tasks can eliminate this redundancy. 

2- page 2 : the claim "Thus, our method is more robust than Lample & Charton (2019) as it is invariant to the data set generation method" seems incorrect: table 4 shows that whereas your model fares a little better than L&C when tested on a different generator than the one used at training, it is by no means invariant to the generation method (see also point 8).

3- page 3 : section 3, target language is "romanian" (instead of roman)? 

4- page 4: Can you explain why the read path in figure 1 represents an expression of shortest length? The expression represented here is 3x^2 + e^x - (1-x^2), and is certainly not the shortest representation of this mathematical object: something like the tree representation of 4x^2 - 1 + exp(x) would probably be the shortest (and even 4x^2 + exp(x) if this is a solution to the integration problem, as you can set the constant to 0 to save tokens). 

5- page 5-6 : Table 1 and figure 2: the mBart model has 12 layers and 16 attention heads, the LC model 6 layers and 8 attention heads. Therefore it is difficult to know whether the increase in accuracy over the FWD and BWD samples are due to the pre-training or to the larger model. Could the comparisons be done with a larger LC model (or a smaller mBart), to minimize the effect of model size?

6- You mention that mBart is biased toward shorter solutions. Can this bias be measured, for instance by comparing over a fixed test set the length of the solutions found by LC and mBart? Also, is it possible to measure if this bias has an impact on the accuracy of mBart? Could it be deactivated and the model tested without it? It is not clear whether this "short solution bias" can improve accuracy. In fact, it might even account for the bad performance over the more difficult ODE sets (by introducing a bias that complicates the model's work). 

7- page 6: Your results suggest that mBart learns faster than LC (this is after all the goal of pre-training). However, the sample size comparison,1M for mBart vs 40-80M for LC, is not correct. 40-80M is the size of the data sets L&C created, not the number of samples needed to train their model to a certain accuracy level. Besides, the difference in size and accuracy between mBart and LC complicate the assessment. I think a correct metric would be to compare the number of samples needed by LC to achieve the same accuracies as mBart after 100k, 500k and 1M examples (judging from figure 2, this would probably suggest that mBart need half the samples LC use). This would not change your conclusion, but result in better supported claims. 
As before, the comparison would be reinforced if the two models had the same number of layers and attention heads. 

8- page 7: As it stands, table 3 does not really support your claim about the distribution shift. On the FWD trained set, mBart accuracy is 10% better than LC on the FWD test set, but only 5% on the BWD test set (and almost 0 on IBP).  For the BWD trained set, the conclusions go in the other way. Table 4, as you observe, is in favor of LC (especially if the higher accuracy of mBart is taken into account), but this may be due to the short solution bias (as you note). I believe a more precise analysis is necessary in order to back up your claim.

9- the GPT-f paper by Polu and Sutskever (https://arxiv.org/abs/2009.03393) is prior work, and should be in the related works. Together with language data (webcrawl in their case) they tried pretraining from Github code, and math articles from MathOverflow and Arxiv (their WebMath dataset) and report slight improvements in final accuracy. It might be interesting to experiment with such different pre-training tasks, to see whether "more topical" pre-training data (i.e. computer science and maths, instead of common text) helps. 

summary_of_the_review: Overall, the paper is interesting and addresses an important subject. However, the results could be made more convincing by making mBart and LC more comparable. This includes : 
- comparing models with similar number of parameters and features
- testing the impact of the bias towards shorter expressions
- better measurement of the number of samples needed to train in both cases
- a better analysis of the distribution shift comparison

Hence my note of 6, which I would raise if these issues can be addressed.
review_title: null

Meta-review: This papers presents a method for solving symbolic mathematic tasks. It first pretrains a transformer model with language translation, and then fine-tunes the pretrained model to the downstream mathematic tasks. It contains interesting points but our reviewers have serious concerns which are not fully resolved in the rebuttal. For the integration task, the proposed method achieves good results comparing with Lample & Charleston 2019 (LC) with much less training data. However, as the authors also noted (see the rebuttal), the higher accuracies in LC are achieved with more data. If the authors could also at least show how much data the proposed method needs to achieve the best result in LC, it will be very helpful for understanding the value of this work. In addition, the proposed method did not show similar improvements on the ODE task. So it is hard to see how this proposed method can be generally useful. Our reviewers also have big concerns on writing. Many sentences are really confusing.
Decision: Reject