Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: In this paper, the authors propose a method for generating two types of symbolic mathematics problems, integration and differential equations, and their solutions. The purpose of the method is to generate datasets for training transformer neural networks that solve integration and differential-equation problems. The authors note that while solving these problems is very difficult, generating solutions first and corresponding problems next automatically is feasible, and their method realizes this observation. The authors report that transformer networks trained on the synthetically generated solution-problem pairs outperform existing symbolic solvers for integration and differential equation. 

Here are the reasons that I like the paper. The observation that solving a symbolic mathematics problem is often a pattern matching process is interesting. It is surprising to know that a transformer network designed to translated the generating problem-solution pairs backward (from problem to solution) works better than the solvers in Mathematica and Matlab. Also, I like nice cute tricks used in the authors' method for generating solution-problem pairs, such as the syntactic condition on a possible position of some constant. The paper is overall clearly written.

I presume that when the authors compare their learned solvers with Mathematica and Matlab, they used a dataset generated by their method. I feel that this comparison is somewhat unfair, although it still impresses me that even for this dataset, the authors' solvers beat Mathematica and Matlab. I suggest to try at least one more experiment on a dataset not generated by the authors' method (integration and differential equation problems from math textbooks or other sources) if possible.

* p3: Why is it important to have a generator that produces the four expression trees in p3 with equal or almost equal probabilities? Do you have any semi-formal or informal justification that the distribution of such a generator better matches the kind of expressions arising in the real world?

* p4: f(x)/x)) ===> f(x)/x)

* "If this equation can be solved in c1", p5: How realistic is this assumption?

* p5: 1/2 e^x(...) ===> 0 = 1/2 e^x(...)

* p5: If you have a thought or an observation on the impact of each of the data-cleaning steps in Section 3.4, I suggest you to share this in the paper.

* p6: Why did you remove expressions with more than 512 tokens?

* p6: compare to ===> compared to

* p7: Would you put the reminder of the size of the training set in Section 4.4? It only mentions that of the test set currently.

* p8: 1-(4x^2 ===> (1-(4x^2
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The authors use a Transformer neural network, originally architected for the purpose of language translation, to solve nontrivial mathematical equations, specifically integrals, first-order differential equations, and second-order differential equations. They also developed rigorous methods for sampling from a large space of relevant equations, which is critical for assembling the type of dataset needed for training such a data-intensive model.

Both the philosophical question posed by the paper (i.e. can neural networks designed for natural language sequence-to-sequence mappings be meaningfully applied to symbolic mathematics) and the resulting answer (i.e. yes, and such a neural network outperforms SOTA commercially-available systems) are interested in their own right, and together make a strong case for paper acceptance.

Details appearing in the OpenReview comments which should be explicitly specified in the paper before publication:
1) How large was the generated training set (40M), and how does this compare to the space of all equations under consideration (1e34).
2) The authors employ beam search in a non-standard manner, where they check for appearance of the equation solution among all of the generated candidates, rather than selecting the top-1. The fact that the reported accuracy with width-10 and width-50 beam searches are in effect measuring top-10 and top-50 accuracy should be clearly stated.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: It is rather interesting for a humble academic to review this paper. It already has a discussion, which I find very valuable, and many tweets and social media exposure and endorsements. It is onerous to review in this setting.

The paper makes a valuable contribution. The adversarial discussions in this website and the unhelpful hype can in this case be addressed to some extent by the authors. I will start with discussing this. Clearly, the title is too broad. This is not deep learning for symbolic mathematics. In no way does this paper address the essence of what is understood by "symbolic mathematics". What the authors address is mapping sequences of discrete quantities to other sequences of discrete quantities. The sequences in this paper correspond to function-integral i/o sequences, and 1st/2nd ODEs-function i/o sequences. I will leave it to the authors to come up with a more informative title, but something like deep learning or transformers for symbolic (1d) integration and simple ODEs with be far more accurate.

To hammer this point, note that Section 3 discusses removing "invalid" expressions: log(0) or sqrt(-2). However, it is the manipulation of infinity and imaginary numbers that could be considered to be one of the greatest achievements of symbolic mathematics over the last couple of hundred years. It is reasonable to expect neural nets to do this one day, because humans can, but this should come with results. It's too early to make the claim in the paper title.

Sentences such as "This suggest (sic) that some deeper understanding of mathematics has been achieved by the model." and "These results are surprising given the incapacity of neural models to perform simpler tasks ..." are speculative, potentially inaccurate and likely to increase hype. This hype is not needed.

Hype and over-claiming aside, I did enjoy reading this paper. The public commenters have already asked important questions about methodology and related work on neural programming that the authors have addressed in comments. I look forward to these being incorporated in the revised pdf.

A big part of the paper is about generating the datasets, and I therefore sympathise with the comment about requesting either a dataset release or the generating code. I see no obvious ethical concerns in this case, and the authors have already kindly offered to do this. This is a commendable and important service to our community and for this alone I would be inclined to vote for acceptance at ICLR.

The paper is clear and well written. However (i) it would be good to show several examples of input and output sequences (as done already in this website) and (ii) the Experiments section needs work. I'll expand on this next.

The seq2seq transformer with 8 heads, 6 layers and dimensionality 512 is a sensible choice. The authors should however explain why they expect this architecture to be able to map the sequences they adopt. That is, it is well known that a deep neural network is just a skeleton for an algorithm. By estimating the parameters, we are coming up with (fitting) the algorithm for the given datasets. What is the resulting algorithm? Why are 6 layers enough? Here some visualization would be helpful. See for example https://arxiv.org/pdf/1904.02679.pdf and https://arxiv.org/pdf/1906.04341.pdf For greater understanding of the problem, it may be useful to also try sparse transformers eg https://arxiv.org/abs/1805.08241

Beam search is a crucial component of the current solution. However, the authors simply cite Koehn 2004 for this. First, that work used language models to compute probabilities for beam search. I assume no language models are used in this case. What I'm getting to is that there are not enough details about the beam search in this paper. The authors should include pseudocode for the beam search and give a few examples. The paper (even better thesis) of Koehn is a good template for what should be included. This is important and should be explained. 

For Mathematica, it would be useful to state it does other things and has not been optimized for the two tasks addressed in this paper only. It would also be useful, now that you have more time, to run it for a week or two and get answers not only for 30s but also for 60s. How often does it take longer than 30s? How do you score it then?

Please do include train and test curves. This would be helpful too. I will of course consider revising my score once the paper is updated. 

Thanks for constructing this dataset and writing this paper. It is very interesting and promising.





review_title: Official Blind Review #1

Meta-review: The paper presents a deep learning approach for tasks such as symbolic integration and solving differential equations. 

The reviewers were positive and the paper has had extensive discussion, which we hope has been positive for the authors. 

We look forward to seeing the engagement with this work at the conference.
Decision: Accept (Spotlight)