Deep Learning For Symbolic Mathematics
Authors: Guillaume Lample
François Charton
Venue:  (2020)
Keywords: symbolic
math
deep learning
transformers

Abstract: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing these mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

Published as a conference paper at ICLR 2020
DEEP LEARNING FOR SYMBOLIC MATHEMATICS
Guillaume Lample* Facebook Al Research glample @ fb.com com
François Charton" Facebook Al Research
fchartong fb. com
ABSTRACT
Neural networks have reputation for being better at solving statistical approxi- mate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequens models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.
INTRODUCTION
longstanding tradition in machine learning opposes alc-bbaed inference to statistical learning (Rumelhart al., 1986), and neural networks clearly stand the statistical side. They have proven to be extremely effective in statistical pattern recognition and now achieve state-of-the-an performance on wide range of problems in computer vision, speech recognition, natural language processing (NLP), etc. However, the success of neural networks in symbolic computation is still extremely limited: combining symbolic reasoning with continuous representations now one of the challenges of machine learning.
Only few studies investigated the capacity of neural network to deal with mathematical objects, and apart from a small number of exceptions (Zaremba et al., 2014; Loos et al., 2017; Allamanis et al., 2017; Arabshahi al., 2018b), the majority of these works focus on arithmetic tasks like integer addition and multiplication (Zaremba & Sutskever, 2014; Kaiser & Sutskever, 2015; Trask et al., 2018). On these tasks, neural approaches tend to perform poorly, and require the introduction of components biased towards the task at hand (Kaiser & Sutskever, 2015; Trask et al., 2018).
In this paper, we consider mathematics, and particularly symbolic calculations, as a target for NLP models. More precisely, we use eequence-to-sqquee models (seq2seq) on two problems of symbolic mathematics: function integration and ordinary differential equations (ODEs). Both are difficult, for trained humans and computer software. For integration, humans are taught set of rules (integration by parts, change of variable, etc.), that are not guaranteed to succeed, and Computer Algebra Systems use complex algorithms (Geddes et al., 1992) that explore large number of specific cases. For instance, the complete description of the Risch algorithm (Risch, 1970) for function integration is more than 100 pages long.
Yet, function integration is actually an example where pattern recognition should be useful: detecting that an expression is of the form yy/ yy'(y² (y² + 1)-1/2 suggests that its primitive will contain v + 1. Detecting this pattern may be easy for small expressions y, but becomes more difficult as the number of operators in increases. However, to the best of our knowledge, no study has investigated the ability of neural networks detect patterns in mathematical expressions.
We first propose representation of mathematical expressions and problems that can be used by seq2seq models, and discuss the size and structure of the resulting problem space. Then, we show how to generate datasets for supervised learning of integration and first and second order differential equations. Finally, we apply seq2seq models to these datasets, and show that they achieve better performance than state-of-the-art computer algebra programs, namely Matlab and Mathematica.
Equal contribution.

---

Published as a conference paper at ICLR 2020
2
MATHEMATICS AS A NATURAL LANGUAGE
2.1 EXPRESSIONS AS TREES
Mathematical expressions can be represented as trees, with operators and functions as internal nodes, operands as children, and numbers, constants and variables as leaves. The following trees represent expressions 2 + 3 x (5 + 2), 3x + cos (2x) L, and 0² 0
22 8
pow
COS
pow
2
Trees disambiguate the order of operations, take care of precedence and associativity and eliminate the need for parentheses. Up the addition of meaningless symbols like spaces, punctuation or redundant parentheses, different expressions result in different trees. With a few assumptions, discussed in Section A of the appendix, there is one-to-one mapping between expressions and trees.
We consider expressions as sequences of mathematical symbols. 2 + 3 and 3 + 2 are different expressions, as are v4x and 2x, and they will be represented by different trees. Most expressions represent meaningful mathematical objects. x 0, v (o) are also legitimate expressions, even though they do not necessarily make mathematical sense.
Since there one-to-one correspondence between trees and expressions, equality between expres- sions will be reflected over their associated trees, as an equivalence: : since 2+3 = 5 = 12-7 1x5, the four trees corresponding these expressions are equivalent.
Many problems of formal mathematics can be reframed as operations over expressions, or trees, For instance, expression simplification amounts finding shorter equivalent representation of tree. In this paper, we consider two problems: symbolic integration and differential equations. Both boil down to transforming an expression into another, e.g. mapping the tree of an equation to the tree of its solution. We regard this as a particular instance of machine translation.
2.2 TREES AS SEQUENCES
Machine translation systems typically operate on sequences (Sutskever al., 2014: Bahdanau et al., 2015). Alternative approaches have been proposed to generate trees, such as Tree-LSTMM (Tai et al., 2015) or Recurrent Neural Network Grammars (RNNG) (Dyer et al., 2016; Eriguchi al., 2017). However, tree-to-tred tree models are more involved and much slower than their seq2seq counterparts, both at training and inference. For the sake simplicity, we use seq2seq models, which were shown to be effective at generating trees, e.g. in the context of constituency parsing (Vinyals et al., 2015), where the task is to predict a syntactic parse tree of input sentences.
Using seq2seq models to generate trees requires to map trees to sequences. To this effect, prefix notation (also known as normal Polish notation), writing each node before its children, listed from left right. For instance, the arithmetic expression 2 3 (5 is represented as the sequence
[+ 2 3 + 5 2]. In contrast to the more common infix notation + 3 * (5 + 2), prefix sequences need no parentheses and are therefore shorter. Inside sequences, operators, functions or variables represented by specific tokens, and integers by sequences of digits preceded by a sign. As in the case between expressions and trees, there exists a one-to-one mapping between trees and prefix sequences.
2.3 GENERATING RANDOM EXPRESSIONS
To create training data, we need to generate sets of random mathematical expressions. However, sampling uniformly expressions with internal nodes not simple task. Naive algorithms (such as
2

---

Published as a conference paper at ICLR 2020
recursive methods techniques using fixed probabilities for nodes to be leaves, unary, binary) tend to favour deep trees over broad trees, or left-leaning over right leaning trees. Here are examples of different trees that we want to generate with the same probability.
cos
X
X
pow
5
T
3
sqrt
T
8111
sin
pow
In Section C the appendix, we present an algorithm to generate random trees and expressions, where the four expression trees above are all generated with the same probability.
2.4 COUNTING EXPRESSIONS
We now investigate the number of possible expressions. Expressions are created from finite set of variables (i.e. literals), constants, integers, and list of operators that can be simple functions (c.g. cos or exp) or more involved operators (e.g. differentiation or integration). More precisely, we define our problem space as:
trees with up internal nodes
set of P1 unary operators (e.g. cos, sin, exp, log)
a set of P2 binary operators (e.g. +, x, pow)
a set of leaf values containing variables (e.g. 4, y. 2), constants (e.g. π), integers (e.g.
{-10,..., 10))
If pi = 0, expressions are represented by binary trees. The number binary trees with n internal nodes is given by the n-th Catalan numbers C, (Sloane, 1996). A binary tree with internal nodes has exactly n + leaves. Each node and leaf can take respectively P2 and different values. As result, the number of expressions with binary operators can be expressed by:
4n Th TH
1
2n
En C.PP 1+
L+t1
with C, n
n
pt > O, expressions are unary-binary trees, and the number of trees with internal nodes the n-th large Schroeder number S, (Sloane, 1996). It can be computed by recurrence using the following equation:
(n + 1)S, = 3(2n 1)5,-1 - (n 2)S..
(1)
Finally, the number E, of expressions with n internal nodes, P1 unary operator, P2 binary operators and L possible leaves is recursively computed
(n E, = (p1 + 22n - 1)En-1 - pi(n En-2
= 1, Equation 2 boils down to Equation If P2 L 1, = 0, we have 1),,
= 2(2n E,- which is the recurrence relation satisfied by Catalan numbers. The derivations and properties of all these formulas are provided in Section of the appendix.
In Figure represent the number of binary trees (C₂) and mmary-binay trees (S,) for different numbers of internal nodes. We also represent the number of possible expressions (E,) for different sets of operators and leaves.
3

---

Published as a conference paper at ICLR 2020
10c2
Number of expressiom
10a1
L=11, p1=15, p2=4 (unary-binary expressions)
L-11, pl=0,, p2=4 (binary expressions)
1040
L-11, p1-15, p2-1
L-11, pl-0, p2-1
1029
L-5, p1-0, p2-1
pl=1, (unary-binary trees)
10
pl=0, p2=1 (binary trees)
107
12 16 20
Internal nodes
Figure Number of trees and expressions for different numbers of operators and leaves. pl and pz correspond the number of unary and binary operators respectively. and to the number of possible leaves. correspond to the number of binary and unary-binary trees (enumerated by Catalan top two curves represent the associated number expressions.
The bottom and Schroeder numbers respectively). observe that adding leaves and binary operators significantly increases the size of the problem space.
GENERATING DATASETS
Having defined a syntax for mathematical problems and techniques to randomly generate expressions, we are now in a position to build the datasets models will use. In the rest of the paper, we focus on two problems of symbolic mathematics: function integration and solving ordinary differential equations (ODE) of the first and second order.
To train our networks, we need datasets of problems and solutions. Idcally, we want to generate representative samples of the problem space, i.e. randomly generate functions to be integrated and differential equations to be solved. Unfortunately, solutions of random problems sometimes do not exist (e.g. the integrals of f(z) exp(x²) ) (a) log(log(æ:)) cannot be expressed with usual functions), or cannot be easily derived. In this section, we propose techniques to generate large training sets for integration and first and second order differential equations.
3.1 INTEGRATION
We propose three approaches to generate functions with their associated integrals.
Forward generation (FWD). A straightforward approach is to generate random functions with up operators (using methods from Section 2) and calculate their integrals with computer algebra system. Functions that the system cannot integrate are discarded. This generates representative sample of the subset of the problem space that can be successfully solved by an external symbolic mathematical framework.
Backward generation (BWD). An issue with the forward approach is that the dataset only contains functions that symbolic frameworks solve (they sometimes fail to compute the integral of integrable functions). Also, integrating large expressions is time expensive, which makes the overall method particularly slow. Instead, the backward approach generates random function f, computes its derivative f', and adds the pair (J', to the training set. Unlike integration, differentiation always possible and extremely fast even for very large expressions. As opposed to the forward approach, this method does not depend on an external symbolic integration system.
Backward generation with integration by parts (IBP). An issue with the backward approach is that it is very unlikely to generate the integral of simple functions like f(x) x² sin(x). Its integral, F(x) = =xd cos(x)) + 3x sin(x) + 6r (z) - sin(z), function with 15 operators, has a very low probability of being generated randomly. Besides, the backward approach tends to generate examples where the integral (the solution) is shorter than the derivative (the problem), while forward generation favors the opposite (see Figure in Section E in the Appendix). To address this issue,
4

---

Published as a conference paper at ICLR 2020
leverage integration by parts: given two randomly generated functions F and G, compute their respective derivatives f and g. If fG already belongs to the training set, we know its integral, and we can compute the integral of Fg as:
/ Fg- f
Similarly, if Fg is in the training set, integral new function, we add it to the training set. If none of fG or Fg are the training set, simply generate
infer the integral of fG. Whenever discover the
functions F and G. With this approach, functions like x (i) without resorting to an external symbolic integration system.
generate the integrals of
Comparing different generation methods. Table in Section 4.1 summarizes the differences between the three generation methods. The FWD method tends to generate short problems with long solutions (that computer algebras can solve). The BWD approach, on the other hand, generates long problems with short solutions. IBP generates datasets comparable to FWD (short problems and long solutions), without an external computer algebra system. A mixture of BWD and IBP generated data should therefore provide a better representation of problem space, without resorting to external tools. Examples of functions/ integrals for the three approaches are given in Table 9 of the Appendix.
3.2 FIRST ORDER DIFFERENTIAL EQUATION (ODE) 1)
We now present method to generate first order differential equations with their solutions. We start from a bivariate function F(x, y) such that the equation F(r,, = e (where is a constant) can be analytically solved in y. In other words, there exists a bivariate function f that satisfies V(x, c), Fz, f(x, = c. By differentiation with respect to ø, we have that Vx, C:
(x, + fc(x)) dr
0F fc(x))
Oy
where fc x f(x..). As result, for any constant fe is solution of the first order differential equation:
(F(r.y)
OF +
Ox
Oy
With this approach, we can use the method described in Section of the appendix to generate arbitrary functions F(x, y) analytically solvable in y, and create dataset of differential equations with their solutions.
Instead of generating random function F, we can generate solution f(x, c), and determine differ- ential equation that it satisfies. If fxz e) is solvable in c, we compute F such that Fzz, f(z, c)) c. Using the above approach, we show that for any constant c, x + fax c) a solution of differential Equation 3. Finally, the resulting differential equation is factorized, and we remove all positive factors from the equation.
necessary condition for this approach to work is that the generated functions [ax, c) can be solved in c. For instance, the function far c) = c x log(æ + c) cannot be analytically solved in i.c. the function F that satisfies F (ar, (xx c)) = e cannot be written with usual functions. Since all the invertible, simple condition ensure the solvability in is to guarantee that only appears once in the leaves of the tree representation of f(x,, c). A straightforward way generate suitable [(x, e) sample random function (xx) by the methods described in Section C of the appendix, and to replace one of the leaves in its tree representation by Below is an example of the whole process:
operators and functions
Generate a random function
f(x) = x log(c / x)
c= re [(m] = F(x, fx))
Solve in c
[(a] [1 + f((x) f(x)
Differentiate in r
=0
x
Simplify
xy - -
5

---

Published as a conference paper at ICLR 2020
3.3 SECOND ORDER DIFFERENTIAL EQUATION (ODE 2)
Our method for generating first order equations can be extended to the second order, by considering functions of three variables f(r, c1, 02) that can be solved in c2. As before, we derive function of three variables F such that F (zr, order differential equation:
C2- Differentiation with respect yields first.
(F(x, + (e (x)
OF (.)
0
dr
dy
Segeo(())
where Ser.a =I Mæ, cI, 02). If this equation can be solved in CI, we can infer another three- variable function G satisfying Vx. Gzz, Ser,e (ar), oo (1)) cr. Differentiating with respect a second time yields the following equation:
aG(x, 2) dr
(G(,,y, + (x))
OG(z, v.
c (x)
0
Oy
8z
y 10 (x) (m)
Therefore, for any constants and cz. la is solution of the second order differential equation:
@G(x,y,y) + ,.,(z,,,'') 11" Oy
Ox
Oz
Using this approach, we create pairs of second order differential equations and solutions, provided we can generate f(x, C1, 02) is solvable c2, and that the corresponding first order differential equation is solvable in c1. To ensure the solvability in C2. we can use the same approach as for first order differential equation, e.g. we create fares so that C2 has exactly one leaf in its tree representation. For C₁, employ simple approach where we simply skip the current equation if we cannot solve in c1. Although naive, we found that the differentiation equation can be solved in e about 50% the time. As an example:
Generate a random function
f(x) = cee² + o2e
Solve in C2
2 = f(x)e" - q F(z,
Differentiate in
(f'(x) - -2cee =0
Solve in C1
c e (f((x) + f(x)) G(z, f(),
Differentiate in Simplify
(ff(x) f(x))
y" - y =0
3.4 DATASET CLEANING
Equation simplification In practice, we simplify generated expressions to reduce the number of unique possible equations in the training set, and to reduce the length of sequences. Also, we do not want to train our model to predict x: + - 1 1 + + when it can simply predict + 5. As a result, sequences [+ 2 + x 3] and [+ + x]] will both be simplified to [+ x 5] as they both represent the expression + 5. Similarly, the expression log(e*+3) will be simplified to + 3, the expression cos2 (a)) + sin" (wr) will be simplified to 1, etc. On the other hand, V(x 1)2 will not be
simplified to x as we do not make any assumption on the sign of x -
the case of first order differential equations, we modify generated expressions by equivalent expressions up to change of variable. For instance, x + tan(3) cr + will be simplified to cr + 1, as a particular choice of the constant c makes these two expressions identical. Similarly, log(x?) clog(x) becomes llg(r)).
Coefficients simplification

---

Published as a conference paper at ICLR 2020
We apply similar technique for second order differential equations, although simplification is sometimes bit more involved because there are two constants e1 and C2. For instance, C1 C22/5 5 22 + 1 is simplified c₁æ + C2, while c can be expressed with o2e etc.
We also perform transformations that are not strictly equivalent, as long as they hold under specific assumptions. For instance, simplify (an(y)2a) cosh(c: + 1) + + tan(C22)). although the constant term can be negative in the second expression, but the first one. Similarly e"e e log(n2) is transformed to cze"
Invalid expressions Finally, we also remove invalid expressions from our dataset. For instance, expressions like (0)(0) or v-2. To detect them, compute in the expression tree the values of subtrees that do not depend on a. If a subtree does not evaluate to a finite real number (e.g. -o, +oo or complex number), we discard the expression.
EXPERIMENTS
4.1 DATASET
For all considered tasks, generate datasets using the method presented in Section 3, with:
expressions with up 15 internal nodes
L 11 leaf values in {z} U (-
{0}
P2 4 binary operators: +, /
P1 15 unary operators: exp, log, sqrt, sin, cos, tan, sin"l, cos"!, tan"l, sinh, cosh, tanh,
sinh" 1 cosh 1 tanh
Statistics about datasets are presented in Table 1. As discussed in Section 3.1, we observe that the backward approach generates derivatives (i.e. inputs) significantly longer than the forward generator. We discuss this in more detail in Section of the appendix.
Forward
Backward
Integration by parts
ODE
ODE
Training set size
20M
40M
20M
40M
40M
Input length
18.916.9
70.2
17.5
123.6.115.77
149.1
Output length
49.6.148.3
28.3
26.41:1..
23.0
24.3
Length ratio
2.7
0.4
2.0
0.4
0.1
Input max length
69
450
226
508
508
Output max length 508
75
206
474
335
Table Training set sizes and length of expressions (in tokens) for different datasets. FWD and IBF tend to generate examples with outputs much longer than the inputs, while the BWD approach generates shorter outputs. Like in the 3ND case, ODE generators tend produce solutions much shorter than their equations.
4.2
MODEL
For all our experiments, we train seq2seq model to predict the solutions of given problems, i.e. to predict primitive given a function, or predict solution given a differential equation. We use transformer model (Vaswani et al., 2017) with 8 attention heads, 6 layers, and dimensionality of 512. our experiences, using larger models did not improve the performance. We train our models with the Adam optimizer (Kingma & Ba, 2014), with learning rate of 10 4, We remove expressions with more than 512 tokens, and train our model with 256 cquations per batch.
At inference, expressions generated by beam search (Koehn, 2004; Sutskever et al., 2014), with early stopping. We normalize the log-liked likelihood scores of hypotheses in the beam by their sequence length. We report results with beam widths of I (i.e. greedy decoding). 10 and 50.
During decoding, nothing prevents the model from generating an invalid prefix expression, e.g. [+ 2 3 3. To address this issue, Dyer al. (2016) constraints during decoding, ensure
7

---

Published as a conference paper at ICLR 2020
that generated sequences can always be converted to valid expression trees. In our case, we found that model generations are almost always valid and we do not use any constraint. When an invalid expression is generated, we simply consider it as incorrect solution and ignore it.
4.3 EVALUATION
At the end of each epoch, we evaluate the ability of the model to predict the solutions of given equations. In machine translation, hypotheses given by the model are compared references written by human translators, typically with metrics like the BLEU score (Papineni et al., 2002) that measure the overlap between hypotheses and references. Evaluating the quality of translations is a very difficult problem, and many studies showed that a better BLEU score does not necessarily correlate with better performance according to human evaluation. Here, however, we can easily verify the correctness of our model by simply comparing generated expressions to their reference solutions.
For instance, for the given differential equation zy y with a reference solution æ log(c/ /x) (where is a constant), our model may generate x log(c) - log(z). We can check that these two solutions are equal, although they are written differently, using a symbolic framework like SymPy (Meurer et al., 2017).
However, our model may also generate re - x log(z) which is also a valid solution, that is actually equivalent to the previous differential equation by the model hypothesis. rry y + x = 0, we conclude that the hypothesis is valid solution. In the case of integral computation, can simply differentiate the model hypothesis, and compare it with the function to integrate. For the three problems, we measure the accuracy of our model on equations from the test set.
for different choice of constant c. In that case, replace y in the
Since we can easily verify the correctness of generated expressions, we consider all hypotheses the beam, and not only the one with the highest score. We verify the correctness of each hypothesis, and consider that the model successfully solved the input equation if one of them is correct. As result, results with "Beam size 10" indicate that at least one of the 10 hypotheses in the beam was correct.
4.4 RESULTS
Table 2 reports the accuracy of our model for function integration and differential equations. For integration, the model achieves close to 100% performance on a held-out test set, even with greedy decoding (beam size 1). This performance is consistent over the three integration datasets (FWD, BWD, and IBP). Greedy decoding (beam size 1) does not work as well for differential equations. In particular, observe improvement in accuracy of almost 40% when using large beam size of 50 for second order differential equations. Unlike in machine translation, where increasing the beam size does not necessarily increase the performance (Ot et al., 2018), always observe significant improvements with wider beams. Typically, using beam size of 50 provides an improvement of 8% accuracy compared a beam size of 10. This makes sense, increasing the beam size will provide more hypotheses, although a wider beam may displace a valid hypothesis to consider invalid ones with better log- probaabilities.
Integration (FWD)
Integration (BND)
Integration (TRP)
ODE (order 1)
ODE (order 2)
Beam size Beam size 10
93.6
98 4
96.8
77.6
43.0
95.6
99 4
99.2
90.5
73.0
Beam size 50
96.2
99.7
99.5
94.0
81.2
Table 2: Accuracy of our models on integration and differential equation solving. Results reported on held out set of 5000 equations. For differential equations, using beam scarch decoding significantly improves the accuracy the model.
4.5 COMPARISON WITH MATHEMATICAL FRAMEWORKS
We compare our model with three popular mathematical frameworks: Mathematica (Wolfram- Research, 2019), Maple and Matlab (Math 2019)'. Prefix sequences in our test are
All experiments were run with Mathematica 12.0.0.0, Maple 2019 and Matlab R2019a.
8

---

Published as a conference paper at ICLR 2020
converted back to their infix representations, and given input to the computer algebra. For specific input, the computer algebra either returns a solution, provides no solution (or a solution including integrals special functions), or, in the case of Mathematica, times out after a preset delay. When Mathematica times out, we conclude that it not able to compute solution (although it might have found solution given construction, the FWD data only consists of integrals generated by computer algebra systems, which makes comparison uninteresting.
time). For integration,
evaluate the BWD test set. By
In Table 3, we present accuracy for our model with different beam sizes, and for Mathematica with a timeout delay of 30 seconds. Table 8 the appendix provides detailed results for different values of timeout, and explains our choice of 30 seconds. In particular, we find that with 30 seconds, only 20% of failures are due to timeouts, and only 10% when the timeout set to minutes. Even with timeout limits, evaluation would take too long on our 5000 test equations, so we only evaluate on smaller test subset of 500 equations, on which we also re-evaluate our model.
Integration (BND)
ODE (order
ODE (order 2)
Mathematica (30s)
84.0
77.2
61.6
Matlab
65.2
Maple
67.4
Beam size
98.4
81.2
40.8
Beam size 10
99.6
94.0
73.2
Beam size 50
99.6
97.0
81.0
Table 3: Comparison our model Mathematica, Maple and Matlab on a test 500 equations. For Mathematica we report results by setting a timeout of 30 seconds per equation. On a given equation, our model typically finds the solution in less than second.
On all tasks, observe that our model significantly outperforms Mathematica. On function integration, our model obtains close to 100% accuracy, while Mathematica barely reaches 85%. On first order differential equations, Mathematica is on par with our model when uses beam size of 1, i.e. with greedy decoding. However, using a beam search of size 50 our model accuracy goes from 81.2% 97.0%, largely surpassing Mathematica. Similar observations can be made for second order differential equations, where beam search is even more critical since the number of equivalent solutions larger. On average. Matlab and Maple have slightly lower performance than Mathematica on the problems we tested.
Table 4 shows examples of functions that our model was able solve, on which Mathematica and Matlab did not find a solution. The denominator of the function to integrate, -16x + 112r? 204x 6 28 - x + 1, can be rewritten as 1 (4x 14x 22²² With the simplified input:
16. 42xx + 2r
(L (4x+ - 14r + x2)²) 1/2
integration becomes easier and Mathematica able to find the solution.
Equation
Solution
16x
42.²
V'
y sin '(xx" 14x x²)
16
112x7
2012
28 x² 1)1/2
3ry cos(ir)
9x2 sin(rr)² + In' By sys(n(()) =0
y cexp ssnh (3x sin()))
C1 + 3x + 3 log (ar)
"nn""" 2 " -3 1 202 6n 2 3x 3x²2"" 9ry' -3g
y
x (c2 + 4x²)
Table Examples problems our model is able to solve, on which Mathematica and Matlab were not able to find a solution. For each equation, our model finds a valid solution with greedy decoding.

---

Published as a conference paper at ICLR 2020
4.6 EQUIVALENT SOLUTIONS
An interesting property of our model is that it is able to generate solutions that exactly equivalent, but written in different ways. For instance, we consider the following first order differential equation, along with one of its solutions:
9
2
1 log
162x log()y)) 2y 1og(z)² llylog( + 81y =
y
VC+ 2x
In Table 5, we report the top 10 hypotheses returned by our model for this equation. We observe that all generations are actually valid solutions, although they are expressed very differently. They are however not all equal: merging the square roots within the first and third equations would give the same expression except that the third one would contain factor 2 in front of the constant e, but up to a change of variable, these two solutions are actually equivalent. The ability of the model to recover equivalent expressions, without having been trained to do so, is very intriguing.
Hypothesis
Score
Hypothesis
Score
9
log (m)
9
-0.017
-0.124
Vc + 2x
clog (x)
+ log (x)
9/
9V2
-0.056
-0.139
Ve 2 / (x)
V Vclog (x) 2x log (1r
9/2.2
(o)
9
-0.115
-0.144
2/c+2
c + 2 log (z)
9/x
-0.117
9,
-0,205
c clog (x) + 2x log (x)
(z) 2 log (x)
9
-0.124
9
-0.232
2/C + ZV /og (x)
clog (x) 2x log (x) + log (x)
Table 5: Top 10 generations of model for the first order differential equation 162r log(x)y' + 22y I 81y log(x)) + 81y = O. generated with beam search. All hypotheses are valid solutions, and are equivalent up to a change of the variable c. Scores are log-probabilities normalized hy sequence lengths.
4.7 GENERALIZATION ACROSS GENERATORS
Models for integration achieve close 100% performance on held-out test samples generated with the same method as their training data. In Table 6, we compare the accuracy on the FWD, BWD and IBP test sets for models trained using different combinations of training data. When the test set is generated with the same generator as the training set, the model performs extremely well. For instance, the three models trained either on BWD, BWD IBP or BWD + IBP + FWD achieve 99.7% accuracy on the BMD test set with a beam size of 50.
On the other hand, even with a beam size of 50, a FWD-trained model only achieves 17.2% accuracy on the BWD test set, and BWD-trained model achieves 27.5% on the FWD test set. This results from the very different structure of the FWD and BWD data sets (cf. Table and the discussion in Section of the appendix). Overall, a model trained on BWD samples learns that integration tends to shorten expressions, a property that does not hold for FWD samples. Adding diversity to the training set improves the results. For instance, adding IBP -gnnerated examples to the BWD-trained model raises the FWD test accuracy from 27.5% to 56. 1%, and with additional FWD training data the model reaches 94.3% accuracy. Generalization is further discussed in Section E of the appendix.
10

---

Published as a conference paper at ICLR 2020
Forward (FWD)
Backward (BKD)
Integration by parts (IBP)
Training data
Beam
Beam 10
Beam 50
Beam
Beam 10
Beam 50
Beam
Beam 10
Beam 50
FWD
93.6
93.6
96.2
10.9
13.9
17.2
85.6
86.8
88.9
BWD
18.9
24.6
27.5
98.4
99.4
99.7
42.9
54.6
59.2
BWD 18P
41.6
54.9
56.1
98.2
99.4
99.7
96.8
99.2
99.5
BND IBP FWD
89.1
93.4
94.3
98.1
99.3
99.7
97.2
99.4
99.7
Table 6: Accuracy of our models on function integration. We report the accuracy of our model on the threc integration datasets: forward (FWD), backward (BWD), and integration by parts (IBP), for four models trained with different combinations of training data. observe that a #WD-trained model performs poorly when it tries to integrate functions from the BMD dataset. Similarly, a BWD-trained model only obtain 27.5% accuracy on the FWD datasct, as it fails to integrate simple functions like x² sin(z). On the other hand, training on both the BMD + IBP datascts allows the model reach up to 56.1 % accuracy on FWD. Training all datasets allows the model perform well on the three distributions.
4.8 GENERALIZATION BEYOND THE GENERATOR SYMPY
Our forward generator, FWD, generates set of pairs F) of functions with their integrals. It relies on an external symbolic framework, SymPy (Meurer et al., 2017), to compute the integral of randomly generated functions. SymPy not perfect, and fails to compute the integral of many integrable functions. In particular, we found that the accuracy of SymPy on the BWD test set is only 30%. Our FWD-trained model only obtains an accuracy of 17.2% on BWD. However, we observed that the FWD-trained model is sometimes able to compute the integral of functions that SymPy cannot compute. This means that by only training on functions that SymPy can integrate, the model was able to generalize to functions that SymPy cannot integrate. Table presents examples of such functions with their integrals.
x² (tan (x) + 1) + 2x tan (x) +
tan (x) + x
1+ cos (2x)
x + asinh (sin (2xx)
sin? (2x) +
tan (x) + log (a cos (z))
log (ar cos (z:))
log (ar cos (x))
2x cos (asin² 2 (x)) asin (x)
v1 x² sin? (asin² 2 (x))
sin (asin" 2 (x))
sin (asin² 2 (x)
2x
1
VI +
+ 1
asinh (x²)
(Vo + x asinh (x²))
Vx² +
2/
3(-30² sin (xx TVa
33
v/r-toos (xx3)
(x log (V + cos (2:33 ())
2 + log (Vi + cos (x²))
2 tan" (log (log (x))) 2
2
2x
log (I:) tan (log (log (x:))
tan (log (log (x)))
tan (log (log (x))
Table 7: Examples of functions / integrals that the FWD-trained model can integrate, but not SymPy. Although the FWD model was only trained on a subset of functions that SymPy can integrate, it learned to generalize to functions that SymPy cannot integrate.

---

Published as a conference paper at ICLR 2020
5 RELATED WORK
Computers were used for symbolic mathematics since the late 1960s (Moses, 1974). Computer algebra systems (CAS), such as Matlab, Mathematica, Maple, PARI and SAGE, are used for variety of mathematical tasks (Gathen & Gerhard, 2013). Modern methods for symbolic integration are based on Risch algorithm (Risch, 1970). Implementations can be found in Bronstein (2005) and Geddes et al. (1992). However, the complete description of the Risch algorithm takes more than 100 pages, and is not fully implemented current mathematical framework.
Deep learning networks have been used to simplify treelike expressions. Zaremba et al. (2014) use recursive neural networks to simplify complex symbolic expressions. They use tree represen- tations for expressions, but provide the model with problem related information: possible rules for simplification. The neural network is trained to select the best rule. Allamanis et al. (2017) propose framework called neural equivalence networks to learn semantic representations of alge- braic expressions. Typically, a model is trained to map different but equivalent expressions (like the 10 expressions proposed in Table 5) to the same representation. However, they only consider Boolean and polynomial expressions. More recently, Arabshahi et al. (2:1));)) used tree-structured neural networks to verify the correctness of given symbolic entities, and predict missing entries in incomplete mathematical equations. They also showed that these networks could be used to predict whether an expression is valid solution of given differential equation.
Most attempts to use deep networks for mathematics have focused on arithmetic over integers (sometimes over polynomials with integer coefficients). For instance, Kaiser & Sutskever (2015) proposed the Neural-GPU urchitecture, and train networks to perform additions and multiplications of numbers given in their binary representations. They show that a model trained on numbers with up-to 20 bits can be applied to much larger numbers at test time, while preserving a perfect accuracy. Freivalds & Liepins (2017) proposed an improved version of the Neural-GPU by using hard non-linear activation functions, and diagonal gating mechanism.
Saxton et al. (2019) use LSTMs (Hochreiter & Schmidhuber, 1997) and transformers on a wide range of problems, from arithmetic to simplification of formal expressions. However, they only consider polynomial functions, and the task of differentiation, which is significantly easier than integration. Trask et al. (2018) propose the Neural arithmetic logic units, learn systematic numerical computation, and that can be used within any neural network. Like Kaiser & Sutskever (2015), they show that at inference their model can extrapolate on numbers orders of magnitude larger than the ones seen during training.
module designed to
6 CONCLUSION
In this paper, we show that standard seq2seq models can be applied to difficult tasks like function integration, solving differential equations. We propose approach to generate arbitrarily large datasets of equations, with their associated solutions. We show that a simple transformer model trained on these datasets can perform extremely well both at computing function integrals, and solving differential equations, outperforming state-of-the-an mathematical frameworks like Matlab or Mathematica that rely on large number of algorithms and heuristics, and complex implementation (Risch, 1970). Results also show that the model is able to write identical expressions in very different ways.
These results are surprising given the difficulty of neural models to perform simpler tasks like integer addition or multiplication. However, proposed hypotheses are sometimes incorrect, and considering multiple beam hypotheses is often necessary to obtain valid solution. The validity of solution itself is not provided by the model, but by an external symbolic framework (Meurer et al., 2017). These results suggest that in the future, standard mathematical frameworks may benefit from integrating neural components in their solvers.
12

---

Published as a conference paper at ICLR 2020
REFERENCES
Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. Learning con- tinuous semantic representations of symbolic expressions. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'II, pp. 80-88. JMLR org, 2017.
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Combining symbolic expressions and black-box function evaluations for training neural programs. In International Conference Learning Representations, 20.8a.
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Towards solving differential equations through neural programming. 2018b.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
M. Bronstein. Symbolic Integration Transcendental Functions. Algorithms and combinatorics. Springer, 2005. ISBN 978-3-540-21493-99
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. Recurrent neural network grammars. In Proceedings the 2016 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 199-209, 2016.
Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. Learning to parse and translate improves neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 72-78, 2017.
Philippe Flajolet and Andrew M. Odlyzko. Singularity analysis of generating functions. SIAM J.
Discrete Math., 3(2):216-244) 1990.
Philippe Flajolet and Robert Sedgewick. Analytic Combinatorics. Cambridge University Press, New York, NY, USA, 1 edition, 2009. ISBN 0521898064, 97800218980665
Karlis Freivalds and Renars Liepins. Improving the neural gpu architecture for algorithm learning. ArXiv, abs/1 702.08727,
2017. 2017.
Joachim von zur Gathen and Jurgen Gerhard. Modern Computer Algebra. Cambridge University Press, New York, NY, USA, 3rd edition, 2013. ISBN 1107039037, 9781 10703993..
Keith O. Geddes, Stephen R. Czapor, and George Labahn. Algorithms for Computer Algebra. Kluwer Academic Publishers, Norwell, MA, USA, 1992. ISBN 0-7923-9259-0.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735 1997.
Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. CoRR, abs/1511.082228 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 4rXi::1412.6980, 2014.
Donald E. Knuth. The Art of Computer Programming, Volume (3rd Ed.): Fundamental Algorithms. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 1997. ISBN 0-201- 89683-4.
Philipp Koehn. Pharaoh: beam search decoder for phrase-basee based statistical machine translation models. In Conference of the Association for Machine Translation in the Americas, pp. 115-124. 124. Springer, 2004.
Sarah Loos, Geoffrey Irving, Christian Szegedy, and Cezary Kaliszyk. Deep network guided proof search. arXiv preprint arX1v:1701.06972,200 2017.
Math Works. Matlab optimization toolbox (r2019a), 2019. The Math Works, Natick, MA, USA.
13

---

Published as a conference paper at ICLR 2020
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondiej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore. Sartaj Singh, Thilina Rathmayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi:
10.7717/peerj-cs. 103. URL https: /t/./iioooooo org/ 10. 7717 --tp:///eerrcc.111
Joel Moses, Macsyma- the fifth year. SIGSAM Bull., 8(3): August 1974. ISSN 0163-5824.
Myle Ott, Michael Auli, David Grangier, et al. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pp. 3953-3962, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting association for computational linguistics, pp. 31 1-318. Association for Computational Linguistics, 2002.
Robert H. Risch. The solution of the problem of integration in finite terms. Bull. Amer. Math. Soc., 76(3):605-600 05 1970.
David E. Rumelhart, James McClelland, and CORPORATE PDP Research Group (eds.). Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, Cambridge, MA, USA, 1986. ISBN 0-262-68053-X.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference Learning Representations, 2019.
N. J. A. Sloane. The encyclopedia of integer sequences, 1996.
Richard P. Stanley. Enumerative Combinatorics: Volume 1. Cambridge University Press, New York, NY, USA, 2nd edition, 2011, ISBN 1107602629, 9781 107602625.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 1104-3112, 2014.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1556-1566, 2015.
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. Advances in Neural Information Processing Systems, pp. 8035-8044, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 6000-6010, 2017.
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In Advances in neural information processing systems, pp. 2773-2781, 2015.
H.S. Wilf. generatinnfimnciionolog:: Third Edition. CRC Press, 2005. ISBN 11-----3--66339 URL https: //t::///www... .wath. www.n.uuuennneecemm edu/ halt/ggologyLinkedz .ww.kkeddd.............
Wolfram-Research. Mathematica, version 12.0, 2019. Champaign, II, 2019.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4660 1410,4615, 2014.
Wojciech Zaremba, Karol Kurach, and Rob Fergus. Learning to discover efficient mathematical identities. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 1, NIPS 14, pp. 1278-1286, Cambridge, MA, USA, 2014. MIT Press.
14

---

Published as a conference paper at ICLR 2020
A A SYNTAX FOR MATHEMATICAL EXPRESSIONS
We represent mathematical expressions trees with operators as internal nodes, and numbers, constants or variables, as leaves. By enumerating nodes in prefix order, we transform trees into sequences suitable for seq2seq architectures.
For this representation to efficient, we want expressions, trees and sequences to be a one-to-one correspondence. Different expressions will always result in different trees and sequences, but for the reverse to hold, we need to take care few special cases.
First, expressions like sums and products may correspond several trees. For instance, the expression 2 + can be represented as any one of those trees:
2
We will assume that all operators have at most two operands, and that, in case of doubt, they are associative the right. 2 + 3 + 5 would then correspond to the rightmost tree.
Second, the distinction between internal nodes (operators) and leaves (mathematical primitive objects) is somewhat arbitrary. For instance, the number -2 could be represented as a basic object, unary minus operator applied the number 2. Similarly, there several ways represent v5. or the function logio- For simplicity, we only consider numbers, constants and variables as possible leaves, and avoid using unary minus. In particular, expressions like - represented as -1 Here are the trees for 22, v5, 42r and
-2
sqrt
x
5
42 pow
Integers represented in positional notation, as sign followed by sequence of digits (from 0 to 9 base 10). For instance, 2354 and -34 are represented as +2 3 5 4 and 3 4. For zero, a unique representation is chosen (+-0 -0).
B MATHEMATICAL DERIVATIONS OF THE PROBLEM SPACE SIZE
In this section, we investigate the size of the problem space by computing the number of expressions with n internal nodes. We first deal with the simpler case where we only have binary operators (pi = 0), then consider trees and expressions composed of unary and binary operators. each case, calculate generating function (Flajolet & Sedgewick, 2009; Wilf, 2005) from which derive closed formula or recurrence on the number of expressions, and an asymptotic expansion.
B.1 BINARY TREES AND EXPRESSIONS
The main part of this derivation follows (Knuth, 1997) (pages 388-389).
Generating function Let b, be the number of binary trees with internal nodes. We have by = 1 and by Any binary tree with n internal nodes can generated by concatenating left and right subtree with & and n - 1 k internal nodes respectively. By summing over all possible values of k; we have that:
b, = bobn-1 + bibn + + bn- + bn-
Let B(2) be the generating function of bn B(z) bo biz + dzz + byz3
15

---

Published as a conference paper at ICLR 2020
B(z)² bo² + (byb) + b1bb)) + (bobz + bib, + b2bn))² +
b1 + 22 + byz² +...
B(2) bo
2
So, =Bx2 B22) +1 Solving for (22) gives:
B + v1 42
2z
and since B(0) by 1, derive the generating function for sequence b,
B 1 V1v114 11-42 4z
2z
We now derive a closed formula for bn By the binomial theorem,
1
11/2
B(2)
1
(-4z)
2z
k
k=0
1
8
1
2k
1+
xk
2z
2k-
k
1
2k
2
2k
(2k
k-1
2(2k 1)
1:
2k + 2k
2(2k + 1)
1 1++1
k=
2k
x*
1: +1
k=
Therefore
12
(2m)!
6
Tu
n2
(n + 1)bm!
These are the Catalan numbers, a closed formula for the number of binary trees with n internal nodes. We now observe that binary tree with internal nodes has exactly a binary tree can represent P2 operators, and each leaf can take L values, we have that a tree with n nodes can take ph 1 possible combinations of operators and leaves. As result, the number of binary expressions with operators is given by:
leaves. Since each node in
(2m)! Ex (n 1)!m!
P2 L n+1
Asymptotic estimate
To derive an asymptotic approximation of b,. we apply the Stirling formula:
n
'2m
4n
4
n!
2
SO
and
bn
mn
n/Tn /
Finally, we have the following formulas for the number of expressions with n internal nodes:
E, 20
(4p2)" 1+1
TI n
16

---

Published as a conference paper at ICLR 2020
B.2 UNARY-BINARY BINARY TREES
Generating function Let 8n be the number of unary-binary trees (i.e. trees where internal nodes can have one two children) with internal nodes. We have so 1 and 81 (the only internal node is either unary or binary).
Any tree with internal nodes is obtained either by adding a unary internal node at the root of a tree with n internal nodes, or by concatenating with a binary operator left and a right subtree with k and n k internal nodes respectively. Summing up before, have:
Sn = Sp-1 + SuSn-1 + $1 S18n-2 + + Sn-180
Let S2)) be the generating function of the Sn, The above formula translates into
S(2) 5(2) 80 5() =5(2)² + (z 1)5(z) + 1
2
solving and taking into account the fact that S(0) we obtain the generating function of the Sn
S(z) 1 - 1 62 22
2=
The numbers en generated by S(2) are known as the Schroeder numbers (OEIS A006318) (Sloane, 1996). They appear in different combinatorial problems (Stanley, 2011). Notably, they correspond to the number of paths from (0, 0) to (n, n)) of n x grid, moving north, east, or northeast, and never rising above the diagonal.
Calculation Schroeder numbers do not have simple closed formula, but recurrence allowing for their calculation can be derived from their generating function. Rewriting (2) as
2z5(2) + -
62 +
and differentiating, we have
3-2
3
2=5 +
225(z))
v1
62 +
1 62 22
3z
x²
(3 [3==)(1
(z) + 28(:)
[+
1 6z + 2
6z + 22
1 1 32
2 2%
(x) + 23())
62 +2
1 1 62 + 22
x(1 - 62 + (z) + (1-32) (z) = +
Replacing S(z) and (z(z) with their n-th coefficient yields, for n > 1
ns, 6(n 1) s)sn,_ 1 + (n 2)sn-2 8n -3, 0
(n 1) Sm 3(2n 1)s,-1 (n - 2),
Together with = 1 and = 2, this allows for fast (O(n)) calculation of Schroeder numbers.
Asymptotic estimate To derive an asymptotic formula of sn, we develop the generating function around its smallest singularity (Flajolet & Odlyzko, 1990), i.e. the radius of convergence of the power series. Since
6z + 2 (1 (3 vB)²) (1 (3 VB)2)
The smallest singular value is
1
(3 + /8)
and the asymptotic formula will have the exponential term
= (3 38)
22
17

---

Published as a conference paper at ICLR 2020
In neighborhood of r1, the generating function can be rewritten
S(z) ~ (1 v2) (1 2 V (3+ (8) + 0(1 - (3 v8)2)3/2
Since
[=n]V] az
VAT
where [(z) denotes the n-th coefficient in the formal series of F, we have
8n + (2)(3 v8)" Comparing with the number of binary trees, have
(1+ 22)a
23/1VTT/*
23/4 23/1/TH) Tn
8n 1.4((146)"',,
B.3 UNARY-BINARY EXPRESSIONS
In the binary case, the number of expressions can be derived from the number of trees. This cannot be done in the unary-binary case, as the number of leaves in tree with internal nodes depends on the number of binary operators (n2 + 1).
Generating function The number of trees with internal nodes and n2 binary operators be derived from the following observation: any unary-binary tree with ny binary internal nodes can be generated from a binary tree by adding unary internal nodes. Each node in the binary tree can receive one or several unary parents.
Since the binary tree has 2n2 nodes and the number of unary internal nodes be added is n22 the number of unary-binary trees that can be created from a specific binary tree is the number of multisets with 2n2 elements on n n2 symbols, that is
122 (2 +
n
r12
2n1
If by denotes the q-th Catalan number, the number of trees with ny binary operators among n is
72
bwz
2nz
Since such trees have ne + leaves, with L leaves, p2 binary and p1 unary operators to choose from, the number of expressions is
E(n, (" + 112
Th n- n2++1 bn P2 pi
2nz
Summing over all values of n2 (from n) yields the number of different expressions
n n n2
E,
22 pi-na L 12 bnaP2
Pi
2m2
n==0
Let E(2) the corresponding generating function.
E(2)
E E,
n=0
1 + n2
bngP2 P1
n2 n-n n2 12 2
2112
n=0 #y=0
12 + n2
Lp2
Una
2n²
P1
1=0 R==0
12 + 122
Lp2
bna
(piz)"
2n12
Pi
1= 02=0
18

---

Published as a conference paper at ICLR 2020
since ("tte?) 2ng
0 when >> f2
B(z) =
0
baz
Lp2
00
n2
(D12)"
2nz
n2
T.p2
2112
baa
(p12)
P1
2nz
===0
00
2n12
bag (Ip2z)
(p) (p12))
2nz
ng 0
applying the binomial formula
E(z) L Σ
50
Uag (Lp2z)
1
(1 p1
#2=
L
00
Lp22
162
buz
P12
p12)2
=2=0
applying the generating function for binary trees
Lp2z
L
[1-P12) 23
E(2) = 1 21
2 (1-Piz)²
Lpız
1 P1
1 1 L,p2
1- 1-1
2p27
(1 222))
= 1-p12
P1
v(1 (1 - p==) -4T.p22
2p22
Reducing, we have
E(2)= = P12 /1 2(p1 2Lpzk)z + 212
2p2z
Calculation As before, there is no closed simple formula for F,,, but derive recurrence formula by differentiating the generating function, rewritten as
2p²=E(z) + piz
/ 2(pi + 2y2L)= 22
P1 2pp P1
2'22E + (2) +p
/I 2(p1 + 2p2 21
(p) + +pp2l 2) - pl - 2ppz E()))
2p22 F3 (z) + 2p2 F(2) + =
1- 1-2(p1 2p2 + 21
2(1 + 2p2 p22)
(p1 + 2pp2 p1z)(1 - p12)
(x) + 2()
[1 1 - 2(p1 + 2p2L)2 p122
1 - 2(p + 2p22L)z + p1
1- (p) + Lp
+ 2p
+ +12) (p) 1)2
(x) + B()
1- 2(pp + 2p2L)z + 21
1 1-2(pi 2 p1
2p22F 2(p + 2p21))= I))= + =p1=²) + 2p F1=)
(p1 + 2-)))))) (2p2 + P1=) Pi - 1)2)
replacing E(z) and E' with their cocfficients
2p2(nEn, 2(p1 - 2p L) 1) En + pı (n - F) 2)) 2)) + 2p2 Fm (p) + 12) Fn-1) = 0
+ 1)E, (pi 22n - (2)
(n FM (p1 + 222 1) Fu p1 (n 2)
19

---

Published as a conference paper at ICLR 2020
which together with
Eg L E1 = (P1 + P2L)L
provides a formula for calculating En
Asymptotic estimate As before, approximations of En for large n can be found by developing (2)) in the neighbourhood of the root with the smallest module of
2(pl + Lp2 + 21
The roots are
T1
P1 + 2p22 v p + 4p 4p2p1L - Pt
p1
12
P1 2p2
Pi
4p 4p2p1 P1
both are positive and the smallest 12 To alleviate notation, let
=
v pf + 4pp + 4p2P1 L P1
Pt
r2 = + 2p2L +
developing E(z) near 72,
1-P172 / 1/2
+2p
2 12
E(z) 2
P1
+0(1
_ 2///
2p2r2
T2
Pí + 2p2l + r vPI + +2p2L 8:26 1 &
E(z)
0
- 3/3/2
2p2p1
T2
and therefore
En 20 Vōr, 1
V8
(p1 + 2p L + 5y
2p2v 2m
2p2 v 2
p n+1
C GENERATING RANDOM EXPRESSIONS
In this section we present algorithms generate random expressions with n internal nodes. We achieve this by generating random trees, and selecting randomly their nodes and leaves. We begin with the simpler binary case (p)
C. BINARY TREES
To generate a random binary tree with n internal nodes, we use the following one-pass procedure. Starting with empty root node, determine each step the position of the next internal nodes among the empty nodes, and repeat until all internal nodes are allocated.
Start with empty node, set 1;
while >> do
Sample position & from (e, nt); Sample the next empty nodes as leaves; Sample operator, create two empty children;
Set e = e + 1 and n = n 1;
end
Algorithm 1: Generate random binary tree
20

---

Published as a conference paper at ICLR 2020
We denote by the number of empty nodes, by the number of operators yet be generated, and by (e, n) the probability distribution of the position (0-indexed) of the next internal node to allocate.
To calculate (e, nt,, let us define (e, n), the number different binary subtrees that can be generated from empty elements, with internal nodes generate. We have
D(0, n) =0 ((e, 0) = D(e,n) D + D(E
The first equation states that no tree can be generated with zero empty node and n operators. The second equation says that if operator is be allocated, empty nodes must all be leaves and there only one possible tree. The last equation states that if we have e > empty nodes, the first one internal node (D(e 1, trees). This
either leaf (and there are (e 1, n) such trees) allows us to compute (e,, n) for all and n.
To calculate distribution K (e, n),, observe that among the D(e, n) trees with empty nodes and operators, D(e + 1,n 1) have binary node their first position. Therefore
P(K(e,n) = 00 D(e + 1,
D(enn)
Of the remaining D(e - n) trees, D(e,n - 1) have a binary node in their first position (same argument for - 1), that is
P(K(e,n)= = =1) D(e,n fle,n-11
D(e,n)
By induction over k. we have the general formula
P(K(e,n) = D(e 1,
D)e,n)
C.2 UNARY-BINARY TREES
In the general case, internal nodes can be of two types: unary or binary. We adapt the previous algorithm by considering the two-dimensional probability distribution L(e, n) of position (0-indexed) and arity of next internal node (i.e. (()(e,n) = (k; a) is the probability that the next internal node is in position & and has arity a).
Start with an empty node, set 1; while do Sample position & and arity a from L(e, n) (if the next internal node is unary); Sample the next empty nodes as leaves; a 1 then
Sample unary operator; Create one empty child;
Set e k;
end
else
Sample binary operator; Create two empty children; Set E e k
end
Set n = n2 1;
end
Algorithm 2: Generate random unary- unary-binary tree
21

---

Published as a conference paper at ICLR 2020
To compute [(e, n), we derive D(e, n). the number of subtrees with internal nodes that can be generated from empty nodes. We have, for all n and e:
D) =
(ee,0)
D(e,n) = D(e 1,n) D(e,n- D(e 1)
The first equation states that no tree can be generated with zero empty node and n > 0 operators. The second says that no operator only one possible tree. The third equation states that with >> O empty nodes, the first one will either be a leaf (D(e- 1, possible trees), unary operator (D(e, 1) trees), or binary operator (D(e 1,n 1) trees).
be allocated, empty nodes must all be leaves and there is
To derive L(e, n)), we observe that among the D(e, n) subtrees with e empty nodes and n internal nodes to be generated, (e,, 1) have unary operator in position zero, and D(e + 1, n 1) have a binary operator in position zero. As a result, we have
D(e,n-1) and P(L(e,n)= = (0, = D(e + 1, 1) D(e,n)
p(L(e,m)= = (0, 1)) D(e,n)
As in the binary case, we can generalize these probabilities to all positions k: in (0... e 1}
P(L(e, n) = (ks, 1)) D(e 1) and P(L(e,n) (kk D(e - k 1, 1) D(e,n)
D)e,n)
C.3 SAMPLING EXPRESSIONS
To generate expressions, we sample random trees (binary, or unary binary), that we "decorate" by randomly selecting their internal nodes and leaves from list of possible operators mathematical entities (integers, variables, constants).
Nodes and leaves be selected uniformly, according to prior probability. For instance, integers between -a and a could be sampled so that small absolute values are more frequent than large ones. For operators, addition and multiplication could be more common than substraction and division.
all L leaves, and pa operators are equiprobable, an alternative approach to generation can be defined by computing D(e, n)
D(O, n) =0
D(e,0) 0) = L
D)e,n) LD(e - 1, pie, p2Dee+ 1, 1)
and normalizing the probabilities P(L(e,n)) as
P(L(e,n) = (k, = (e and P(L(e,))=(( -k, LeD(e - 1,n 1)
D(e.n)
D(e,n)
Samples then become dependent on the number of possible leaves and operators.
D IMPACT OF TIMEOUT ON MATHEMATICA
In the case of Mathematica, we use function DSolve to solve differential equations, and function Integrate to integrate functions. Since computations can take long time, we set finite timeout to limit the time spent on each equation. Table 8 shows the impact of the timeout value on the accuracy with Mathematica. Increasing the timeout delay naturally improves the accuracy. With timeout of 30 seconds, Mathematica times out on 20% unsolved equations. With limit of 3 minutes, timeouts represent about 10% of failed equations. This indicates that even in the ideal scenario where Mathematica would succeed on all equations where it times out, the accuracy would not exceed 86.2%.
22

---

Published as a conference paper at ICLR 2020
Timeout (8)
Success
Failure
Timeout
5
77.8
9.8
12.4
10
82.2
11.6
6.2
30
84.0
12.8
3.2
60
84.4
13.4
2.2
180
84.6
13.8
1.6
Table 8: Accuracy of Mathematica on 500 functions to integrate, for different timeout values. As timcout delay increases, the percentage of failures due to timcouts decreases. With limit of 3 minutes, timeouts only represent 10% of failures. As result, the accuracy without timeout would not exceed 86.2%.
E GENERALIZATION ACROSS GENERATORS
On the integration problem, we achieve (c.f. Table 6) near perfect performance when the training and test data are generated by the same method (either FWD, BWD, or IBP). Given the relatively small size of the training set (4.10" examples), the model cannot overfit to the entire problem space (10 possible expressions). This shows that:
Our model generalizes well functions created by the training generator.
This property holds for the three considered generators, FWD, BWD, and IBP.
Table 6 also measures the ability of our model to generalize across generators. A FWDD-triined model achieves low performance (17.2% with beam 50) BWD-generated test set. A BWD-trained model does little better on the FWD test set (27.5%), but accuracy remains low. On the other hand, FWD-trained models achieve very good accuracy over IBP-generated set (88.9%), and BWD-trained models stand in the middle (59.2%).
Figure provides explanation for these results. The input/output pairs produced by FWD and BWD have very different distributions: integration tends to shorten BWD generated expressions, and to expand FWD generated expressions. As result, model trained on BWD generated data will learn this shortening feature of integration, which will prove wrong will happen on FWD trained model with a BMD test set. Since IBP keeps average expression lengths unchanged, BWD and FWD-trained models will generalize better to IBP test sets (and be more accurate on FWD-trained models, since their input length distributions are closer).
FWD test set. Similar problems
Length of derivatives
Length of integrals
Forward
Forward
0.06
Backward
0.04
Backward
Integration
Integration parts
Density
Density
0.03
0.04
0.02
0.02
0.01
0.00
0.00
20
40
60
100
20
40
60
80
100
Number of tokens
Number of tokens
Figure 2: Distribution of input and output lengths for different integration datascts. The FWD generator produces short problems with long solutions. Conversely, the BWD generator creates long problems, with short solutions. The IBP approach stands in the middle, and generates short problems with short solutions.
This suggests that what looks at first glance like a generalization problem (bad accuracy of BWD- trained models on FWD generated sets, and the converse) is in fact consequence of data generation. BAD and FWD methods generate training sets with specific properties, that our model will leam. But this can be addressed by adding IBP FWD data the BWD dataset, shown in the two last lines of Table 6. practice, a better approach could be implemented with self-supervised learning, where new training examples generated by the model itself.
23

---

Published as a conference paper at ICLR 2020
Functions and their primitives generated with the forward approach (FWD)
cos" (ar)
a cos"
(x)
x
2x
r sin (2x)
cos (2z)
a: (2x: cos (2r))
3
2
a (ar + 4)
2²
+ 2x 4 log (x: + 2)
x + 2
2
cos (2xc)
log (cos (I:) - 1) log (cos (x) + 1)
+ 2 cos (x)
sin (x:)
2
2
x² 4rx
4xx² +1
3x "inhh" (2x)
x² sinh" (2xr
6
12
log (x²) 4
x log (x²)s
3x log (x²)²
3x log (x²)
3x
x log (z²)² a
4
2
4
Functions and their primitives generated with the backward approach (BWD)
cos (x) + tan (1) +
J: + sin (x) + tan (x)
1
a +
22V/ IV
2x
+ tan (x) tan (x)
o' tta (x:)
cos2 (x:)
ct' +
(x 1)
tan
coo?
c
x tan
I
1
1
1 log (log (a:)
log (x) log (log (x))²
log (log (x:)
-2x² sin (x²) tan (x) + x (tan (x) + 1) cos (x²) + cos (x²) tan (x)
I cos (x²) tan (x)
Functions and their primitives generated with the integration by parts approach (IBP)
x22 (4x + 6 log (x) 3)
x (x + log (+))
12
E + (xr + 3) log (x + +3)
(x + 3)²
x + 3
+ 2 cos2 (x)
+ v2 tan (x) + log (cos (æ))
2 (27x² + 24x log (x) + 94r + 90 log (I))
(2x + 5) (3x log (z)) + 1)
18
x sin? (w)
2x + tau (x)
log (x)
x: log (x) + tan (r)
sin (x)
sin (x) tan (xr)
x sinh (x:
cosh (x) 3x sinh (x) + 6z cosh (x) - 6 sinh (α)
Table 9: Examples of functions with their integrals, generated by our FWD, BWD and IBP approaches. We observe that the F FWD and TPP approaches tend to generate short functions, with long integrals, while the BWID approach generates short functions with long derivatives.
24


Reviews:
[{"review_id": "rkec-H1UKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It is rather interesting for a humble academic to review this paper. It already has a discussion, which I find very valuable, and many tweets and social media exposure and endorsements. It is onerous to review in this setting.\n\nThe paper makes a valuable contribution. The adversarial discussions in this website and the unhelpful hype can in this case be addressed to some extent by the authors. I will start with discussing this. Clearly, the title is too broad. This is not deep learning for symbolic mathematics. In no way does this paper address the essence of what is understood by \"symbolic mathematics\". What the authors address is mapping sequences of discrete quantities to other sequences of discrete quantities. The sequences in this paper correspond to function-integral i/o sequences, and 1st/2nd ODEs-function i/o sequences. I will leave it to the authors to come up with a more informative title, but something like deep learning or transformers for symbolic (1d) integration and simple ODEs with be far more accurate.\n\nTo hammer this point, note that Section 3 discusses removing \"invalid\" expressions: log(0) or sqrt(-2). However, it is the manipulation of infinity and imaginary numbers that could be considered to be one of the greatest achievements of symbolic mathematics over the last couple of hundred years. It is reasonable to expect neural nets to do this one day, because humans can, but this should come with results. It's too early to make the claim in the paper title.\n\nSentences such as \"This suggest (sic) that some deeper understanding of mathematics has been achieved by the model.\" and \"These results are surprising given the incapacity of neural models to perform simpler tasks ...\" are speculative, potentially inaccurate and likely to increase hype. This hype is not needed.\n\nHype and over-claiming aside, I did enjoy reading this paper. The public commenters have already asked important questions about methodology and related work on neural programming that the authors have addressed in comments. I look forward to these being incorporated in the revised pdf.\n\nA big part of the paper is about generating the datasets, and I therefore sympathise with the comment about requesting either a dataset release or the generating code. I see no obvious ethical concerns in this case, and the authors have already kindly offered to do this. This is a commendable and important service to our community and for this alone I would be inclined to vote for acceptance at ICLR.\n\nThe paper is clear and well written. However (i) it would be good to show several examples of input and output sequences (as done already in this website) and (ii) the Experiments section needs work. I'll expand on this next.\n\nThe seq2seq transformer with 8 heads, 6 layers and dimensionality 512 is a sensible choice. The authors should however explain why they expect this architecture to be able to map the sequences they adopt. That is, it is well known that a deep neural network is just a skeleton for an algorithm. By estimating the parameters, we are coming up with (fitting) the algorithm for the given datasets. What is the resulting algorithm? Why are 6 layers enough? Here some visualization would be helpful. See for example https://arxiv.org/pdf/1904.02679.pdf and https://arxiv.org/pdf/1906.04341.pdf For greater understanding of the problem, it may be useful to also try sparse transformers eg https://arxiv.org/abs/1805.08241\n\nBeam search is a crucial component of the current solution. However, the authors simply cite Koehn 2004 for this. First, that work used language models to compute probabilities for beam search. I assume no language models are used in this case. What I'm getting to is that there are not enough details about the beam search in this paper. The authors should include pseudocode for the beam search and give a few examples. The paper (even better thesis) of Koehn is a good template for what should be included. This is important and should be explained. \n\nFor Mathematica, it would be useful to state it does other things and has not been optimized for the two tasks addressed in this paper only. It would also be useful, now that you have more time, to run it for a week or two and get answers not only for 30s but also for 60s. How often does it take longer than 30s? How do you score it then?\n\nPlease do include train and test curves. This would be helpful too. I will of course consider revising my score once the paper is updated. \n\nThanks for constructing this dataset and writing this paper. It is very interesting and promising.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJgm1yNFFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors use a Transformer neural network, originally architected for the purpose of language translation, to solve nontrivial mathematical equations, specifically integrals, first-order differential equations, and second-order differential equations. They also developed rigorous methods for sampling from a large space of relevant equations, which is critical for assembling the type of dataset needed for training such a data-intensive model.\n\nBoth the philosophical question posed by the paper (i.e. can neural networks designed for natural language sequence-to-sequence mappings be meaningfully applied to symbolic mathematics) and the resulting answer (i.e. yes, and such a neural network outperforms SOTA commercially-available systems) are interested in their own right, and together make a strong case for paper acceptance.\n\nDetails appearing in the OpenReview comments which should be explicitly specified in the paper before publication:\n1) How large was the generated training set (40M), and how does this compare to the space of all equations under consideration (1e34).\n2) The authors employ beam search in a non-standard manner, where they check for appearance of the equation solution among all of the generated candidates, rather than selecting the top-1. The fact that the reported accuracy with width-10 and width-50 beam searches are in effect measuring top-10 and top-50 accuracy should be clearly stated.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJgxJL_CKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a method for generating two types of symbolic mathematics problems, integration and differential equations, and their solutions. The purpose of the method is to generate datasets for training transformer neural networks that solve integration and differential-equation problems. The authors note that while solving these problems is very difficult, generating solutions first and corresponding problems next automatically is feasible, and their method realizes this observation. The authors report that transformer networks trained on the synthetically generated solution-problem pairs outperform existing symbolic solvers for integration and differential equation. \n\nHere are the reasons that I like the paper. The observation that solving a symbolic mathematics problem is often a pattern matching process is interesting. It is surprising to know that a transformer network designed to translated the generating problem-solution pairs backward (from problem to solution) works better than the solvers in Mathematica and Matlab. Also, I like nice cute tricks used in the authors' method for generating solution-problem pairs, such as the syntactic condition on a possible position of some constant. The paper is overall clearly written.\n\nI presume that when the authors compare their learned solvers with Mathematica and Matlab, they used a dataset generated by their method. I feel that this comparison is somewhat unfair, although it still impresses me that even for this dataset, the authors' solvers beat Mathematica and Matlab. I suggest to try at least one more experiment on a dataset not generated by the authors' method (integration and differential equation problems from math textbooks or other sources) if possible.\n\n* p3: Why is it important to have a generator that produces the four expression trees in p3 with equal or almost equal probabilities? Do you have any semi-formal or informal justification that the distribution of such a generator better matches the kind of expressions arising in the real world?\n\n* p4: f(x)/x)) ===> f(x)/x)\n\n* \"If this equation can be solved in c1\", p5: How realistic is this assumption?\n\n* p5: 1/2 e^x(...) ===> 0 = 1/2 e^x(...)\n\n* p5: If you have a thought or an observation on the impact of each of the data-cleaning steps in Section 3.4, I suggest you to share this in the paper.\n\n* p6: Why did you remove expressions with more than 512 tokens?\n\n* p6: compare to ===> compared to\n\n* p7: Would you put the reminder of the size of the training set in Section 4.4? It only mentions that of the test set currently.\n\n* p8: 1-(4x^2 ===> (1-(4x^2", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a deep learning approach for tasks such as symbolic integration and solving differential equations. 

The reviewers were positive and the paper has had extensive discussion, which we hope has been positive for the authors. 

We look forward to seeing the engagement with this work at the conference.
Decision: Accept (Spotlight)