Learning advanced mathematical computations from examples
Authors: Francois Charton
Amaury Hayat
Guillaume Lample
Venue: ICLR 2021 Poster (2021)
Keywords: differential equations
computation
transformers
deep learning

Abstract: Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and controllability. We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system. This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.

Published as a conference paper at ICLR 2021
LEARNING ADVANCED MATHEMATICAL COMPUTATIONS FROM EXAMPLES
François Charton" Facebook AI Research fcharton@fb.c. com
Amaury Hayat"
Guillaume Lample
Ecole des Ponts Paristech, Rutgers University Camden amaury www..yaeaacc....... fr
Facebook AI Research glampleffb.com
ABSTRACT
Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such local stability, be- havior at infinity and controllability. We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system. This demonstrates that neural networks learn to per- form complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.
INTRODUCTION
Scientists solve problems of mathematics by applying rules and computational methods to the data at hand. These rules are derived from theory, they are taught in schools or implemented software libraries, and guarantee that a correct solution will be found. Over time, mathematicians have developed a rich set of computational tools that can be applied to many problems, and have been said to be "unreasonably effective" (Wigner) 1960).
Deep learning, on the other hand, learns from examples and solves problems by improving random initial solution, without rolying on domain-related theory and computational rules. Deep networks have proven be extremely efficient for large number of lasks, but struggle on relatively simple, rule-driven arithmetic problems (Saxton et al. 2019 Trask et al. 2018 Zaremba and Sutskever 2014).
Yet, recent studies show that deep learning models can learn complex rules from examples. In natural language processing, models learn output grammatically correct sentences without prior knowledge of grammar and syntax (Radford et al.] 2019). or to automatically map one language into another (Bahdanau al., 2014) (Sutskever et al. 2014). In mathematics, deep learning models have been trained to perform logical inference (Evans et al. 2018). SAT solving (Selsam et al. (2018) or basic arithmetic (Kaiser and Sutskever (2015). Lample and Charton (2220) showed that transformers can be trained Irom generated data to perform symbol manipulation tasks, such as function integration and finding formal solutions of ordinary differential equations.
In this paper, we investigate the use of deep learning models for complex mathematical tasks involving both symbolic and numerical computations. We show that models can predict the qualitative and quantitative properties mathematical objects, without built-in mathematical knowlodge. We consider three advanced problems of mathematics: the local stability and collabbiaily of differential systems, and the existence and behavior al infinity of solutions of partial differential equations. All three problems have been widely researched and have many applications outside of pure mathematics. They have known solutions that rely on advanced symbolic and computational techniques, from formal differentiation, Fourier transform, algebraic full-rank conditions, to function evaluation, matrix inversion, and computation of complex eigenvalues. We find that neural networks can solve these problems with very high accuracy, by simply looking at instances of problems and their solutions, while being totally unaware of the underlying theory. In one of the quantitative problems
Equal contribution, names alphabetic order.

---

Published as a conference paper at ICLR 2021
where several solutions possible (predicting control feedback matrix), neural networks are even able to predict different solutions that those generated with the mathematical algorithms we used for training.
After reviewing prior applications of deep learning to related areas we introduce the three problems consider, describe how we generate datasets, and detail how we train our models. Finally, we present our experiments and discuss their results.
2 RELATED WORK
Applications of neural networks to differential equations have mainly focused on two themes: numerical approximation and formal resolution. Whercas most differential systems and partial differential equations cannot be solved explicitly, their solutions can be approximated
numerically, and neural networks have been used for this purpose (Lagaris et al. 1998 2000
Lee and Kang 1990 Rudd 2013: Sirignano and Spiliopoulos 2018). This approach relies on the universal approximation theorem, that states that any continuuus function approximated by a neural network with one hidden layer over a wide range of activation functions (Cybenko 1989: Hornik et al. 1990 Hornik 1991 Petersen and Voigtlaender 2018
be
Pinkus 1999b. This has proven to be especially efficient for high dimensional problems.
For formal resolution, Lample and Charton (2020) proposed several approaches to generate arbitrarily large datasets of functions with their integrals, and ordinary differential equations with their solutions. They found that transformer model (Vaswani et. al. 2017) trained on millions of examples could outperform state-of- saate-of-the-art the-art symbolic frameworks such as Mathematica or MATLAB WWolfram-Resent Research, 2019 MathWorks 2019) on particular subset of equations. Their model was used to guess solutions, while verification (arguably simpler task) was left to a symbolic framework (Meurer et al. (017). Arabshahi et al. (2018ab) b) proposed to use neural networks to verify the solutions of differential equations, and lound that LSee-LSTMs TTai et al. 2015) were better than sequential LSTMs (Hochreiter and Schmidhuber 1997 at generalizing beyond the training distribution.
Other approaches investigated the capacity of neural nerksrks to perform arithmetic opera-
tions (Kiser and Sutskever, 2015 Saxton et al. 2019 Trask et al. 2018) to short
computer programs (Zaremba and Sutskever 2014). More recently, Saxton et al. (2019) found that neural networks were good at solving arithmetic problems at performing operations such as differentiation or polynomial expansion, but struggled on tasks like prime number docomposition compute. Unlike the questions considered here, most of those problems can be solved by simple algorithmic computations.
on primality tests that require a significant number of steps to
DIFFERENTIAL SYSTEMS AND THEIR STABILITY
A differential system of degree n is system of equations of n variables in ((), xn In, (1),
dx;(1) = (1)) for i= 1...
dt
or, in vector form, with IR" and f IR" R",
dx(t) fI((t)
dt
Many problems can be set as differential systems. Special cases include n-th order ordinary differential equations (letting equations, and some particular partial differential equations (separable equations or equations with characteristics). Differential systems are one of the most studied areas of mathematical sciences. They are found in physics, mechanics, chemistry, biology, and economics as well as in pure mathematics. Most differential systems have no explicit solution. Therefore, mathematicians have studied the properties of their solutions, and first and foremost their stability, notion of paramount importance in many engineering applications.
y, =
(()- systems of coupled differential
2

---

Published as a conference paper at ICLR 2021
3.1 LOCAL STABILITY
Let &e E R" be an equilibrium point, that is, - O. If all solutions x(() converge to Te when their imitial positions z=0) t are close enough, the equilibrium is said be locally stable (see Appendix [ for proper mathematical definition). This problem is well known, f differentiable in Fe; an answer is provided by the Spectral Mapping Theorem (SMT) (Coron 2007 Theorem 10.10):
Theorem 3.1. Let (Fe) be the Jacobian matria of f in De (the matrix of its partial derivatives relative to its variables). Let X be the largest real part of its eigenvalues. If X positive, Ie is an unstable equilibrium. If X is negative, then be is locolly stable equilibrium.
Predicting the stability of given system at a point ze is our first problem. We will also predict X, which represents the speed of convergence when negative, a second experiment. Therefore, to apply the SMT, we need lo:
1. differentiate each function with respect to each variable, obtain the formal Jacobian (xx
f(x) (cOS(22) 1aa---siieee - J(x) ( (-cos(z1)
- -in((1))
21 v1 + x22
2x1
- (2 x22 )
2. evaluate J(xe), the Jacobian in te (a real or complex matrix)
Ze = (0.1,...0.1) E R", ((x,) (- co.(
sin(0.1)
- -2
0.2
3. calculate the eigenvalues N = I...n of (xxe)
\ = -1003, A = 00.441
4. compute À max(RRel(A)))) and return the stability (resp, À the speed of convergence)
A A= 0.441 > 0 + locally stable with decay rate 0.441
3.2 CONTROL THEORY
One of the lessons of the spectral mapping theorem is that instability is very common. In fact, unstable systems are plenty in nature (Lagrange points, epidemics, satellite orbits, etc.), and the idea of trying to control them through external variables comes naturally. This is the controllability problem. It has a lot practical applications, including space launch and the landing the moon, the US Navy automated pilot, recently autonomous cars [Bernhard et al. 2017 Minorsky 1930: Funke et al. 2016). Formally, we are given a system
dr
dt = fx(())
(1)
where RD is the state of the system. We want to find function u(() E RP, the control action, such that, beginning from position ip at = 0, we can reach a position I1 at given
t = T (see Appendix B.. The first rigorous mathematical analysis of this problem by Maxwell (1868). but a turning point was reached in 1963, when Kalman gave a precise condition for linear system (Kalman et al. 1963), later adapted to nonlinear system:
Theorem 3.2 (Kalman condition). Let A 0, ue) and B = (zf if
Spanç Bu u E R" i E {0, (0,....7) n -- 1}} = RR"
(2)
then the system is locally controllable around x Le; u Ue-
When this condition holds, a solution to the control problem that makes the system locally stable in Da is u(t) Wh + ((x((() Iee (c.f. Coron (2007); Kleinman (1970): Lukes (1968
3

---

Published as a conference paper at ICLR 2021
and appendix B.4 for key steps of the proof), where K is the m x control feedback matrix:
AT % BB e e e-A'TT
K = -- c c-ATT
(3)
In the non-autonomous case, where f f= f(x, u, (and A and B) depends on t, 1 can be replaced by:
SpanfD;u E R (0,..., 2n 1) R").
(4)
A())D,()) All these theorems make use of advanced mathematical results, such as the Hamilton theorem, or LaSalle invariance principle. Learning them by predicting controllability and computing the control feedback matrix K is our second problem. To measure whether the system is controllable at a point Te; need to:
where Do(() B()) and D1+1(1)
D DI())
1. differentiate the system with respect to its internal variables, obtain A(x, u)
2. differentiate the system with respect to its control variables, obtain B(x, u)
3. evaluate A and B in (ee)
4. calculate the controllability matrix C with 2 (resp. if non-autonomous)
5. calculate the rank d of C, if d = the system is controllable
6. (optionally) if d = n, compute the control feedback matrix K with
n d Out: System is controllable
(sin(xf) + log(1 [1 + x2) + In: fxx u)
atan( uzz) 1482
Te = [011 =1
22 - 11
K (-22.8 44.0)
A step by step derivation of this example is given in Section
A of the appendix.
3.3 STABILITY PARTIAL DIFFERENTIAL EQUATIONS USING FOURIER TRANSFORM
Partial Differential Equations (PDFs) naturally appear when studying continuous phenomena (e.g. sound, electromagnetism, gravitation). Over such problems, ordinary differential systems not sufficient. Like differential systems, PDEs seldom have explicit solutions, and studying their stability has many practical applications. It is also a much more difficult subject, where few general theorems exist. We consider linear PDEs of the form
0cu(t,z)+ Σ of =0,
la|sk
where f, E R", and u(t, x) are time, position, and state. (a1 ..., an) ) E R" is multi-index and are constants. Famous examples of such problems include the heat equation, transporl equations or Schrodinger equation (Evans) 2010). We want to determine whether solution a(!, of (5) exists for given initial condition u(0, x) No, and if it
tends to zero as - +00. This is matherntticlll answered (see appendix B.4 aaddd Evans (2010) Bahouri al. (2011) for similar arguments) by:
Proposition 3.1. Given
S'R(RR),, the space of tempered distribution, there exists solution S' S'(RR) if there erists a constant C such that
VEER" R" To(E) = or Real(f(E)) C,
(6)
where ūo is the Fourier transform of uo and f(E) is the Fourier polynomial associated with the differential operator D. = Episk Oz In addition, if C' > 0, this solution u(t, x) goes to zero when t + +oc.
Learning this proposition and predicting, given an input D, and ug whether a solution exists, if so, whether it vanishes at infinite time, will be our third and last problem.
4

---

Published as a conference paper at ICLR 2021
To predict whether our PDE has solution under given initial conditions, and determine its behavior at infinity, we need to: find the Fourier polynomial f(E) associated to Dz: find the Fourier transform mn(() of un; minimize f(E) on F; output (0,0) if this minimum is infinite, (1,0) is finite and negative, (1,1) if finite and positive. Optionally, output F. A step by step example is given in Appendix A
In: D2 20% + + +0, -0 1.50,, 8 Out:(1,0) - there exists solution it does not vanish at t t00
DATASETS AND MODELS
To generate datasets, we randomly sample problems and compute their solutions with mathematical software (Virtanen et al. 2020 Meurer et al., 2017) using the techniques described in Section 3 For stability and controllability we generate differential systems with equations and n + q variables (i.c. random functions, g > 0 for controllability).
Following Lample and Charton 2020). we generate random functions by sampling unary- binary trees, and randomly selecting operators, variables and integers for their internal nodes and leaves. We use +, -, /,/ exp, log, sqrt, sin, cos, tan, sin-1, cos" tan as operators, and integers between - 10 and 10 as leaves. When generating functions with n + q variables, we build trees with up to 2(n + q + 1) operators.
Generated trees are enumerated in prefix order (normal Polish notation) and converted into sequences of tokens compatible with our models. Integers and floating point reals are also represented as sequences: 142 as [INT+, 1, 4, 2], and 0.314 as [FLOAT+, 3, DOT, 1, 4, E, INT-, 1]. A derivation of the size the problem space is provided in appendix D.4
Local stability Datasets for local stability include systems with 2 to equations (in equal proportion). Functions that are not differentiable at the equilibrium and degenerate systems are discarded. Since many of the operators we use are undefined at zero, setting x. = 0 would result in biasing the dalaset by reducing the frequency of operators like division, square root, or logarithms, Instead, we select ze with all coordinates equal to 0.01 (denoted as z, = [0.01]). This is, of course, strictly equivalent mathematically to sampling systems with equilibrium at the origin or at any other point.
When predicting overall stability, since stable systems become exponentially rare dimension increases, we use rejection sampling to build a balanced dataset with 50% stable systems. When predicting convergence speed, work from uniform (i.c. unbalanced) sample. The value of À at ze is expressed as a floating point decimal rounded to 4 significant digits. For this problem, we generate two datasets with 50 million systems each.
Control theory Datascts for automonous control include systems with 3 to 6 equations, and 4 to variables (1 to 3 control variables). In the non-aulonomous case, we generale systems with 2 or 3 equations. As above, we discard undefined degenerale systems. We also skip functions with complex Jacobians in Ie (since the Jacobian represents local acceleration, expects its coordinates to be real). We have xe =0.5] or [0.9].
In the autonomous case, more than 95% of the systems are controllable. When predicting controllability, we use rejection sampling to create a balanced datasct. In the non-autonomons case, we use a uniform sample with 83% controllable cases. Finally, to predict foedback matrices, we restrict generation to controllable systems and express the matrix sequence of floating point decimals. All 3 datasets have more than 50 million examples each.
Stability of partial differential equations using Fourier Transform We generate a differential operator (a polynomial in 2n;) and initial condition un- uo is the product of n functions f(a,x,) with known Fourier transforms, and d operators exp (ibbbzz)), with 0 d < 2n and az. by E (-100..... 100). We calculate the existence of solutions, their behavior when - +00, and the set of frequencies, and express these three values as a sequence of 2 Booleans and floating point decimals. Our dataset is over 50 million examples.
5

---

Published as a conference paper at ICLR 2021
Models and evaluation In all experiments, attention heads. We vary the dimension from 64 to 1024, and the number of layers from 1 to 8. We train our models with the Adam optimizer (Kingma and Ba 2014). a learning rate 10-1 and the learning rate scheduler in Vaswani et al. (2017) 7 over mini-batches of 1024 examples. Additional information be found in appendix D.T Training is performed on 8 V100 GPUs with float16 operations. Our qualitative models (predicting stability, controllability and existence of solutions) were trained for about 12 hours, but accuracies close to the optimal values were reached after about 6 hours. Learning curves for this problem can be found in appendix D.3 On quantitative models, more training time and examples were needed: 76 hours for convergence speed, 73 hours for control matrices.
transformer architecture with 8
Evaluation is performed on held-out validation and test sets 10000 examples. We ensure that validation and test examples are never seen during training (given the size of the problem space, this never happens in practice). Model output evaluated either by comparing with the reference solution using problem-specific metric.
EXPERIMENTS
5.1 PREDICTING QUALITATIVE PROPERTIES OF DIFFERENTIAL SYSTEMS
In these experiments, the model is given n functions f R"** + R (n E (2.....6]. for stability, 0 for controllability) and is trained to predict whether the corresponding system is stable, resp. controllable, at a given point zg. This is a classification problem.
To provide baseline for results, use Test (Joulin et al. 2016), text classification tool, which estimates, using a bag of words model, the probability of a qualitative feature (stability) conditional to the distribution of tokens and of small fixed sequences (N-grams of up to five tokens). Such a model can detect simple correlations ions bet ween inputs and outputs, such the impact on stability of the presence of given operator, or the number of equations in the system. It would also find out obvious solutions, due to the specifics of one problem or glitches in the data generator. FastTTex was trained over million examples from our datasct (training over larger sets docs not improve accuracy).
saate-of-thh-art
6-layer transformer with 512 dimensions correctly predicts the system stability in 96.4% of the cases. Since the dataset is balanced, random guessing would achieve 50%. FastTexx achieves 60.6%, demonstrating that whereas some easy cases can be learnt by simple text classifiers, no trivial general solution exists for this dataset. Prediction accuracy decreases with the degree, but remains high even for large systems (Table
Table 1: Accuracy of predictions of stability (chance level: 50%)
Degree
Degree 3 Degree 4 Degree
Overall
Pest
Accuracy
98.2
97.3
95.9
94.1
96.4
60.6
For aulonomous controllability over a balanced dalaset, 6-layer transformer with 512 dimensions correctly predicts 97.4% of the cases. The Fast baseline 70.5%, above the 50% chance level. Whereas accuracy increases with model size (dimension and number of layers), even very small models (dimension 64 and only 1 or layers) achieve performance over 80%, above the Fast baseline (Table D
Table 2: Accuracy of autonomous control task over a balanced sample of systems with 3 to 6 equations.
Dimension 64
Dimension 128
Dimension 256
Dimension 512
Fest
layers
81.0
85.5
88.3
90.4
layers
82.7
88.0
93.9
95.5
layers
84.1
89.2
95.6
96.9
layers
84.2
90.7
96.3
97.4
70.5
6

---

Published as a conference paper at ICLR 2021
For non-autonomout systems, our dataset features systems of degree 2 and 3, 83% controllable. Fast'Text achieves 85.3%, barely above the chance level of 83%. This shows that text classifiers have difficulty handling difficult problems like this one, even in low dimensions. Our model achieves 99.7% accuracy. Again, small models, that would be unsuitable for natural language processing, achieve near perfect accuracy (Table
Table 3: Accuracy for non-autonomous control over systems with 2 to 3 equations.
Dimension 64
Dimension 128
Dimension 256
Dimension 512
Fast
layer
97.9
98.3
98.5
98.9
layers
98.9
99.3
99.5
layers
98.6
99.1
99.4
99.6
6 layers
98.7
99.1
99.5
99.7
85.3
5.2 PREDICTING NUMERICAL PROPERTIES OF DIFFERENTIAL SYSTEMS
Speed of convergence In these experiments, the model trained to predict X, the convergence speed to the equilibrium, up to certain precision. Here, consider predictions to be correct when they fall within 10% the ground truth. Further experiments with different levels of precision (2, or 4 decimal digits) are provided in Appendix C
model with 8 layers and a dimension of 1024 predicts convergence speed with an accuracy of 86.6% overall. While reasonably good results can be achieved with smaller models, the accuracy decrease quickly when model size falls under a certain value, unlike when qualitative properties were predicted. Table summarizes the results.
Table 4: Prediction of local convergence speed (within 10%).
Degree
Degree
Degree 4
Degree
Degree
Overall
4 layers, dim 512
88.0
74.3
63.8
54.2
45.0
65.1
layers, dim 512
93.6
85.5
77.4
71.5
64.9
78.6
8 layers, dim 512
95.3
88.4
83.4
79.2
72.4
83.8
4 layers, dim 1024
91.2
80.1
71.6
61.8
54.4
71.9
6 layers, dim 1024
95.7
89.0
83.4
78.4
72.6
83.8
8 layers, dim 1024
96.3
90.4
86.2
82.7
77.3
86.6
Control feedback matrices In these experiments, we train the model (6 layers, 512
dimensions) to predict a feedback matrix ensuring stability of an autonomous system. We use two metrics to evaluate accuracy:
1) prediction within 10% of all coefficients in the target matrix K given by 3 and provided in the training set,
2) verifying that the model outputs correct feedback matrix Ki, i.e. that all eigenvalues in A+ + BK1 have negative real parts. This makes more mathematical sense, as it verifies that the model provides an actual solution to the control problem (like a differential equation, feedback control problem can have many different solutions).
Using the first metric, 15.8% of target matrices are predicted with less than 10% error. Accuracy is 50.0% for systems with 3 equations, but drops fast as systems becomes larger. These results are very low, although well above chance level (=0.0001%) With the second metric (i.e. the one that actually matters mathematically), we achieve 66.5% accuracy, much better result. Accuracy decreases with system size, but even degree 6 systemss, with 1 x 6 to 3 x 6 feedback matrices, are correctly predicted 41.5% of the time. Therefore, while the model fails to approximate K to a satisfactory level, it does learn to predict correct solutions to the control problem in 66.5% of the cases. This result is very surprising, it suggests that a mathematical property characterizing feedback matrices might have been learned.

---

Published as a conference paper at ICLR 2021
Table 5: Prediction of feedback matrices Approximation vs. feedback.
correct mathematical
Degree 3
Degree
Degree
Degree 6
Overall
Prediction within 10%
50.0
9.3
2.1
0.4
15.8 66.5
Correct feedback matrix
87.5
77.4
58.0
41.5
5.3 PREDICTING QUALITATIVE PROPERTIES OF PDEs
In this setting, the model given a diffcrential operator D3 and an initial condition uy- It is trained to predict if a solution to 0,u Dzu exists and, if so, whether converges to when 1 + +oo. The space dimension (i.e. dimension of x)) is between 2 and 6.
In first series of experiments the model is only trained to predict the existence and convergence of solutions. Overall accuracy is 98.4%. In a second series, we introduce an auxiliary task by adding to the output the frequency bounds JF of uo- We observe it significantly contributes to the stability of the model with respect to hyper-parameeeess In particular, without the auxiliary task, the model is very sensitive to the learning rate scheduling and often fails to converge to something better than random guessing. However, in case convergence, the model reaches the same overall accuracy, with and without auxiliary lask. Table G details the results.
Table 6: Accuracy on the existence and behavior of solutions at infinity.
Space dimension for Dim 2
Dim 3
Dim 4
Dim 5
Dim 6
Overall
Accuracy
99.4
98.9
98.7
98.0
96.9
98.4
DISCUSSION
We studied five problems of advanced mathematics from widely researched areas of mathemat- ical analysis. In three of them, we predict qualitative and theoretical features of differential systems. In two, we perform numerical computations. According to mathematical theory, solving these problems requires combination of advanced techniques, symbolic and numeri- cal, that seem unlikely to be learnable from examples. Yet, our model achieves more than 95% accuracy on all qualitative tasks, and between 65 and 85% on numerical computations.
When working from synthotic data, a question naturally arises about the impact of data generation the results of experiments. In particular, one might wonder whether the model exploiting a defect in the generator or a trivial property of the problems that allows for easier solutions. We believe this is very unlikely. First, because our results are consistent over different problems, using datasets generated with different techniques. Second, because trivial solution would be found by the bag of words model because we build our datasets by direcly sampling problems from a distribution that includes all possible functions (up to the basis operators and the random number generator). This eliminates the biases that can result from sampling special instances or solutions (Yehuda) al. 2020). It also means that the training set is an extremely tiny sample of the whole problem space (over the 50 million examples generated, we did not get a single duplicate).
a baseline. And finally,
Learning from very large samples often raises questions about overfitting and generalisation out of the training distribution. Due to the size of the problem space, it is very unlikely that the model could memorize large number of cases and interpolate between them. Note that because the space of functions from R to I2" has infinite dimension, the universal approximation theorem does not apply here. Note also that for some our problems (e.g. local stability), mathematical theory states that solutions cannot be obtained by simple interpolation. To investigate out-of-distributia generalization, we modified our data generator to produce 10 new test sets for stability prediction. We changed the distribution of operators and variables, and experimented with systems with longer expressions and more equations. Table 7 (see Appendix C.2 for detailed analysis) summarizes key results.
8

---

Published as a conference paper at ICLR 2021
Changes in the distribution of operators and variables have very little impact accuracy, demonstrating that the model can generalize out the training distribution. Our trained model also performs well on systems with longer expressions than the training data. This is interesting because generalizing to longer sequences a known limitation of many sequence to sequence architectures. Finally, model trained systems with 2 to equations predicts the stability of systems of 6 equations to high accuracy (78%). Being able to generalize to larger problem space, with one additional variable, very surprising result, that tends to confirm that some mathematical properties of differential systems have been learned.
Table End to end stability: generalization over different test sets.
Overall
Degree 2
Degree 3
Degree 4
Degree 5
Baseline: training distribution
96.4
98.4
97.3
95.9
94.1
No trig operators
95.7
98.8
97.3
95.5
91.2
Variables and integers: 10% integers
96.1
98.6
97.3
94.7
93.8
Expression lengths: n-3 to 3n+33
89.5
96.5
92.6
90.0
77.9
System degree: degree 6
78.7
seems unlikely that the model follows the same mathematical procedure as human solvers. For instance, problems involving more computational steps, such as non-autonomons controllability, do not result in lower accuracy. Also, providing at train time intermediate results that would help human calculator (frequencies for PDE, Jacobians for stability) does not improve performance. Understanding how the model finds solutions would be very interesting, no simpler solutions than the classical mathematical steps are known.
To this effect, we tried to analyze model behavior by looking at the attention heads and the tokens the models focus on when it predicts specific sequence (following Clark et al. (2019)). Unfortunately, we were not able to extract specific patterns, and found that each head in the model, from the first layer onwards, attends many more tokens than in usual natural language tasks (i.e. attention weights tend to be uniformly distributed). This makes interpretation very difficult.
These results open many perspectives for transformers in fields that need both symbolic and numerical computations. There is even hope that our models could help solve mathematical problems that are still open. On more practical level, they sometimes provide fast alterna- tives classical solvers. The algorithmic complexity of transformer inference and classical algorithms for the problems we consider here is discussed in appendix E.1 However, for the problems in our datasct, the simpler and parallelizable computations used by transformers allow for 10 to 100 times shorter evaluation times (see Appendix E.2
7 CONCLUSION
In this paper, we show that by training transformers over generated datasets of mathematical problems, advanced and complex computations can be learned, and qualitative and numerical tasks performed with high accuracy. Our models have no built-in mathematical knowledge, and learn from examples only. However, solving problems with high accuracy does not mean that our models have learned the techniques we use compute their solutions. Problems such as non-autonomous control involve long and complex chains of computations, which some of the smaller models we used could probably not handle.
Most probably, our models learn shortcuts that allow them to solve specific problems, without having to learn or understand their theoretical background. Such a situation common in everyday life. Most of us learn and use language without understanding its rules. On many practical subjects, we have tacit knowledge and know more than we can tell (Polanyi and Sen ). This may be the way neural networks learn advanced mathematics. Understanding what these shortcuts are, how neural networks discover them, and how they can impact mat hematiciaal practice, is subject for future research.

---

Published as a conference paper at ICLR 2021
REFERENCES
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Combining symbolic ex- pressions and black-box function evaluations for training neural programs. In International Conference on Learning Representations, 2018a.
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Towards solving differential equations through neural programming. 2018b.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to aligu and translate. ar preprint arXiv:1409.0 0z 0473, 2014.
Hajer Bahouri, Jven-Yves Chemin, and Raphaäl Danchin. Fourier analysis and nonlinear partial differential equations, volume 343. Springer Science & Business Media, 2011.
Pierre Bernhard, Mare Deschamps, et al. Kalman on dynamics and contro, linear system theory, optimal control, and filter. Technical report, 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning, What does bert look at? an analysis of bert's attention. arXiv preprint arXiv: 04341, 2019. 2019.
Jean-Michel Coron. Control and nonlinearity, volume 136 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 2007. ISBN 978-0-8218- 3668-2; 0-8218-3668-4.
George Cybenko.
Approximation by superpositions of a sigmoidal function. Malhemalics of control, signals and systems, 2(4):303-31 1989.
Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette, Can neural networks understand logical entailment? arXw preprint 1 08535, 2018.
Joseph Funke, Matthew Brown, Stephen M Erlien, and Christian Gerdes. Collision avoid- ance and stabilization for autonomous vehicles in emergency scenarios. IEEE Transactions Control Systems Technology, 25(4):1204-122 1216, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9
(8):1735-1780, 1997.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networkss, 4(2):251 257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert While. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural nelworks, 3(5):551- 560, 1990.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint ar Xio:1607700155 2016.
Lukasz Kaiser and Llya Sutskever. Neural gpus learn algorithms. CoRR, abs/ 15110151.,8228, 2015.
Rudolf E. Kalman, Yu-Chi Ho, and Kumpati S. Narendra. Controllability of linear dynamical systems. Contributions to Differentiol Equations,
1963. ISSN 0589-5839.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint 112 2014.
David Kleinman. An easy way to stabilize a linear constant system. JEFE Transactions on Automatic Control, 15(6):692 692, 1970.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987-1000, 1998.
10

---

Published as a conference paper at ICLR 2021
Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgion. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041-1049, 2000.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Represenlations, 2020. URL htttps://openreview
hett/forum?iddS1leeeeHHYP
Hyuk Lee and In Scok Kang. Neural algorithm for solving differential equations. Journal of Computational Physics, 91(1):110-13) 1311 1990.
Dahlard L Lukes. Stabilizability and optimal control. Funkcial. Ekvac, 11:39-50, 1968.
MathWorks. Matlab optimization toolbox (r2019a), 2019. The MathWorks, Natick, MA, USA.
James Clerk Maxwell. 1. on governors. Proceedings of the Royal Society of London, pages 270 283, 1868.
Aaron Meurer, Christopher Smith, Mateusz Paprocki, Ondrej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMIT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina. Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717 pttr:/peerssssssseee c0.. 2rg/10.7717/peerjj-ooo
URL https://do:
Nicolas Minorsky. Automatic steering lests. Journal of the American Sociely for Naval Engineers, 42(2):285 310, 1930.
Philipp Petersen and Felix Voigtlaender. Optimal approximation piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-338 2018.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 143-195, 1999.
Michael Polanyi and Amartya Sen. The Tacit Dimension. University of Chicago Press, 2009. ISBN 9780226672984.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Keith Rudd. Solving partial differential equations using artificial neural networks. PhD thesis, Duke University Durham, NC, 2013.
David Saxton, Fdward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathe- matical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L Dill. Learning sat solver from single-bit supervision. arXiv preprint orXim:1800 03685, 2018.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Compulational Physics, 375:1339-1383 1364, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic rep- resentations from tree-structured long short-term memory networks. arXiv preprint NVi:1503300077 2015.
11

---

Published as a conference paper at ICLR 2021
Andrew Trask, Felix Hill, Scott E Reed, Jack Rac, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. Advances in Neural Information Processing Systems, pages 8035-8044, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010, 6010, 2017.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haborland, Tyler Roddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Malthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, F. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ríbeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. O Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 272, 2020. doi: https:/ /oi.org///////////o 10..003//$4192-0111111 11922-99--9-9-0
Eugene P Wigner. The unreasonable effectiveness of mathematios in the natural sciences. communications on pure and applied mathematics, 12:1-14, 1960.
Wolfram-Researde Mathematica, version 12.0, 2019. Champaign, IL, 2019.
Gal Ychuda, Moshe Gabel, and Assaf Schuster. It's not what machines can learn, it's what cannot teach. arXiv preprint (xXi:2002.00339, 2020.
Wojciech Zaremba and Llya Sutskever. Learning execute. arXiv preprint Xiv: 141.:440.44115, 2014.
12

---

Published as a conference paper at ICLR 2021
A EXAMPLES OF COMPUTATIONS
A.1 STEP BY STEP EXAMPLE AUTONOMOUS CONTROL
To measure whether the system
dx (t) sin(xf) + log(1 + x2) + atan(urr)
dt
1 1+22
(t) 32 12
dt
is controllable at point with asymptotic control using Kalman condition we need to
1. differentiate the system with respect to its internal variables, obtain the Jacobian A(z, u)
(200 cos(27) (20))) + a(1+2))
l+u2x9 (1 + x2) -1 atan(ux;)
A(x,u) u)
(1+x2)²
120
1 - --ee 2
2. differentiate the system with respect to its control variables, obtain matrix B(x, u)
(2 (2i((1 u² xz))) ==22)=
B(x, u)
3. evaluate A and B in He = =0.5]. Uh =
1.50 0.46
B(xe, ue) = 0.27
A(Ze, ue) =
0.64
0.36
4. calculate the controllability matrix given by
2
= [B, u.)) = ((02)
(10.64 (1.6
1.50
0.46
0.27
= 0.27
0.40
0.36
0
-0.17
5. output n - d, with d the rank of the controllability matrix, the system is controllable if d= 0
rank(C) 2 - 2 = 0: System is controllable in (I. [0.5]. 1)
6. (optionally) n d 0, compute the control feedback matrix K in 3
K = (-22.8 44.0).
A.2 STEP BY STEP EXAMPLE: STABILITY OF LINEAR PDE
To find the existence and behavior at infinite time of solution, given differential operator D2 and an initial condition we proceed as follows
1. find the Fourier polynomial f(E) associated D
D, = 20 + 0.502 + 04 - 702 - 1.50,, 02 20.00
f(E) = - - E + 2 + 14 + 3iTE
2. find the Fourier transform io(E)) of uo(z) e- 3izz sin(ro) 2.bin1 e =²
To(E) 3/2 //322 (-(2x) 1(2m) (/(6)) (o -e
3. find the set F of frequency & for which To({) # (2.5(2x)-1)
F= (- (2.50)))) x (-20,
13

---

Published as a conference paper at ICLR 2021
4. minimize f(E) F min)( 22.6
5. output (0,0) if this minimum is infinite, (1,0) is finite and negative, (1,1) if finite and posilive. (optionally) output F
Out (1,0) there exists solution it does not vanish at / -> +o0
A.3 EXAMPLES OF INPUTS AND OUTPUTS
A.3.1 LOCAL STABILITY
Speed of convergence al Dr = [0.01]
System
0.01
To atan (8mg2z)
atau (0.0008)
01
(9tng) cos (0.09)
-1250
d:
11 + 2- To
0.01 + 0.1
4
2z3
0.182
Ho
22
5)
01 (x) - (I2 ex:) (tan (xr) + 3)) (log (3) + in)
+3.0log (3) + 3.00
-0.445
2t asin s log
--4) asin (0.06 + 0.01
d: 20 du (20-02)
ain (0.01-o2)
d
1.01e
11 0.06 6x
6.0 (locally stable)
2 201 + +2
dt = =1 sin (xr) - 9.9 10-5
01 7.75.10 2 atan (atan (x1))
de +9
-0.0384
4.22 (rı asin (9)) 6 (3)++17
-(..0 asin (9)) e (3)
400 2=(7- 1/v1) + 0.0178 - 0.00111 47Vi
9
63 cos ([og a>
0.000379 +
3.52.10 11 (locally stable)
J2
-Fp
I1 asin cos (mo) + 32
-1.55 1.32i
14

---

Published as a conference paper at ICLR 2021
A.3.2 CONTROLLABILITY: AUTONOMOUS SYSTEMS
Dimension of uncontrollable space
Autonomous system
at 3e = [0.5]. Wh [0.5]
dsa
asin
[i tan (cos (10))
asin
tan (cos (10))
0.0556
0 (controllabee)
32 + log (10
san (21)
22.36
dt
utoo
dra
2x, + 1/2 - 1.5
de
dag de
u
asin (Do) - 0.5 +
dei
Zo
I1 2xz
atan (zr) 1.46
1
dt
do2
5rg
2.85
(x2)
Azn de
6u + 6.0 G
da1 di
0.75 + .²7 cos (u F2)
2
dez
1o + xo + log (e*²)= 0.75
dzg
de
+To
cos
22
+
asin (u)
0.5 cos
(1) (3)
(controllable)
dai
4(x2+44
d 2.5
1.8 12x022 x1 + 108e"
dzg
s0
3rg log
22 6.54
dz
sin 99 +
-
dt
dza
4 lan
4an
4 tan (4)
15

---

Published as a conference paper at ICLR 2021
A.3.3 CONTROLLABILITY: NON-AUTONOMOUS SYSTEMS
Local controllability
Non-autonomous system
at [0.5], Ur 0.5
dzn (I2 0.5) E asin (8)
de dz e - e + te + 1 - 2e
Falsc
di
dx2 dt
tx2 0.5) asin (6) + tta (8)
Aza
stan (v/rs)
2atan
V2
di
20
dz
Vanat +3 x2 log (20)
di
False
log(2)- 0.5 + (1/ 22)
dr2 = (00 0.5)
dag
Za-T
df
sin [Boe")+
dzy
Prze
(viz(
False
200 + asin (L.02 + 4)
dr3
di
Aza 1
0.5 - 12 + tan (xo) - tan (0.5)
True
(t+ (m) (++u))]
0.5(F+ooo (0.54+0.25))
d=g 2.75- 20 (u 4)
dag
M (u To tan (8)) + 0.5 (8))
1ay
6t 2+(-2+7) 12 (4 11-r) T
dt
Zpdt
True
dea
7(u 0.5) 7 tan (log (x2))
de
+7 tan (log (0.5)
A.3.4 STABILITY OF PARTIAL DIFFERENTIAL EQUATIONS USING FOURIER TRANSFORM
Existence of a solution, + 0 at + to0
PDE 0,u + D²u 0 and initial condition
D2 20.o (20g 04 303 m1 z1
302
no
False False
uo 50(-1800) do
16.6622. ) 89k00
86492 +89k:)
D2 = -404, - 50 - 602 02, 02, 302, Oz, 40
True False
uo (162x002)- (6:1-25xzo- 9622) sin (54x0) sin (3z2))
D2 = 0z, (405 0n, + 402 5g
90,, g
+203 21
05, 403, 01
20xz)
True, False
(33x0) (8izz - 56izi 1622+ 67kxz sin (33xo)
D2 -60 02 + 05 96
904 02
909 a
23
+ +702 06
+ 482
95 6
True, True
"o 60(881)) )e
152)
16

---

Published as a conference paper at ICLR 2021
B MATHEMATICAL DEFINITIONS AND THEOREMS
B.1 NOTIONS STABILITY
Let consider system
dx(t) f(x(t))
(7)
dt
Te is an attractor, if there exists p > such that
(x(0)
Fel p
lim s(t) Te. 60
(8)
But, counter intuitive as may seem, this not enough for asymptotic stability take place.
Definition B.1. We say that is locally (asymptotically) stable equiilibrium if the two following conditions are salisfied:
(i) is a stable point, iee. for every & > 0, there crists such that
(z(0) - xel < < (a(() - xe <E, V2 O.
(9)
(wi) Tee is attractor, i.c. there erists >> such that
[x(00) Tel <
lim to
axt) = Fe-
(10)
In fact, the SMT of Subsection 3.1 deals with even stronger notion of stability, namely the exponential stability defined as follows:
Definition B.2. We say that xe is an exponentially stable equilibrium if xe is locally stable equilibrium and, in addilion, there exist p > 0, X O, and M such that
(x0) xel < |x(t)) Me -=(0))
In this definition, X is called the exponential convergence rale, which is the quantity predicted in our first task. Of course, if re is locally exponentially stable it in addition locally asymptotically stable.
B.2 CONTROLLABILITY
We give here a proper mathematical definition of controllability. Let us consider a non- autonomous system
dx(()) ((x((),((),))
(11)
dt
such that f(ze, we) = 0.
Definition B.3. Let T > 0, we say that the nontinear system 11 is locally controllable at the equilibrium Fe in time T with asymptotic control u, if. for every E > 0, there erists r > such that, for every (æo, 21) E Rd x R with 2o xal S n and [x] xel there erists trojectory (x, u) such that.
x(0)= To. x(T) x1 [m(() u.l < V E [0,
(12)
An interesting remark is that if the system is autonomous, the local controllability does not depend the time considered, which explains that it is not precised in Theorem 3.2
17

---

Published as a conference paper at ICLR 2021
B.3 TEMPERED DISTRIBUTION
We start by recalling the multi-index notation: let a = an)) E N", x E R", and I C (RR), we denole
x² 11 x ... X x
(13)
82 0n ... 02r
is said to be a multi-index and |a| = Di=1 las Then we give the definition of the Schwartz functions:
Definition B.4. A function o C belongs to the Schwartz space S(R") if, for any
multi-inder a and 3,
1200 EER
sup
+00.
(14)
Finally, we define the space of tempered distributions:
Definition B.5. A tempered distribution & E (RR" is linear form on S(R") such that there crists p > and C such that
Ku, Φ)) S C Σ
sup
Ix 0202 A Φ E S(R").
(15)
|a1.181<p
TER"
B.4 PROOFS OF THEOREMS
B.4.1 ANALYSIS OF PROBLEM 2
The proofs of Theorem 3.2 of validity of the leedback matrix given by the expression
and of the extension of Theorem 3.2 to the non-autonomous system given by condition
can be found in Coron 2007). We give here the key steps of the proof for showing that the matrix K given by 3 is valid feedback matrix to illustrate the underlying mechanisms;
Setting V(x(1)) (()" where is solution x' z'(t) f(x,, us K..x xx)), and
Cr A
[-" e -t e - e e-A"T)
BB
(16)
Showing, using the form of Cr, that
di (V((())) = -IB" -IB") -B E
-TA" CT22(1)2
Showing that, if for any / [0, C xz1)² == 0, then for any i E {0,..., n1 1},
x"C 0, Vt [0.
Deducing from the controllability condition
that
=(1)" = 0, A [0,T]. and therefore from the invertibility of CT
r(t) = 0, V t E [0.
Concluding from the previous and LaSalle invariance principle that the system is locally exponentially stable
B.4.2 ANALYSIS OF PROBLEM 3
In this section we prove Proposition
3.1
We study the problem
0,u +
Σ
aa 0.
0 on R, R.
(17)
Sk
18

---

Published as a conference paper at ICLR 2021
with initial condition
mu(0-) = un E (RR),
(18)
and we want to find solution u E ('))))
Denoting the Fourier transform of with respect to æ, the problem is equivalent to
d, + Σ alt
(19)
lo|SA
with initial condition uo E S(R"). As the only derivative now is with respect to time, we can check that
E(t,E) (o(E))
(20)
where f(5) = En (it)", is weak solution to 19 belonging to the space
((0)))))). D'(R")). Indeed, first of all we can check that for any E [0, +00), E - exp (-)) is continuous function and üo belongs to S'(IR") C DRR"", thus ū(t,)) be- longs to ()(..). Besides, t + e eE((E) is C function whose derivative in time are of the form P(g)e- where P(() is polynomial function. 7 is continuous in time and E (1, +00), ('R())). Now we check that is a weak solution to (9)) with initial condition ug. Let & E ([O, +oo) R") the space of smooth functions with compact support, we have
(,.) + Σ aa()t)) (ū, Φ) (ūo, の)
(21)
(mo, 0: 1(9)
x)) ( (ūo, f()) 1191 x} + (To, (())((E))) + (ūo, Φ)
-0.
Hence, u defined by (00 is indeed a weak solution of 19 in (10, +oc), D'())) Now, this does not answer our question as this only tells us that at time t 0, u(t, ('R") which is a less regular space than the space of tempered distribution S'(R"). In other words, at 1= 0, 7 ūo has higher regularity by being in S'(IR") and we would like to know equation 19h preserves this regularity. This is more than regularity issue as, if not, one because such function might not exist. Assume now that there exists constant O such that
cannot define a solution u as the inverse Fourier Transform
4A E R" To(E) or Re(f(E)) > C.
(22)
V E R", Isupp(ão) 1(5) SCt
(23)
This implies that, for any 1> 0, ū E ('R""). Besides, defining for any E N,
Ny(d)= = Σ sup
0500 (())),
(24)
|al.1/<pp
SER"
then for t1, 12 E [0,
Nyp(e
No(((--11)4 1(814 - -
Σ sup 15 (F 6)1,
(25)
|a|18/<p SER
where Po(E, Φ) is polynomial with f(E), Φ(E), and their derivatives of order strictly smaller than p. Besides, each term of this polynomial tend to 0 when i tends to l2 on supp(uo), the set of frequency of ug. Indeed, let Br be multi-index, E N, and Qi(E) be polynomials in E, where i E {O....,).. k}.
Σ e MEM Q:(E)tse-115))) -1sv4) )
k
supp(no) 00 ((E)
(26)
tle -(()))
max a ((E))((5))) EER
max supp(un)
19

---

Published as a conference paper at ICLR 2021
From (22). the time-dependant terms in the right-hand sides converge to 0 when tends to t2. This implies that u E ( S' (R")). Finally let us show the property of the behavior at infinity, Assume that C > 0, one has, for any ø E S(R"))
(ū(t,-), Φ) (uo, e
f(E) 6).
(27)
Let us set g(E) = e- -T(EE) o(E), one has for two multi-indes a and |
1s o g()) s 15" (())((())())))) 1()))
(28)
where Q is sum of polynomials, each multiplied by x(E) or one of its derivatives. Thus (=Q(E) belongs to S(R") and therefore, from assumption (22)
- 0 [ S max [ (E)()) Ct
(29)
EER
which goes to 0 when / +00. This imply that ā(t, -) + 0 in S'(R") when 400, and hence u(t,.) -) + 0. This ends the proof of Proposition 3.1
Let us note that one could try to find solutions with lower regularity, where u is distribution of D' ( IR"), and satisfies the equation
dru + Σ oa 6,=0 on R+ x R"
(30)
Sa
This could be done using for instance Malgrangg-Erhenprreis theorem, however, studying the behavior al. L - +00 may be harder mal mathemaiically, hence this approach was not considered in this paper.
C ADDITIONAL EXPERIMENTS
C.1 PREDICTION OF SPEED OF CONVERGENCE WITH HIGHER PRECISION
In Section 5.1 À is predicted with a 10% margin error. Prediction of to better accuracy can be achieved by training models on data rounded to 2, 3 4 significant digits, and measuring the number of exact predictions on the test sample. Overall, we predict À with two significant digits in 59.2% of test cases. Table summarizes the results for different precisions (for transformors with 6 layers and a dimensionality of 512).
Table 8: Exact prediction of local convergence speed to given precision.
Degree
Degree 3
Degree 4
Degree 5
Degree
Overall
2 digits
83.5
68.6
55.6
48.3
40.0
59.2
3 digits
75.3
53.2
39.4
33.4
26.8
45.7
4 digits
62.0
35.9
25.0
19.0
14.0
31.3
C.2 OUT-OF-DISTRIBUTIO CENERALIZATION
In all our experiments, trained models are tested on held-out samples generated with the same procedure as the training data, and our results prove that the model can generalize out of the training data. However, training and test. data come from the same statistical distribution (iid). This would not happen in practical cases: problems would come from some unknown distribution over problem space. Therefore, is interesting to investigate how the model performs when the test set follows different statistical distribution. This provides insight about how learned properties generalize, and may indicate specific cases over which the model struggles.
To this purpose, we modified the data generator to produce new test datasets for end to end stability prediction (section 5.1. Four modifications considered:
20

---

Published as a conference paper at ICLR 2021
1. Unary operators: varying the distribution of operators in the system. In the training data, mary operators are selected at random from a set of nine, three trigonometric functions, three inverse trigonometric functions, logarithm and ex- ponential, and square root (the four basic operations are always present). In this set. of experiments, without logs and exponentials, only with square roots, and with a different balance of operators (mostly square roots).
generated four test sets, without trigonometric functions,
2. Variables and integers: varying the distribution of variables in the system. In the training data, 30% of the leaves are numbers, the rest variables. We changed this probability to 0.1, 0.5 and 0.7. This has no impact on expression length, but higher probabilities make the Jacobians more sparse.
3. Exprcssion lengths: making expressions longer than in the train set. In the training data, for system of equations, operators. In this experiments, we tried functions between n + and 3n + 3 and 2n + 3 and 4n + This means that the lest sequences are, on average, much longer that those seen at training, a known weakness of sequence to sequence models.
generate functions with to 2n + 3
4. Larger degree: our models were trained on systems with to 5 equations, we tried to test it on systems with 6 equations. Again, this usually proves difficult for transformers.
Note that the two first sets of experiments feature out-of-distribution tests, exploring different distributions over the same problem space as the training data. The two last sets, on the other hand, explore a different problem space, featuring longer sequences.
Table g presents the results of these experiments. Changing the distribution of operators, variables and inlegers has little impact on accuracy, up to two limiting cases. First, over systems of degree five (the largest in set, and more difficult for the transformers) change in operator distribution has a small adverse impact on performance (but not change in variable distribution). Second, which the proportion of integers become very large, and therefore Jacobians become very sparse, the degree of the systems has less impact on performance. But overall results remain over 95%, and the model proves to be very resistant to changes in distribution over the same problem space.
Over systems with longer expressions, overall accuracy lends to decreases. Yel, systems of two or three equations are not affected by a doubling of the number of operators (and sequence length), compared to the training data. Most of the loss performance concentrates on larger degrees, which suggests that it results from the fact that the transformer is presented at test time with much longer sequences that what saw at training. In any case, all results but one are well above the Text bascline (60.5%).
When tested on systems with six equations, the trained model predicts stability in 78.7% of cases. This is a very interesting result, where the model is extrapolating out of the problem space (i.e. no system of six equations have been seen during training) with an accuracy well above chance level, and the fast Text baseline.
21

---

Published as a conference paper at ICLR 2021
Table End to end stability: generalization over different test sets.
Overall
Degree
Degree
Degree 4
Degree 5
Baseline: training distribution
96.4
98.4
97.3
95.9
94.1
Unary operators: no trigs
95.7
98.8
97.3
95.5
91.2
Unary operators: no logs
95.3
98.2
97.1
95.2
90.8
Unary operators: no logs and trigs
95.7
98.8
97.7
95.2
91.0
Unary operators: less logs and trigs
95.9
98.8
96.8
95.0
93.1
Variables and integers: 10% integers
96.1
98.6
97.3
94.7
93.8
Variables and integers: 50% integers
95.6
97.8
96.7
94.3
93.1
Variables and integers: 70% integers
95.7
95.7
95.9
95.7
95.5
Expression lengths: n-3 to 3n++3
89.5
96.5
92.6
90.0
77.9
Expression lengths: 2n-33 4n++3
79.3
93.3
88.3
73.4
58.2
System degree: degree 6
78.7
D MODEL AND PROBLEM SPACE
D.1 MODEL ARCHITECTURE
The networks used in this paper are very close to the one described in Vaswani et al. (2017). They use encoder decoodrr architecture. The encoder stack contains 6 transtormer layers, each with a 8 head self-attention layer, a normalization layer, and a one layer feed forward network with 2048 hidden units. Inputs is fed through trainable embedding and positional embedding, and the encoder stack learns a representation of dimension 512. The decoder contains 6 transformer layers, each with (8-head) self-attention layer, attention (pointing to the encoder output) layer, normalization and feed forward linear layer. Representation dimension is the samne as the encoder (512). The final output sent to linear layer that decodes the results.
The training loss is the cross entropy between the model predicted output and actual result from the dataset. During training, we use the Adam optimizer, with a learning rate of 0.0001 and scheduling (as in Vaswani other, typically between 32 and 128 examples.
(2017)). Mini-batch size varies from one problem the
During training, we use GPU. The model is distributed across GPUs, so that all of them have access to the same shared copy of the model. At cach itcration, every GPU processes an independently generated batch, and the optimizer updated the model weights using the gradients accumulated by all GPU. Overall, this is equivalent to training single GPU, but with times larger batches.
D.2 MODEL BEHAVIOR AND ATTENTIONS HEADS
We tried to analyze model behavior by looking at the attention heads and the tokens the models focus on when it predicts specific sequence. As each head altends many more tokens Ihan in usual natural language lasks, and to improve visualization, we tried to reduce the number of hidden states head can attend by using a top-k on the attention weights, but this deteriorated the performance, and we did not investigate more in this direction. We also ran a sequeence-to-seqqeneee model without attention, so that each input equation mapped to fixed-sized representation. We then fed set of input equations into the model, and used a t-SNE visualization to see whether we can observe clusters of equations. What we observed is mainly that equations with nearby representations have similar length tokens. However, even embeddings in similar locations can lead to different decoded sequences. The relevance of the representations built in the encoder depends how the computation is split between the encoder and the decoder. the decoder does the majority the work, encoder representations become less meaningful.
22

---

Published as a conference paper at ICLR 2021
D.3 LEARNING CURVES
Although all generated datasets included more than 50 million examples, most models were trained less. Figure [ shows how performance increases with the number of training examples, for the end to end stability problem (i.e. predicting whether syslems of degree to 5 are stable). There are twelve curves corresponding lo many experiments over shuffled versions of the dataset (i.e. different experiments used different parts of the dataset).
Overall, less than 10 million examples are needed achieve close optimal accuracy, Learning curves from different experiments are close, which proves the stability of the learning process.
Accuracy vs #training examples
95
90
85
Accuracy
80
75
70
65
60
55
0
10
20
30
40
Training examples (millions)
End to end stability accuracy number of training examples. 12 models, trained over shuffled versions of the same dataset.
Figure 1:
D.4 SIZE OF THE PROBLEM SPACE
Lample and Charton 2020) provide the following formula to calculate the number of functions with m operators:
En =
Ei (g) + L2L
(m + 1) = (qi + 22m - 1) (m)))mmm Em
Where L is the number of possible leaves (integers or variables), and 9ı and 92 the number of unary and binary operators. In the stability and controllability problems, we have qi Q2 and L 20 + q, with the number of variables.
Replacing, we have, for function with variables and operators
Eo(g) 20 +
E, (q) (89 + 420 + 4)
(m Fm (g) (169 82m 1)Em. 4(m Fm.-2
In the stability problem, we sampled systems n functions, with n variables, from 2 to 6. Functions have between 3 and 2n 2 operators. The number of possible systems is
6
2n+2
PS. = Σ ΣE
Emm(n) (n))
Em
(6) 2.10
n=2
m==3
(since Em (n) increases exponentially with m and n, the dominant factor in the sum the term with largest and n))
23

---

Published as a conference paper at ICLR 2021
In the autonomous controllability problem, generated systems with n functions (n between 3 and 6), and n + variables (p between 1 and n/2). Functions had between n + p and 2n + 2p + 2 operators. The number of systems is
n/2 (n+p+1) Σ
En (h(m++p)) E2) x 4.10
PSaut Σ
Σ
m=ntp
For the non-autonomous case, the number of variables in + ++ 1, between 2 and 3 and p I, therefore
2(++2)
Em + 2) Em(n+2)
PS Σ
Σ M=n+1
E10(5) 3.1074
Because expressions with undefinite degenerate jacobians skipped, the actual problem space size will be smaller by several orders magnitude. Yet, problem space remains large enough for overfitting by memorizing problems and solutions to be impossible.
E
COMPUTATION EFFICIENCY
E.1 ALGORITHMIC COMPLEXITY
Let n be the system degree, p the number of variables and g the average length (in tokens) of functions in the system. In all problems considered here, have p On). Differentiating or evaluating an expression with tokens is O(q), and calculating the Jacobian of our system is O(npq), i.c. O(n²g).
In the stability experiment, calculating the eigenvalues of the Jacobian will be O(n²) in most practical situations. In the autonomous controllability experiments, construction of the x np Kalman matrix is O(nāp), and computing its rank, via singular value decomposition or any equivalent algorithm, will be O(n²p) well. The computations (multiplication, exponentiation and inversion are all O for a square n matrix). As result, for controllability, complexity is Onr²). Overall, the classical algorithms have a complexity of O(n²q) for Jacobian calculation, and Onn²) (stability) and O(n²) (controllability) for the problem specific computations.
complexity arise for feedback matrix
Current transformer architectures are quadratic in the length of the sequence, in our case ng. a transformer will be O(n2g²) (in speed and memory usage). Therefore, the final comparison will depend on how q, the average length of equations, varies with n, the number of parameters. If q m0=0p O(1) or ((log(n)), transformers have large advantage over classical methods. This means sparse Jacobians, condition often met in practice. For controllability, the advantage remains if q = 1/2) and the two methods are asymptotically equivalent O(n.
However, current research working on improving transformer complexity to log-linear linear. If this happened (and there seem to be theoretical reason preventing it), transformers would have lower asymptotic complexity in all cases.
24

---

Published as a conference paper at ICLR 2021
E.2 COMPUTATION TIME VERSUS EVALUATION TIME
Table IO compares the average time needed to solve one problem, for a trained transformer running on GPU, and Python implementation of the algorithms, running on MacBook Pro.
Table 10: Speed comparison between trained transformers and mathematical libraries. Average time solve one system, seconds.
Task
Mathematical libraries
Trained transformers
Stability end end
0.02
0,0008
Stability largest eigenvalue
0.02
0.002
Controllability (autonomous)
0.05
0.001
Predicting feedback matrix
0.4
0.002
25


Reviews:
[{"review_id": "HQW-C05114", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=Quality=\nHigh: well executed and motivated\n\n=Clarity=\nWell-situated wrt to related work.  Baseline needs to be explained more (see below).\n\n=Originality=\nThe ML method is standard. The novelty is in setting up the datasets and evaluation metrics. Doing was technically complex. The datasets very valuable and will hopefully be open-sourced.\n\n=Significance=\nThis paper demonstrates that neural networks are surpisingly good at the task of predicting certain properties of differential systems, such as their stability. This is a neat result and is sufficient for publication. However, the paper would have more impact if there was a concrete proof of concept of how such a tool could be used to improve the lives of practioners.\n\n=Motivation/Introduction=\nI agree with you that it is very impressive that neural networks can nearly solve these tasks. It would be helpful if you devoted more of the exposition to explaining how your classifiers could be used by practitioners to improve their workflows. Are there engineering applications, for example,  where the classifier could be used to quickly screen proposed systems?\n\nAlso, I'd like to better understand what the novel capability of your tool is. Is it faster than numerical methods for verifying these properties? More accurate? Applicable in situations where symbolic manipulation is impossible?\n\n=Predicting Control feedback matrices=\nI found this section very cool and encouraging for future work! I'd mention it in the intro more.\n\n\n=Baseline classifier=\nMy primary hesitation with the paper is that there is not enough justification for why your baseline is sufficient. \n\nYou need to provide far more details about the FastText baseline. Describing the name of the software package is insufficient. What is the actual model? \n\n\"Such high performances over difficult mathematical tasks may come as a surprise, and one\nmight wonder whether the model is exploiting some defect in the dataset, or some trivial\nproperty of the problems that would allow an easy way to correct solutions. We believe\nthis is very unlikely...because a trivial solution would be found by the text classification tool we use as a baseline.\"\n\nThis argument is weak without an explanation of what the baseline model is and what its inductive biases are.\n\n\nAre there no heuristics from the application community that would serve as baselines? \n\n\nIn the discussion, you mention in passing that \"providing at train time intermediate results that would help a human calculator (frequencies for PDE, or Jacobians for stability) does not improve accuracy.\" It seems to me that a baseline method would be a simple machine learning model on top of some hand-crafted features of these intermediate results.\n\n\n=Discussion section=\n\"in some of our problems, even a model with one layer and 64 dimensions\nobtains a high accuracy, and such a small model would never be able to memorize that\nmany examples.\"\nEither make this precise or remove it. At first glance, it seems to be that 64 dimensions has a lot of capacity. For example, 2^64 is much bigger than the size of your dataset.\n\n\n=Open Sourcing=\nWill you be able to open-source the datasets? Doing so would considerably increase the future impact of your work, as it would provide a benchmark for future ML methods.\n\n=Evaluation set=\nIt would be cool if you could have a couple of anecdotes of applying your classifier to famous equations from papers, particularly ones where the derivations to prove stability, for example, were quite tedious. You argue that the test set is representative, since it is uniformly sampled, but are the equations of interest to the community in some corner of this space?\n\n=Analysis/interpretation=\n\nDo the attention patterns of the transformer reveal anything interesting?\n\nWhat seems to characterize the equations that the model makes mistakes on?\n\nCan you use your model to get  per-equation embeddings? Do they reveal interesting cluster structure in the data?\n\n*** After reading the authors' responses ***\nI have raised my score to a 7. I felt that some of the key questions, e.g. regarding generalization to mathematical expressions that are qualitatively different than the training data,  were answered well. This paper should not be reviewed as being methods-driven. It's about demonstrating a new way that deep learning could be transformative for engineering, by allowing engineers to screen proposed designs for stability, etc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "H3CdLD7vPLj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that transformer models can be used to accurately learn advanced mathematical computations from millions of examples.  The problems are drawn from the fields of differential equations and control theory.  The selected problems are ones that are solvable using known algorithms; however, these algorithms involve a sequence of advanced mathematical operations (e.g., differentiation, calculating the rank of a matrix, calculating the eigenvalues of a matrix, etc), for which no known simple shortcuts exist. For the experiments in this paper, for each problem a large number (50 million) of training examples are randomly generated, and are then used to train a transformer model.  Across these problems, the paper shows that the neural network is able to solve these problems at high accuracy (96-99.7% accuracy).\n\nStrengths\n- The paper ask a well-motivated question regarding whether neural networks are able to learn complex mathematical operations from examples.\n- The paper is clearly written, and the experimental rigor/quality appears quite high.\n- The empirical results are quite intriguing, and raises many interesting questions for future research.  For example, (1) how is a transformer model managing to attain such high accuracy on these problems, (2) what other complicated mathematical problems might be similarly learnable, (3) what are the practical implications to real systems of these results.\n- The paper does a good job considering the various potential counterarguments to its conclusions in the discussion section (Section 5.4).  For example, the paper argues convincingly that (1) it’s unlikely the model is exploiting some trivial distinction between the positive vs negative examples because the examples are sampled randomly, and that (2) it’s unlikely the model is interpolating between solutions because the problem space is so much larger than the training set (and because small models which would be unable to memorize the training set also perform well).\n\n Weaknesses\n- I would have appreciated more of a discussion about the computational cost of solving these problems mathematically vs. solving them with a neural network.  What is the computational complexity (Big-O) of each of the known mathematical algorithms for solving these problems?  Are there large computational savings from using a neural network?  What are the practical implications of the results shown in this paper?\n- I think including a few more baselines would have been useful.  Also, a brief description of the FastText model would make the paper more self-contained.   One question: Why is the FastText model in Section 5.1 only trained with 2 million examples, while the transformer model is trained with 50 million examples?\n- In the current experiments, the test set is drawn from the same exact (random) distribution as the training set.  I was very curious whether the model would have been able to attain high test-time accuracy, had the test examples been drawn from a different distribution.  In particular, I’d be curious how difficult it would be to construct a test distribution on which the current model performs terribly.  This line of questioning would be able to start better answering whether the model is solving the problem in a way that truly generalizes.\n- I think the paper could be strengthened by adding additional error analysis to try to better understand the errors made by the transformer model.\n- The paper says “training is performed on 8 V100 GPUs with float16 operations”: It would have been nice to hear more about the training process: For example, low long did training take for the various problems, and how was training distributed across the GPUs?\n- I would have appreciated more of a discussion about the broader implications and significance of these results.\n\nOverall, I thought the paper was very interesting and well executed, and I think it would make a very nice addition to ICLR 2021.", "extra_scores": {}, "extra_text": {}}, {"review_id": "483s-M6Uc-4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the use of deep learning models, and specifically transformers, to learn mathematical properties of differential systems. Authors tackle three problems:\n\n* Local stability: where the goal is predicting the stability of a given system at a given point, similar to Spectral Mapping Theorem.\n* Control theory: where the goal is predicting controllability and computing the control feedback matrix as in Kalman condition.\n* Stability of PDEs using Fourier transform: where the goal is to predict the existence and stability of a PDE given its differential operator.\n\nOverall, this paper is well written and tackles an intersting problem with potential useful insights. I believe it can be improved by making the discussion more rigorous:\n\n1. How general are the class of functions considered in each problem? For example, you generate random functions by sampling unary- binary trees. Does this give the guarantee that the majority of solutions for each of three problems can learned by deep learning methods?\n\n2. How this work compares to previous ones such as (Lample and Charton 2020) that study learning symbolic mathematics using deep learning? Any distinct insight when investigating numerical and differential systems?\n\n3. It is useful to see if generalizability of learned mathematical computation can be assessed by experiments, where training datasets are generated using some class of functions, and they are tested on a separated class of functions. Are such experiments feasible using unary- binary trees?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Lpxs4k-mgd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically demonstrated the effectiveness of neural networks for learning to predict different mathematical properties of dynamical systems, which achieve high accuracy on synthetic datasets generated by the authors.\n\n\nPros:\n+ the paper is clearly written and easy to follow\n+ the dataset seems to be carefully generated and is large.\n\nCons:\n- The authors do not clearly state their methodology, including the concrete architecture of the transformer-based model, and the training loss for optimizing the model. As shown in the experiments, the accuracy of the proposed model is much higher than the baseline, but it is not clear what the major reason is, and why.\n- As the dataset contains about 50 million samples, and only 10000 of them are held out for test and validation, which means the training dataset contains sufficient data and the result could just be overfitting. Besides, the authors do not describe clearly how they generate the dataset and what is the problem distribution. For some distribution (e.g., those with smaller variance), maybe 50 million is large enough and there won't be a generalization issue. As a well-known fact that neural network has universal approximation ability, it is not surprising that it can learn to predict the mathematical properties given enough data. It will be better if the authors could show a '#training sample VS test accuracy' curve to show how the method behaves differently given different numbers of training samples.\n\nOverall this paper does not have enough technical contributions, so I vote for a clear reject.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that transformer models can be used to learn certain advanced mathematical concepts such as the local stability of differential equations. Reviewers found this surprising and useful for engineers, and the evaluation was adequate. They also felt that it opens the doors to similar studies on other aspects of mathematics.
Decision: Accept (Poster)