Learning advanced mathematical computations from examples

1 Introduction . Scientists solve problems of mathematics by applying rules and computational methods to the data at hand . These rules are derived from theory , they are taught in schools or implemented in software libraries , and guarantee that a correct solution will be found . Over time , mathematicians have developed a rich set of computational tools that can be applied to many problems , and have been said to be “ unreasonably effective ” ( Wigner , 1960 ) . Deep learning , on the other hand , learns from examples and solves problems by improving a random initial solution , without relying on domain-related theory and computational rules . Deep networks have proven to be extremely efficient for a large number of tasks , but struggle on relatively simple , rule-driven arithmetic problems ( Saxton et al. , 2019 ; Trask et al. , 2018 ; Zaremba and Sutskever , 2014 ) . Yet , recent studies show that deep learning models can learn complex rules from examples . In natural language processing , models learn to output grammatically correct sentences without prior knowledge of grammar and syntax ( Radford et al. , 2019 ) , or to automatically map one language into another ( Bahdanau et al. , 2014 ; Sutskever et al. , 2014 ) . In mathematics , deep learning models have been trained to perform logical inference ( Evans et al. , 2018 ) , SAT solving ( Selsam et al. , 2018 ) or basic arithmetic ( Kaiser and Sutskever , 2015 ) . Lample and Charton ( 2020 ) showed that transformers can be trained from generated data to perform symbol manipulation tasks , such as function integration and finding formal solutions of ordinary differential equations . In this paper , we investigate the use of deep learning models for complex mathematical tasks involving both symbolic and numerical computations . We show that models can predict the qualitative and quantitative properties of mathematical objects , without built-in mathematical knowledge . We consider three advanced problems of mathematics : the local stability and controllability of differential systems , and the existence and behavior at infinity of solutions of partial differential equations . All three problems have been widely researched and have many applications outside of pure mathematics . They have known solutions that rely on advanced symbolic and computational techniques , from formal differentiation , Fourier transform , algebraic full-rank conditions , to function evaluation , matrix inversion , and computation of complex eigenvalues . We find that neural networks can solve these problems with a very high accuracy , by simply looking at instances of problems and their solutions , while being totally unaware of the underlying theory . In one of the quantitative problems ∗ Equal contribution , names in alphabetic order . where several solutions are possible ( predicting control feedback matrix ) , neural networks are even able to predict different solutions that those generated with the mathematical algorithms we used for training . After reviewing prior applications of deep learning to related areas we introduce the three problems we consider , describe how we generate datasets , and detail how we train our models . Finally , we present our experiments and discuss their results . 2 Related work . Applications of neural networks to differential equations have mainly focused on two themes : numerical approximation and formal resolution . Whereas most differential systems and partial differential equations can not be solved explicitly , their solutions can be approximated numerically , and neural networks have been used for this purpose ( Lagaris et al. , 1998 ; 2000 ; Lee and Kang , 1990 ; Rudd , 2013 ; Sirignano and Spiliopoulos , 2018 ) . This approach relies on the universal approximation theorem , that states that any continuous function can be approximated by a neural network with one hidden layer over a wide range of activation functions ( Cybenko , 1989 ; Hornik et al. , 1990 ; Hornik , 1991 ; Petersen and Voigtlaender , 2018 ; Pinkus , 1999 ) . This has proven to be especially efficient for high dimensional problems . For formal resolution , Lample and Charton ( 2020 ) proposed several approaches to generate arbitrarily large datasets of functions with their integrals , and ordinary differential equations with their solutions . They found that a transformer model ( Vaswani et al. , 2017 ) trained on millions of examples could outperform state-of-the-art symbolic frameworks such as Mathematica or MATLAB ( Wolfram-Research , 2019 ; MathWorks , 2019 ) on a particular subset of equations . Their model was used to guess solutions , while verification ( arguably a simpler task ) was left to a symbolic framework ( Meurer et al. , 2017 ) . Arabshahi et al . ( 2018a ; b ) proposed to use neural networks to verify the solutions of differential equations , and found that Tree-LSTMs ( Tai et al. , 2015 ) were better than sequential LSTMs ( Hochreiter and Schmidhuber , 1997 ) at generalizing beyond the training distribution . Other approaches investigated the capacity of neural networks to perform arithmetic operations ( Kaiser and Sutskever , 2015 ; Saxton et al. , 2019 ; Trask et al. , 2018 ) or to run short computer programs ( Zaremba and Sutskever , 2014 ) . More recently , Saxton et al . ( 2019 ) found that neural networks were good at solving arithmetic problems or at performing operations such as differentiation or polynomial expansion , but struggled on tasks like prime number decomposition or on primality tests that require a significant number of steps to compute . Unlike the questions considered here , most of those problems can be solved by simple algorithmic computations . 3 Differential systems and their stability . A differential system of degree n is a system of n equations of n variables x1 ( t ) , ... , xn ( t ) , dxi ( t ) dt = fi ( x1 ( t ) , x2 ( t ) , ... , xn ( t ) ) , for i = 1 ... n or , in vector form , with x ∈ Rn and f : Rn → Rn , dx ( t ) dt = f ( x ( t ) ) Many problems can be set as differential systems . Special cases include n-th order ordinary differential equations ( letting x1 = y , x2 = y′ , ... xn = y ( n−1 ) ) , systems of coupled differential equations , and some particular partial differential equations ( separable equations or equations with characteristics ) . Differential systems are one of the most studied areas of mathematical sciences . They are found in physics , mechanics , chemistry , biology , and economics as well as in pure mathematics . Most differential systems have no explicit solution . Therefore , mathematicians have studied the properties of their solutions , and first and foremost their stability , a notion of paramount importance in many engineering applications . 3.1 Local stability . Let xe ∈ Rn be an equilibrium point , that is , f ( xe ) = 0 . If all solutions x ( t ) converge to xe when their initial positions x ( 0 ) at t = 0 are close enough , the equilibrium is said to be locally stable ( see Appendix B for a proper mathematical definition ) . This problem is well known , if f is differentiable in xe , an answer is provided by the Spectral Mapping Theorem ( SMT ) ( Coron , 2007 , Theorem 10.10 ) : Theorem 3.1 . Let J ( f ) ( xe ) be the Jacobian matrix of f in xe ( the matrix of its partial derivatives relative to its variables ) . Let λ be the largest real part of its eigenvalues . If λ is positive , xe is an unstable equilibrium . If λ is negative , then xe is a locally stable equilibrium . Predicting the stability of a given system at a point xe is our first problem . We will also predict λ , which represents the speed of convergence when negative , in a second experiment . Therefore , to apply the SMT , we need to : 1. differentiate each function with respect to each variable , obtain the formal Jacobian J ( x ) f ( x ) = ( cos ( x2 ) − 1− sin ( x1 ) x21 − √ 1 + x2 ) , J ( x ) = ( − cos ( x1 ) − sin ( x2 ) 2x1 − ( 2 √ 1 + x2 ) −1 ) 2. evaluate J ( xe ) , the Jacobian in xe ( a real or complex matrix ) xe = ( 0.1 , ... 0.1 ) ∈ Rn , J ( xe ) = ( − cos ( 0.1 ) − sin ( 0.1 ) 0.2 − ( 2 √ 1 + 0.1 ) −1 ) , 3. calculate the eigenvalues λi , i = 1 ... n of J ( xe ) λ1 = −1.031 , λ2 = −0.441 4. compute λ = −max ( Real ( λi ) ) and return the stability ( resp . λ the speed of convergence ) λ = 0.441 > 0→ locally stable with decay rate 0.441 3.2 Control theory . One of the lessons of the spectral mapping theorem is that instability is very common . In fact , unstable systems are plenty in nature ( Lagrange points , epidemics , satellite orbits , etc . ) , and the idea of trying to control them through external variables comes naturally . This is the controllability problem . It has a lot of practical applications , including space launch and the landing on the moon , the US Navy automated pilot , or recently autonomous cars ( Bernhard et al. , 2017 ; Minorsky , 1930 ; Funke et al. , 2016 ) . Formally , we are given a system dx dt = f ( x ( t ) , u ( t ) ) , ( 1 ) where x ∈ Rn is the state of the system . We want to find a function u ( t ) ∈ Rp , the control action , such that , beginning from a position x0 at t = 0 , we can reach a position x1 at t = T ( see Appendix B ) . The first rigorous mathematical analysis of this problem was given by Maxwell ( 1868 ) , but a turning point was reached in 1963 , when Kalman gave a precise condition for a linear system ( Kalman et al. , 1963 ) , later adapted to nonlinear system : Theorem 3.2 ( Kalman condition ) . Let A = ∂xf ( xe , ue ) and B = ∂uf ( xe , ue ) , if Span { AiBu : u ∈ Rm , i ∈ { 0 , ... , n− 1 } } = Rn , ( 2 ) then the system is locally controllable around x = xe , u = ue . When this condition holds , a solution to the control problem that makes the system locally stable in xe is u ( t ) = ue +K ( x ( t ) − xe ) ( c.f . Coron ( 2007 ) ; Kleinman ( 1970 ) ; Lukes ( 1968 ) and appendix B.4 for key steps of the proof ) , where K is the m×n control feedback matrix : K = −Btr ( e−AT [ ∫ T 0 e−AtBBtre−A trtdt ] e−A trT ) −1 . ( 3 ) In the non-autonomous case , where f = f ( x , u , t ) ( and A and B ) depends on t , ( 2 ) can be replaced by : Span { Diu : u ∈ Rm , i ∈ { 0 , ... , 2n− 1 } = Rn } , ( 4 ) where D0 ( t ) = B ( t ) and Di+1 ( t ) = D′i ( t ) − A ( t ) Di ( t ) . All these theorems make use of advanced mathematical results , such as the Cayley-Hamilton theorem , or LaSalle invariance principle . Learning them by predicting controllability and computing the control feedback matrix K is our second problem . To measure whether the system is controllable at a point xe , we need to : 1. differentiate the system with respect to its internal variables , obtain A ( x , u ) 2. differentiate the system with respect to its control variables , obtain B ( x , u ) 3. evaluate A and B in ( xe , ue ) 4. calculate the controllability matrix C with ( 2 ) ( resp . ( 4 ) if non-autonomous ) 5. calculate the rank d of C , if d = n , the system is controllable 6 . ( optionally ) if d = n , compute the control feedback matrix K with ( 3 ) In : f ( x , u ) = ( sin ( x21 ) + log ( 1 + x2 ) + atan ( ux1 ) 1+x2 x2 − ex1x2 ) , xe = [ 0.1 ] ue = 1 , Out : n− d = 0 System is controllable K = ( −22.8 44.0 ) A step by step derivation of this example is given in Section A of the appendix .

Summary: This paper shows that transformer models can be used to accurately learn advanced mathematical computations from millions of examples.  The problems are drawn from the fields of differential equations and control theory.  The selected problems are ones that are solvable using known algorithms; however, these algorithms involve a sequence of advanced mathematical operations (e.g., differentiation, calculating the rank of a matrix, calculating the eigenvalues of a matrix, etc), for which no known simple shortcuts exist. For the experiments in this paper, for each problem a large number (50 million) of training examples are randomly generated, and are then used to train a transformer model.  Across these problems, the paper shows that the neural network is able to solve these problems at high accuracy (96-99.7% accuracy).