Analysing Mathematical Reasoning Abilities of Neural Models

1 INTRODUCTION . Deep learning , powered by convolutional and recurrent networks , has had remarkable success in areas involving pattern matching ( such as in images ( Krizhevsky et al. , 2012 ) , machine translation ( Bahdanau et al. , 2014 ; Vaswani et al. , 2017 ) , and reinforcement learning ( Mnih et al. , 2015 ; Silver et al. , 2016 ) ) . However , deep models are far from achieving the robustness and flexibility exhibited by humans . They are limited in their ability to generalize beyond the environments they have experienced and are extremely brittle in the presence of adversarially constructed inputs ( Szegedy et al. , 2013 ) . One area where human intelligence still differs and excels compared to neural models is discrete compositional reasoning about objects and entities , that “ algebraically generalize ” ( Marcus , 2003 ) . Our ability to generalise within this domain is complex , multi-faceted , and patently different from the sorts of generalisations that permit us to , for example , translate new sentence of French into English . For example , consider the following question from mathematics , with answer `` −70x− 165 '' . What is g ( h ( f ( x ) ) ) , where f ( x ) = 2x+ 3 , g ( x ) = 7x− 4 , and h ( x ) = −5x− 8 ? To solve this problem , humans use a variety of cognitive skills : • Parsing the characters into entities such as numbers , arithmetic operators , variables ( which together form functions ) and words ( determining the question ) . • Planning ( for example , identifying the functions in the correct order to compose ) . • Using sub-algorithms for function composition ( addition , multiplication ) . • Exploiting working memory to store intermediate values ( such as the composition h ( f ( x ) ) ) . • Generally applying acquired knowledge of rules , transformations , processes , and axioms . In this paper , we introduce a dataset consisting of many different types of mathematics problems , with the motivation that it should be harder for a model to do well across a range of problem types ( including generalization , which we detail below ) without possessing at least some part of these abilities that allow for algebraic generalization . This domain is an important one for the analysis of neural architectures in general . In addition to providing a wide range of questions , there are several other advantages : Mathematics offers a self-consistent universe ; notation is the same across different problem types , which allows for an easily extendable dataset ; and rules and methods learnt on one problem type often apply elsewhere . Addition of numbers ( for example ) obeys the same rules everywhere , and occurs as a “ subroutine '' in other problems ( such as concretely in multiplication , and both concretely and more abstractly in addition of polynomials ) ; models that possess the ability to transfer knowledge will do well on the dataset ( and knowledge transfer may be a necessity for solving harder problems ) . Mathematics is also an interesting domain in its own right ; although models solving the mostly school-level problems in this dataset would not themselves have applications , they may lead on to more powerful models that can solve interesting and substantial new mathematical problems . But more generally , it is no coincidence that experiments seeking to validate new architectures which aim capture algorithmic/systematic reasoning have often been drawn from this domain ( Graves et al. , 2016 ; Kaiser & Sutskever , 2015 ; Joulin & Mikolov , 2015 ) , and thus in providing a large scale training and evaluation framework for such models , we hope to provide a solid foundation upon which to continue such research into machine reasoning beyond mathematics . Question : Solve -42 * r + 27 * c = -1167 and 130 * r + 4 * c = 372 for r. Answer : 4 Question : Calculate -841880142.544 + 411127 . Answer : -841469015.544 Question : Let x ( g ) = 9 * g + 1 . Let q ( c ) = 2 * c + 1 . Let f ( i ) = 3 * i - 39 . Let w ( j ) = q ( x ( j ) ) . Calculate f ( w ( a ) ) . Answer : 54 * a - 30 Question : Let e ( l ) = l - 6 . Is 2 a factor of both e ( 9 ) and 2 ? Answer : False Question : Let u ( n ) = -n * * 3 - n * * 2 . Let e ( c ) = -2 * c * * 3 + c. Let l ( j ) = -118 * e ( j ) + 54 * u ( j ) . What is the derivative of l ( a ) ? Answer : 546 * a * * 2 - 108 * a - 118 Question : Three letters picked without replacement from qqqkkklkqkkk . Give prob of sequence qql . Answer : 1/110 Figure 1 : Examples from the dataset . 1.1 OUR CONTRIBUTIONS . Dataset and generalization tests We release1 a sequence-to-sequence dataset consisting of many different types of mathematics questions ( see Figure 1 ) for measuring mathematical reasoning , with the provision of both generation code and pre-generated questions . The dataset comes with two sets of tests : interpolation tests , one for each type of question occurring in the training set ; and extrapolation tests , that measure generalization along various axes of difficulty to beyond that seen during training . We include extrapolation tests as an additional measure of whether models are employing abilities that allow them to algebraically generalize . Experiments and model analysis We perform an experimental evaluation to investigate the algebraic abilities of state-of-the-art neural architectures , and show that they do well on some types of questions , but certainly not all , and furthermore have only moderate amounts of generalization . We give some insights into how they learn to answer mathematics questions , and their failure modes . 1Dataset will be available at https : //github.com/deepmind/mathematics_dataset 1.2 RELATED WORK . There are various papers with datasets with a discrete reasoning nature . Kaiser & Sutskever ( 2015 ) use an adapted convolutional architecture to solve addition and multiplication with good generalization ; Allamanis et al . ( 2016 ) and Evans et al . ( 2018 ) use tree networks to predict polynomial or logical equivalence or logical entailment ; Selsam et al . ( 2018 ) uses message passing networks with a bipartite graph structure to decide satisfiability in formulas in conjunctive normal form , and so on . The difference between those problems and the dataset in this paper is that the former all have a single well-defined input structure that can be easily mapped into narrow architectures suited to the problem structure , avoiding the need for general reasoning skills like parsing or generic working memory . Zaremba & Sutskever ( 2014 ) analyze the ability of LSTMs to map short Python programs ( addition or for-loops ) to their output . Some mathematics problems are of a similar imperative nature ( e.g . arithmetic ) , but we also cover many other types of problems , so our dataset subsumes learning-to-execute . There are a few other synthetically generated datasets designed to assess reasoning of some form . The bAbI dataset of Weston et al . ( 2015 ) consists of textual questions , testing the ability to extract knowledge from a story-like sequence of questions . The CLEVR dataset of Johnson et al . ( 2017 ) consists of image-question pairs , where the image is of a set of objects , and the question asks for some property of the scene ; this dataset is designed to assess visual analysis . Santoro et al . ( 2018b ) use Raven ’ s progressive matrix puzzles to measure abstract reasoning of networks . There has also been a recent interest in solving algebraic word problems . These questions tend to be crowd sourced or obtained from exercise books , and existing datasets include Allen Institute for AI ( 2014 ) ; Kushman et al . ( 2014 ) ; Huang et al . ( 2016 ) ; Upadhyay & Chang ( 2016 ) ; Wang et al . ( 2017 ) ; Ling et al . ( 2017 ) . These range in size from hundreds to up to one hundred thousand examples , with different variations and focuses ; for example , containing supervised “ answer rationale '' , or focusing on more narrow types of problems , or additionally containing geometry problems ( although some of these are too small to train deep learning models without extensive prior mathematical knowledge ) . Our dataset differs from these in that our focus is mathematical reasoning rather than linguistic comprehension ; we cover more areas of mathematics , but with less variation in problem specification , and we see mathematical reasoning as a partially orthogonal and complementary direction to linguistic understanding existing in these other datasets . 2 THE DATASET . 2.1 DESIGN CHOICES . Modular structure and procedural generation There are two choices for obtaining mathematical questions : either crowd-sourced , or synthetically generated . While crowd-sourcing has the advantage of introducing linguistic diversity , as well as a diversity of problem types , it is difficult to collect and validate such data at scale . In contrast , procedural generation is sufficient for our purposes in most respects : it ( 1 ) easily provides a larger number of training examples , with ( 2 ) precise controls over difficulty levels , permitting ( 3 ) analysis of performance by question type , and ( 4 ) better guarantees on question correctness , with ( 5 ) potential for more efficient model training by varying the time spent on each module , and ( 6 ) ease of testing generalization ( since one can precisely vary different axes of difficulty in different question types ) . Freeform question/answers Given that we synthetically generate the data , we could of course provide the questions as parsed into some structure appropriate for each question type ( e.g . a tree or graph ) . However , we opt for freeform—as a sequence of characters—because ( 1 ) it is a powerful and flexible format , allowing us to express many question types ( whereas trees or graphs are only appropriate for some problems ) , ( 2 ) the ability to properly semantically parse is a non-negligible part of cognition , and ( 3 ) sequences are much simpler objects than graphs and trees , which simplifies development of the dataset and models . Perhaps most importantly , using freeform inputs and outputs means that the input and output space for models evaluated on the benchmark tasks in this dataset is the same as required to address a variety of “ real world ” mathematics exams questions . While it is not plausible that models trained on our data would perform well on such actual tests due to restricted linguistic variation in how questions and answers are formulated , it is nonetheless a desirable feature of our data that future models which do attack real world tests can be “ unit tested ” on our benchmarks during their development . Compositionality The questions can be seen as mappings with input and output types . For example , function evaluation maps a function and an integer to another integer , function composition maps a pair of functions to a function , and so on . We use this to generate additional composed questions by chaining modules with matching types , where intermediate values from one sub-problem are used as inputs to the next sub-problem . For example , for a single intermediate value , this composition may be phrased as Let x = < description > . < question ( x ) > . See Figure 1 for examples . This makes the dataset more interesting and challenging in several ways . Many rules in mathematics appear when different concepts are composed . For example , when differentiation is composed with function composition , the chain rule appears ; when addition is composed with factorization , distributivity can emerge ; and so on . Composition moves the questions away from pure perception , since intermediate results must be stored ( working memory ) and manipulated ( reuse of sub-routines ) .

Summary: This paper presents a new synthetic dataset to evaluate the mathematical reasoning ability of sequence-to-sequence models. It consists of math problems in various categories such as algebra, arithmetic, calculus, etc. The dataset is designed carefully so that it is very unlikely there will be any duplicate between train/test split and the difficulty can be controlled. Several models including LSTM, LSTM + Attention, Transformer are evaluated on the proposed dataset. The result showed some interesting insights about the evaluated models. The evaluation of mathematical reasoning ability is an interesting perspective. However, the un-standard design of the LSTM models makes it unclear whether the comparisons are solid enough.