A Survey of Deep Learning for Mathematical Reasoning
Year: 2023

Abstract:
Mathematical reasoning is a fundamental aspect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems in language has garnered significant interest in the fields of machine learning and natural language processing. For example, mathematics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and modeling advances. On the other hand, recent advances in large-scale neural language models have opened up new benchmarks and opportunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the intersection of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain.

heading: 1 Introduction
text: “The study of mathematics, like the Nile, begins in minuteness but ends in magnificence.”
— Charles Caleb Colton, English writer
Mathematical reasoning is a key aspect of human intelligence that enables us to comprehend and make decisions based on numerical data and language. It is applicable in various fields, including science, engineering, finance, and everyday life, and encompasses a range of abilities, from basic skills such as pattern recognition and numerical operations to more advanced skills like problemsolving, logical reasoning, and abstract thinking. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems in language has been a long-standing focus of research in the fields of machine learning and
∗denotes co-senior authors.
natural language processing (NLP), dating back to the 1960s (Feigenbaum et al., 1963; Bobrow, 1964). In recent years, there has been a surge of interest in this area: for instance, the number of papers has grown from approximately 10 in 2018 to 66 in 2022 (see Figure 3 in the Appendix).
As deep learning continues to revolutionize NLP tasks such as question answering and machine translation (Sutskever et al., 2014; Devlin et al., 2019), it has also made significant strides in the field of mathematical reasoning (Wang et al., 2017; Yang and Deng, 2019; Geva et al., 2020; Wei et al., 2022). However, despite the impressive capabilities of these models, there is still a lack of a clear taxonomy of the different types of mathematical reasoning tasks and the specific capabilities required of deep learning models to solve them.
Previous literature has been limited to the discussion of specific aspects, such as solving math word problems (Bhattacharya, 2017; Zhang et al., 2019; Ughade and Kumbhar, 2019), representing numbers representation (Thawani et al., 2021), or solving informal problems (Meadows and Freitas, 2022). Additionally, with the recent advancements in large language models like GPT-3 (Brown et al., 2020), there is a growing need to understand the capabilities and limitations of these models in the context of mathematical reasoning. This is where a comprehensive survey of this rapidly advancing domain becomes crucial, as it can provide an overview of the current state and limitations of the field, and indicate further research areas.
In this paper, we survey over 180 papers from the NLP and AI communities in the field of deep learning for mathematical reasoning. We study various types of mathematical reasoning problems, such as math word problems, theorem proving, geometry problem solving, math question answering, and other quantitative problems (§2, §A). Additionally, we explore different deep learning architectures for mathematical reasoning, including neural networks
14605
(§3), pre-trained language models (§4), and recent in-context learning for large language models (§5).
We also analyze existing benchmarks and find that there is less focus on multi-modal and lowresource settings (§6.1). Our evidence-based studies suggest that current numeracy representations are insufficient and deep learning methods are inconsistent for mathematical reasoning (§6.2). Following this, we suggest future research directions related to generalization and robustness, trustworthy reasoning, learning from feedback, and multimodal mathematical reasoning (§7).

heading: 2 Mathematical Reasoning Tasks
text: In this section, we briefly introduce different tasks for mathematical reasoning. A detailed summary and discussion of commonly used datasets can be found in Table 7 and Appendix A. Math Word Problem Solving. Developing algorithms to automatically solve math word problems (MWPs) has been of interest to NLP researchers for decades (Feigenbaum et al., 1963; Bobrow, 1964). An example of a MWP is shown in Table 1. A question involves four basic arithmetic operations with single or multiple operation steps. The challenge posed by MWPs lies in the need for language com-
prehension, semantic parsing, and the application of multiple mathematical reasoning skills. Theorem Proving. Automating theorem proving is a long-standing challenge in AI (Newell et al., 1957; Feigenbaum et al., 1963). The problem is to demonstrate the truth of a mathematical claim (a theorem) through a sequence of logical arguments (a proof ). Theorem proving tests various skills, such as choosing effective multi-step strategies, using background knowledge, and performing symbolic manipulations. Geometry Problem Solving. Automated geometry problem solving (GPS) is also a long-standing mathematical reasoning task (Gelernter et al., 1960; Wen-Tsun, 1986). As shown in Figure 2, a geometry problem consists of a textual description and a diagram. The multimodal inputs describe the entities, attributes, and relationships of geometric elements, and the goal is to find the numeric solution to an unknown variable.
Math Question Answering. There is a wide range of question answering (QA) benchmarks that center around mathematical reasoning, which we refer to as math question answering (MathQA). For example, DROP (Dua et al., 2019) is a MathQA dataset that requires discrete reasoning to answer questions such as “Which kicker kicked the most field goals?” over the content of paragraphs.

heading: 3 Neural Networks for Mathematical Reasoning
text: Neural networks have become a popular tool in the field of mathematical reasoning, mirroring their success in NLP. In recent years, a number of different neural network architectures have been proposed for mathematical reasoning tasks, including Seq2Seq-based networks, graph-based networks, and attention-based networks. These methods are outlined in more detail in Table 8 in the Appendix.

heading: 3.1 Seq2Seq-based Networks for Math
text: Sequence-to-sequence (Seq2Seq) (Sutskever et al., 2014) neural networks have been successfully applied to mathematical reasoning tasks, such as math word problem solving (Wang et al., 2017), theorem proving (Yang and Deng, 2019), geometry problem solving (Robaidek et al., 2018), and math question answering (Tafjord et al., 2019). A Seq2Seq model uses an encoder-decoder architecture and usually formalizes mathematical reasoning as a sequence generation task. The basic idea behind this approach is to map an input sequence (e.g. a mathematical problem) to an output sequence (e.g. an equation, program, and proof). Common encoders and decoders include Long Short Term Memory network (LSTM) (Hochreiter and Schmidhuber, 1997), Gated Recurrent Unit (GRU) (Cho et al., 2014), and their bidirectional variants: BiLSTM and BiGRU. A large amount of work has shown the performance advantage of Seq2Seq models over previous statistical learning approaches (Ling et al., 2017; Wang et al., 2018a; Huang et al., 2018; Wang et al., 2019; Li et al., 2019).

heading: 3.2 Graph-based Networks for Math
text: Seq2Seq approaches show their advantages of generating mathematical expressions without relying on hand-crafted features. It is noteworthy that mathematical expressions can be represented as tree-based structures, such as abstract syntax trees (ASTs) and graph-based structures, which capture the structural information in the expressions. However, Seq2Seq methods do not explicitly this important information. To address this limitation, graph-based neural networks have been developed to explicitly model the structure within expressions. Sequence-to-tree (Seq2Tree) models explicitly model the tree structure when encoding the output sequences (Xie and Sun, 2019; Wu et al., 2020; Zaporojets et al., 2021; Qin et al., 2021). For example, Liu et al. (2019a) devise a Seq2Tree model to better use information from an equation’s AST. Seq2DAG (Cao et al., 2021), instead, applies a sequence-to-graph (Seq2Graph) framework when generating the equations since the graph decoder is able to extract complex relationships among multiple variables. The graph-based information can also be embedded when encoding the input mathematical sequences (Zhang et al., 2020b; Shen and Jin, 2020; Li et al., 2020b; Wu et al., 2021a).

heading: 3.3 Attention-based Networks for Math
text: The attention mechanism has been successfully applied to NLP (Bahdanau et al., 2015) and vision problems (Xu et al., 2015; Woo et al., 2018), taking into account the hidden vectors of the inputs during the decoding processing. Recently, researchers have been exploring its usefulness in mathematical reasoning tasks, as it can be used to identify the most important relationships between mathematical concepts. For instance, MATH-EN (Wang et al., 2018a) is a math word problem solver which benefits from long-distance dependency information learned by self-attention. Attention-based methods have also been applied to other mathematical reasoning tasks such as geometry problems solving (Robaidek et al., 2018; Chen et al., 2021a) and theorem proving (Yang and Deng, 2019). Various attention mechanisms have been studied to extract better representations, such as Group-ATT (Li et al., 2019) which uses different multi-head attention to extract various types of MWP features, and graph attention which is applied to extract knowledgeaware information in (Wu et al., 2020).

heading: 3.4 Other Neural Networks for Math
text: Deep learning approaches to mathematical reasoning tasks can also make use of other neural networks, such as convolutional neural networks (CNN) and multimodal networks. Some work encodes the input text using a convolutional neural network architecture, giving the model the ability to capture long-term relationships between symbols in the input (Gehring et al., 2017; Wang et al., 2018a,a; Robaidek et al., 2018; Alemi et al., 2016; Loos et al., 2017). For example, the first application of deep neural networks for theorem proving is proposed in (Alemi et al., 2016), which relies on convolutional networks for premise selection.
Multimodal mathematical reasoning tasks, such as geometry problem solving and diagram-based mathematical reasoning, are formalized as visual question answer (VQA) problems (Kafle et al., 2018; Chen et al., 2021a; Lu et al., 2021b). In this domain, visual inputs are encoded using ResNet (He et al., 2016) or Faster-RCNN (Ren et al., 2015), while textual representations are obtained via GRU or LTSM. Subsequently, the joint representation is learned using multimodal fusion models, such as BAN (Kim et al., 2018), FiLM (Perez et al., 2018), and DAFA (Gao et al., 2019).
Other deep neural network structures can also be used in mathematical reasoning. A Graph Neural Network (GNN) is employed for geometry problem parsing in Zhang et al. (2022), taking advantage of its success in spatial reasoning. WaveNet has been applied to theorem proving (Loos et al., 2017; Bansal et al., 2019), due to its ability to address longitudinal time-series data. Furthermore, Transformers are found to outperform GRU in generating mathematical equations in DDT (Meng and Rumshisky, 2019). Finally, MathDQN (Wang et al., 2018b) is the first work to explore reinforcement learning for math word problem solving, taking advantage of its strong search capabilities.

heading: 4 Pre-trained Language Models for Mathematical Reasoning
text: Pre-trained language models (Devlin et al., 2019; Radford et al., 2020; Brown et al., 2020) have demonstrated remarkable performance gains on a wide range of NLP tasks. By pre-training on a large corpus of text, the models learn valuable world knowledge (Guu et al., 2020), which could be applied to downstream tasks. Similar ideas can be applied to math-related problems, and previous
work has shown the promising performance of pretrained language models in answering math word problems (Kim et al., 2020), assisting with theorem proving (Wu et al., 2022b), as well as solving other mathematical tasks (Charton, 2022).
However, though large language models excel in modeling natural language, there are several challenges to using them for mathematical reasoning. First, pre-trained language models are not specifically trained on mathematical data. This likely contributes to them being less proficient in math-related tasks compared to natural language tasks. There is also less mathematical or scientific data available for large-scale pre-training compared to text data. Second, the size of pre-trained models continues to grow, making it expensive to train the entire model from scratch for specific downstream tasks. Additionally, downstream tasks may deal with different input formats or modalities, such as structured tables (Zhao et al., 2022) or diagrams (Lu et al., 2021b). To address these challenges, researchers have to adjust pre-trained models by finetuning them on downstream tasks or adapting the neural architectures.

heading: 4.1 Self-Supervised Learning for Math
text: Self-supervised learning is a machine learning approach in which an algorithm learns to perform a task without being explicitly provided with labeled training data. Table 2 provides a list of language models pre-trained with self-supervised tasks for mathematical reasoning. Model scale. There is a clear trend that pre-trained language models have become increasingly larger in the past few years (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Radford et al., 2020; Brown et al., 2020). A recent study (Liang et al., 2022a) shows that model scale within a model family reliably predicts model accuracy. The study also mentions an interesting thresholding effect: “all models that win head-to-head model comparisons for accuracy at a rate well above chance are at least 50B parameters”. A similar size-growing trend can be observed in the field of mathematical reasoning with pre-trained language models. For example, MWP-BERT (Liang et al., 2022b) uses a backbone of BERT (110M) (Devlin et al., 2019) and RoBERTa (123M) (Liu et al., 2019b) for Math Word Problems. Most recently, Minerva (Lewkowycz et al., 2022), which is based on the PaLM (Chowdhery et al., 2022) pre-trained
language model, has a size up to 540B parameters.
Pre-training corpus. There are generally two types of pre-training corpus for mathematical language models. (i) Curated datasets from openly accessible sources. For example, Hendrycks et al. (2021b) present the first large-scale mathematics pre-training dataset with step-by-step solutions in natural language and LATEX, called the Auxiliary Mathematics Problems and Solutions (AMPS). AMPS consists of Khan Academy and Mathematica data. Minerva (Lewkowycz et al., 2022) collects a high-quality dataset containing scientific and mathematical data, which contains 38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server. Thor (Jiang et al., 2022b) pre-trains a language model on the GitHub + arXiv subsets of The Pile (Gao et al., 2020). (ii) Synthetic datasets based on templates or interaction with engines. Recent work (Wu et al., 2021d; Krishna et al., 2021; Ri and Tsuruoka, 2022; Anderson and Farrell, 2022; Wu et al., 2022c) shows that pre-training on data that is fully synthetically generated—synthetic pre-training can actually provide substantial gains. Representative work includes TaPEX (Liu et al., 2022b), which obtains a pre-training corpus by automatically synthesizing executable SQL queries and their execution outputs. LISA (Jiang et al., 2021) extracts lemmas and theorems by interacting with the Isabelle standard library and the Archive of Formal Proofs. GenBERT (Geva et al., 2020) generates numerical and textual pre-training datasets based on manually crafted and extracted templates.
Pre-training tasks. General pre-training language models have two typical self-supervised learning tasks: (i) Masked Language Modeling (MLM), where it randomly masks a portion of words in each sequence to predict the outcome; (ii) Causal Lan-
guage Modeling (CLM), where the model is trained to predict the next token in a sequence of tokens. Following the same paradigm, researchers pre-train language models with MLM and CLM tasks on mathematical or scientific corpora for downstream tasks (Polu and Sutskever, 2020; Hendrycks et al., 2021b; Han et al., 2022; Jiang et al., 2022b).
There is also recent work that designs customized tasks to inject mathematical reasoning capabilities into language models. For instance, Liang et al. (2022b) pre-train language models with a suite of 8 numeracy-augmented tasks with consideration of reasoning logic and numerical properties. LIME (Wu et al., 2021d) proposes synthetic pretraining tasks to learn three reasoning primitives: deduction, induction, and abduction before learning more complex reasoning skills, which also be regarded as a form of curriculum learning.

heading: 4.2 Task-specific Fine-tuning for Math
text: Task-specific fine-tuning is a technique to improve the performance of a pre-trained language model on a specific task. This is also a common practice when there is not enough data for training the large models from scratch. As shown in Table 3, existing work fine-tunes pre-trained language models on a variety of downstream tasks, such as math word problems (Kim et al., 2020; Shen et al., 2021), MathQA (Zhao et al., 2022), geometry problem solving (Lu et al., 2021a), linear algebra (Charton, 2022), and theorem proving (Welleck et al., 2022a). Apart from fine-tuning the model parameters, some work also uses pre-trained language models as encoders and ensembles them with other modules for downstream tasks (Lu et al., 2021b).

heading: 5 In-context Learning for Mathematical Reasoning
text: Large language models (LLMs), such as GPT3 (Brown et al., 2020), have recently revolutionized the field of natural language processing (NLP), especially on account of their powerful few-shot incontext learning capabilities (Brown et al., 2020). In-context Learning (ICL) enables LLMs to perform target tasks by providing some task examples as conditions at inference time, without updating model parameters (Radford et al., 2020; Brown et al., 2020). ICL allows users to quickly build models for new use cases without worrying about fine-tuning and storing a large amount of new parameters for each task, so it is widely used in fewshot settings nowadays (Min et al., 2022).
An in-context example typically contains an input-output pair with some prompt words, e.g., Please select the largest number from the list. Input: [2, 4, 1, 5, 8]. Output: 8, and few-shot works by giving multiple examples, and then a final input example, where the model is expected to predict the output. However, such standard few-shot promptings, in which the LLM is given in-context examples of input–output pairs in front of test-time examples, have not yet proved sufficient to achieve high performance on challenging tasks such as mathematical reasoning (Rae et al., 2021).
Chain-of-thought prompting (CoT) (Wei et al., 2022) leverages intermediate natural language rationales as prompts to enable LLMs to first generate reasoning chains and then predict an answer for an input question. For example, a CoT prompt for solving the math word problem could be
Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. Then, how many tennis balls does Roger have now? Answer: Roger started with 5 balls. 2 cans of 3 tennis balls each are 6 tennis balls. 5 + 6 = 11. The answer is 11.
Apart from Kojima et al. (2022) showing that LLMs are decent zero-shot reasoners when given the “Let’s think step by step!” prompt, most of the recent work has focused on how to improve chainof-thought reasoning under the few-shot setting. This work is mainly divided into two parts, (i) selecting better in-context examples and (ii) creating better reasoning chains.

heading: 5.1 In-context Example Selection
text: Early chain-of-thought work randomly or heuristically selects in-context examples. However, recent studies have shown that this type of few-shot learning can be highly unstable across different selections of in-context examples (Rubin et al., 2022; Liu et al., 2022a). Therefore, which incontext reasoning examples make the most effective prompts is still an unknown problem in the literature. To address the limitation, recent work has investigated various methods to optimize the in-context examples selection process (Rubin et al., 2022; Zhang et al., 2023; Lu et al., 2022b; Yu et al., 2023; Fu et al., 2023). For example, Rubin et al. (2022) attempt to address this issue by retrieving semantically similar examples. In addition, Fu et al. (2023) propose complexity-based prompting, which chooses examples with complex reasoning chains, i.e., chains with more reasoning steps, as the prompt. PromptPG (Lu et al., 2022b) learns to select optimal in-context examples via reinforcement learning (RL) from a candidate pool.

heading: 5.2 High-quality Reasoning Chains
text: Early chain of thought work (e.g., Wei et al. (2022)) mainly relies on a single human-annotated reasoning chain as a prompt. However, manually creating reasoning chains has two disadvantages. First, as tasks become more complex, current models may not be sufficient to learn to perform all necessary reasoning steps and cannot easily generalize to different tasks. Second, a single decoding process is vulnerable to incorrect inference steps, leading to an incorrect prediction as the final answer. To address this limitation, recent studies mainly fo-
cus on two aspects, (i) hand-crafting more complex demonstrations, which we refer to as process-based approaches (Zhou et al., 2023; Chen et al., 2022b), (ii) leveraging ensemble-like methods, which we refer to as outcome-based approaches (Wang et al., 2023; Li et al., 2022a).
Process-based approaches aim to improve the chain-of-thought reasoning quality, especially for complex reasoning tasks. In least-to-most prompting (Zhou et al., 2023), the problem-solving process is implemented through two-stage prompting: (i) reducing a complex problem into a list of subproblems; (ii) solving these sub-problems sequentially, so that solving a given sub-problem is facilitated by the answers to previously solved subproblems. Similarly, Khot et al. (2022) leverage diverse decomposition structures and use different prompts to answer each sub-question. Apart from these multi-step reasoning methods, Chen et al. (2022b); Gao et al. (2022) propose programof-thoughts (PoT), an alternative solution that uses large language models to express the reasoning process as a program. The computation is then relegated to an external computer, which executes the generated programs to derive the answer. A more recent work, Chameleon (Lu et al., 2023), integrates different tools to enhance the abilities of LLMs for compositional reasoning.
Outcome-based approaches acknowledge the potential incorrectness of an individual reasoning path, and instead use multiple reasoning paths (Wang et al., 2023; Li et al., 2022a). Selfconsistency (Wang et al., 2023) generates a set of reasoning paths by sampling from the language model, and marginalizes out the reasoning paths by choosing the most common answer. In addition to using sampling with a single prompt to produce multiple reasoning paths, Li et al. (2022a) propose to introduce diverse prompts through “selfteaching”, as a complementary solution to produce
a higher degree of diversity.

heading: 6 Discussion and Findings

heading: 6.1 Analysis of Benchmarks
text: The multi-modal setting is underexplored but is gaining increasing attention. Most existing benchmarks for mathematical reasoning have targeted the textual-only modality. However, visual elements can provide a rich source of quantitative information, making multi-modal datasets beneficial for reasoning over quantitative relations in natural images (Lu et al., 2022a), abstract diagrams (Lu et al., 2021b), figures (Kahou et al., 2018), and charts (Kafle et al., 2018). Tables, which are commonly found in daily documents and contain hierarchically structured information, have also been the focus of tasks that require quantitative reasoning over textual and tabular context (Chen et al., 2021c; Zhu et al., 2021; Zhao et al., 2022; Lu et al., 2022b). In addition, recent datasets have been developed for mathematical reasoning grounded on conversations (Sun et al., 2019; Zhang et al., 2021; Chen et al., 2022c), as well as reports (Chen et al., 2022c).
Pioneering work is emerging in the exploration of low-resource settings. Despite the creation of various datasets, mathematical reasoning in lowresource settings remains largely under-explored. Pioneering research has developed mathematical reasoning benchmarks for financial (Chen et al., 2021c; Zhu et al., 2021; Zhao et al., 2022) and scientific domains (Lu et al., 2022a). Additionally, there have been attempts to build non-English datasets for Chinese (Wang et al., 2017; Qin et al., 2020; Yu et al., 2021a) and Arabic (Alghamdi et al., 2022) for mathematical reasoning.
Diverse rationale annotations have been widely explored. Complex reasoning usually involves multiple steps to arrive at the final answer. To bridge this gap, datasets annotated with intermediate rationales such as logic forms (Tafjord et al.,
T5 UnifiedQA GPT-3 GPT-3 (Large) (Large) (davinci-002) (davinci-003)
3 balls + 5 balls = ✗ 5 balls 8 balls 8 balls 23 balls + 145 balls = ✗ ✗ 58 balls 168 balls 23 balls + 1,855 balls = ✗ ✗ 2,878 balls 2,988 balls
Table 5: Language models struggle with large numbers.
2019; Lu et al., 2021a), programs (Amini et al., 2019; Chen et al., 2021c,a; Cao and Xiao, 2022; Chen et al., 2022a), and reasoning graphs (Zhang et al., 2021) have been proposed to train models for complex reasoning tasks. Python programs are used as reasoning annotations in (Austin et al., 2021; Mishra et al., 2022a) due to their enhanced accessibility and readability. To imitate the reasoning process of a human, a more recent trend is to annotate solutions in natural language (Ling et al., 2017; Cobbe et al., 2021; Lu et al., 2022b; Hendrycks et al., 2021b; Lu et al., 2022a).

heading: 6.2 Analysis of Deep Learning Methods
text: Is the current representation of numeracy sufficient? The standard practice for deep learning techniques is to treat numbers in the same way as words. Early neural network methods create a vocabulary that maps input words and numbers to token IDs, resulting in less frequent numbers being collapsed into an “UNK” token. Recent language models use subword tokenization techniques (Wu et al., 2016; Sennrich et al., 2016) to split numbers into atomic tokens. Recent studies have shown that these tokenization approaches are suboptimal (Wallace et al., 2019; Lin et al., 2020; Zhang et al., 2020d; Thawani et al., 2022).
Two numbers on the same or close number line could have surface forms with no shared common tokens. For example, a number like 1598 is tokenized as “15” and “98” in GPT-3, while another format like 1, 598 is split as three different tokens: “1”, “,”, and “598”. This lack of consistent representation can make it difficult for deep learning models to effectively process numbers, especially when compared to pure text. The insufficient representations of numbers can lead to out-of-distribution (OOD) problems. Table 5 provides examples of where language models tend to struggle with large numbers. Although increasing model scales could help, even the state-of-the-art large language model GPT-3 performs poorly when reasoning over large numbers. Some recent work suggests that using scientific notation (Zhang et al., 2020d) and digitlevel decomposition (Geva et al., 2020) may be helpful in improving numeracy representation, but
this remains an open problem.
Are deep learning methods consistent for mathematical reasoning? Recent developments in deep learning have led to impressive results on various mathematical reasoning tasks. The zero-shotCoT Minerva 540B achieves a score of 75.0% on the MMLU-STEM benchmark (Hendrycks et al., 2021a), which assesses multitask reasoning ability in the fields of science, technology, engineering, and mathematics (STEM) at both high school and college levels. Similarly, few-shot-CoT GPT-3 175B achieves a high accuracy of 93.0% on the MultiArith task. However, the question remains as to whether these methods are sufficiently advanced to tackle more complex problems.
There is strong evidence that deep learning methods for mathematical reasoning are not robust and susceptible to adversarial attacks (Lin et al., 2020; Patel et al., 2021; Mishra et al., 2022b,a; Welleck et al., 2022b). The SVAMP (Patel et al., 2021) dataset is a collection of one-unknown arithmetic word problems up to grade 4, with slight word variations from previous datasets. It is surprising that current state-of-the-art (SOTA) methods perform poorly on this dataset, with Graph2Tree achieving only a 43.8% accuracy and zero-shot-CoT GPT-3 (175B) only reaching 63.7%, which is just above an “F” grade. Table 6 also shows the inconsistent performance of the zero-shot GPT-3 model in scenarios with slightly different descriptions, while human performance remains unchanged. This indicates a lack of consistency in the mathematical reasoning ability of SOTA large language models.

heading: 7 Future Work

heading: 7.1 Generalization and Robustness
text: Despite impressive progress, neural models commonly display generalization and robustness fail-
ures on reasoning tasks. For example, above we discussed difficulties in generalizing to larger numbers (Table 5) or remaining robust to nearby problems (Table 6), while others identify failures in generalizing to longer problems than those observed in training (e.g., Anil et al. (2022)). One direction is to explore new inference-time (Jung et al., 2022; Mitchell et al., 2022) or fine-tuning (Anil et al., 2022) strategies.
Another aspect of generalization relates to the role of memorization. For example, is the ability to produce a complex solution dependent on seeing many similar solutions during training, or even on memorizing the solution? Term frequency in the pretraining corpus is known to impact accuracy in simple arithmetic tasks (Razeghi et al., 2022) or factual question answering (Kandpal et al., 2022). On the other hand, Lewkowycz et al. (2022) did not find evidence of memorization in complex outputs, yet their training set and model are not available for inspection. Gaining a full understanding of these factors for complex problems and outputs (e.g., multi-step solutions or proofs) requires more analysis, as well as accessible datasets and models.

heading: 7.2 Trustworthy Reasoning
text: Recent advances in language models have demonstrated their powerful capabilities for mathematical reasoning. However, due to the potential for generating ungrounded answers (Nakano et al., 2021), users can’t always trust the predicted outcomes or have to verify then with extra efforts. Even with recent prompting strategies that provide rationales before making predictions (Wei et al., 2022), language models can still hallucinate statements, produce flawed reasoning, and output wrong answers. Consequently, novel approaches that enable more reliable reasoning are needed urgently. Some potential directions for this include: (i) using language models to provide evidence, such as theorems, to support the reasoning process; (ii) incorporating a mechanism that makes a judgment when the model is unsure of the answer; and (iii) using a model itself or another module to detect and locate mistakes in a model’s reasoning.

heading: 7.3 Learning from Feedback
text: Another important direction to further improve language models for mathematical reasoning is to let the model learn from feedback. Such a process makes the continual improvement of models’ output quality and safety possible. An example is us-
ing reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) to align language models with instructions. The idea is to let humans rank the generated outputs of language models and use the learned reward function to finetune the language model with policy gradient (Ouyang et al., 2022; Glaese et al., 2022; Qiu et al., 2022a). In the context of mathematical reasoning, feedback does not necessarily come from humans directly. The outcome of a theorem-proof engine (Jiang et al., 2021; Wu et al., 2021d, 2022c) or the execution result of model-generated scripts can also be used as the reward source (Polu and Sutskever, 2020).

heading: 7.4 Multi-modal Mathematical Reasoning
text: In recent years, there has been growing interest in multi-modal mathematical reasoning, which involves using multiple sources of information, such as text, tables, natural images, and diagrams (Kahou et al., 2018; Kafle et al., 2018; Lu et al., 2021b, 2022b). However, currently available datasets in this domain tend to be small (Zhao et al., 2022), generated from templates (Kahou et al., 2018), or focus on specific topics (Lu et al., 2021a; Chen et al., 2022a). One line of current research involves applying VQA-based frameworks to analyze figures and plots, but this approach can result in significant semantic gaps due to the fact that most VQA models are trained on natural images. One potential direction for future work is to enhance the ability of multi-modal mathematical reasoning systems to tackle more complex and realistic problems. This may involve creating unified models for interpreting and integrating different modalities, as well as developing better evaluation benchmarks to assess the performance of these systems.

heading: 8 Conclusion
text: In this paper, we present a comprehensive survey of deep learning for mathematical reasoning. We review the various tasks, datasets, and deep learning approaches. We also identify several gaps in the existing datasets and methods. Finally, we outline directions for future research and highlight the potential for further exploration in this field. Our goal with this paper is to provide a comprehensive and useful resource for readers interested in the development of deep learning for mathematical reasoning. To aid in this effort, we have created a reading list that will be continually updated in a GitHub repository at https://github.com/lupantech/dl4math.
Limitations
One limitation of our survey work is that it is focused on the intersection of mathematical reasoning and deep learning over the past decade, which may not encompass the entire field and its history. Additionally, our evaluation of existing benchmarks and methods is based on a curated set of papers and may not fully represent the state of the art in the field. Furthermore, due to the fast-paced nature of the field, our survey may not reflect the latest developments and advancements which may have come out close to or after the survey was conducted. Despite these limitations, our survey still provides a valuable overview of the current state and key trends in the field of mathematical reasoning and deep learning, and can serve as a valuable resource for researchers and practitioners working in this field.
Broader Impact
Our survey paper on the intersection of mathematical reasoning and deep learning has the potential to significantly impact the field of artificial intelligence. By providing a comprehensive overview of the key tasks, datasets, and methods that have been developed in the past decade, we give researchers and practitioners a clear understanding of the current state-of-the-art and help them make informed decisions about their own research. Additionally, by evaluating existing benchmarks and methods and discussing future research directions, we aim to identify gaps in the current state of the art and guide future research and development efforts towards more advanced and effective mathematical reasoning systems. Overall, our survey has the potential to contribute to the advancement of mathematical reasoning and deep learning, and have a profound impact on machine learning and natural language processing.

heading: A Mathematical Reasoning Datasets
text: In this section, we will examine the various datasets currently available for the study of mathematical reasoning using deep learning methods. A summary of the commonly used datasets in this field can be found in Table 7.
A.1 Math Word Problem Solving
Developing algorithms to solve math word problems (MWPs) automatically has been an interest of NLP researchers for decades (Feigenbaum et al., 1963; Bobrow, 1964). A math word problem (also termed an algebraic or arithmetic word problem) describes a brief narrative that involves characters, entities, and quantities. The mathematical relationship of an MWP can be modeled with a set of equations whose solution reveals the final answer to the question. A typical example is shown in Table 1. A question involves the four basic arithmetic operations of addition, subtraction, multiplication, and division with single or multiple operation steps. The challenge of MWPs for NLP systems lies in the need for language comprehension, semantic parsing, and multiple mathematical reasoning skills.
Existing MWP datasets cover grade school problems, which are crawled from online learning websites (Koncel-Kedziorski et al., 2015), collected from textbooks, or manually annotated by human workers (Patel et al., 2021). Early math word problem datasets are relatively small or limited to a small number of operation steps (Hosseini et al., 2014; Kushman et al., 2014; Roy et al., 2015). Some recently curated datasets aim to increase problem diversity and difficulty levels. For example, Ape210K (Zhao et al., 2020) consists of 210k elementary math word problems, which is the largest publicly available. The problems in GSM8K (Cobbe et al., 2021) can involve up to 8 steps to
solve. SVAMP (Patel et al., 2021) is a benchmark that tests the robustness of deep learning models to math word problems with simple variations. More recently built datasets involve modalities beyond text. For example, IconQA (Lu et al., 2021b) provides an abstract diagram as a visual context, while TabMWP (Lu et al., 2022b) provides a tabular context for each problem.
Most MWP datasets provide annotated equations as a rationale for the solution (e.g., Table 1). To improve the performance and interpretability of the learned solvers, MathQA (Tafjord et al., 2019) is annotated with precise operation programs, and MathQA-Python (Austin et al., 2021) is provided with specific Python programs instead. Another line of datasets annotates the problems with multistep natural language solutions that are regarded as more human-readable (Ling et al., 2017; Cobbe et al., 2021; Lu et al., 2022b). Lila (Mishra et al., 2022a) annotates many of the previously mentioned MWP datasets with Python program rationales.
A.2 Theorem Proving
Recently, there has been increased interest in using language models for theorem proving in formal interactive theorem provers (ITP) (e.g., Polu and Sutskever (2020); Han et al. (2022); Polu et al. (2023); Jiang et al. (2022b,a); Lample et al. (2022)). Example ITPs include Lean (Moura et al., 2015), Isabelle (Paulson, 1994), Coq (Barras et al., 1999), and Metamath (Megill and Wheeler, 2019). To prove a theorem in an ITP, the theorem is stated in the ITP’s programming language, then simplified by generating “proof steps” until it is reduced to known facts. The result is a sequence of steps that constitutes a verified proof.
Data sources for neural theorem proving in ITPs include interactive learning environments that interface with ITPs, and datasets derived from proofs in ITP libraries. For example, CoqGym (Yang and Deng, 2019) provides an interactive environment and 71K human-written proofs for the Coq ITP. For Isabelle, PISA (Jiang et al., 2021) enables interaction and provides a dataset of 183k proofs mined from the Isabelle standard library and Archive of Formal Proofs. For Lean, LeanStep (Han et al., 2022) provides a dataset of proof-steps from Lean’s mathematical library along with auxiliary tasks, while Lean-Gym (Polu et al., 2023) provides an interactive REPL. The miniF2F (Zheng et al., 2022) benchmark aims to provide a shared benchmark
across ITPs, consisting of 488 problem statements sourced from mathematical competitions.
Other resources provide proxy environments or tasks. For example, INT (Wu et al., 2021c) provide a synthetic proving environment to measure six different types of generalization. Li et al. construct IsarStep using the Isabelle Archive of Formal Proofs, and propose a task of filling in a missing intermediate proposition. Early applications of deep learning for formal theorem proving focus on selecting relevant premises (Alemi et al., 2016).
Informal theorem proving presents an alternative medium for theorem proving, in which statements and proofs are written in the mixture of natural language and symbols used in “standard” mathematics (e.g., in LATEX), and are checked for correctness by humans. Early work focuses on selecting relevant premises (Ferreira and Freitas, 2020b,a). Welleck et al. (2021) develop NaturalProofs, a large-scale dataset of 32k informal mathematical theorems, definitions, and proofs, and provide a benchmark for premise selection via retrieval and generation tasks. Welleck et al. (2022a) adapt NaturalProofs for full proof generation, and provide a human evaluation protocol and proxy automatic metrics.
An emerging area of research aims to combine elements of informal and formal theorem proving. For example, Wu et al. (2022b) explore translating informal statements into formal statements, while Jiang et al. (2022a) release a new version of the miniF2F benchmark augmented with informal statements and proofs, which we refer to as miniF2F+informal. Jiang et al. (2022a) explore translating provided (or generated) informal proofs into formal proofs.
A.3 Geometry Problem Solving
Automated geometry problem solving (GPS) is also a long-standing AI task in mathematical reasoning research (Gelernter et al., 1960; Wen-Tsun, 1986; Chou et al., 1996; Ye et al., 2008) and has attracted much attention in recent years. Different from a math word problem, a geometry problem consists of a textual description in natural language and a geometric diagram. As shown in Figure 2, the multimodal inputs describe the entities, attributes, and relationships of geometric elements, and the goal is to find the numeric solution to an unknown variable. GPS is a challenging task for deep learning methods due to the complex skills it requires. It involves the ability to parse multimodal informa-
tion, perform symbolic abstraction, utilize theorem knowledge, and conduct quantitative reasoning.
Some early datasets are proposed to facilitate research in this domain (Seo et al., 2015; Alvin et al., 2017; Sachan et al., 2017; Sachan and Xing, 2017). However, these datasets are relatively small or not publicly available, which limits the development of deep learning methods. In response to this limitation, Lu et al. create the Geometry3K dataset, which consists of 3,002 multi-choice geometry problems with unified logic form annotations for the multimodal inputs. More recently, largerscale datasets such as GeoQA (Chen et al., 2021a), GeoQA+ (Cao and Xiao, 2022), and UniGeo (Chen et al., 2022a) have been introduced and are annotated with programs that can be learned by neural solvers and executed to obtain the final answers.
A.4 Math Question Answering
Numerical reasoning is a core ability within human intelligence and plays an important role in many NLP tasks. Aside from theorem proving and gradelevel math word problem solving, there is a wide range of question answering (QA) benchmarks that center around mathematical reasoning. In this work, we refer to these tasks as math question answering (MathQA). A large number of datasets have been presented recently. For example, QuaRel (Tafjord et al., 2019) is a dataset of diverse story questions that involve 19 different types of quantities. McTaco (Zhou et al., 2019) studies temporal commonsense problems, while Fermi (Kalyan et al., 2021) studies Fermi problems whose answers can only be approximately estimated.
Recent studies have shown that state-of-the-art mathematical reasoning systems might suffer from brittleness in reasoning, in that the models rely on spurious signals and plug-and-chug calculations in the specific dataset to achieve “satisfactory” performance (Hendrycks et al., 2021b; Mishra et al., 2022b). To address this issue, new benchmarks are proposed from various aspects. The Mathematics dataset (Saxton et al., 2020) consists of many different types of mathematics problems, covering arithmetic, algebra, probability, and calculus. The dataset allows for measuring the algebraic generalization ability of a model. Similarly, MATH (Hendrycks et al., 2021b) consists of challenging competition mathematics to measure the problemsolving ability of models in complex scenarios.
Some work incorporates tabular contexts in the
question inputs. For example, FinQA (Chen et al., 2021c), TAT-QA (Zhu et al., 2021), and MultiHiertt (Zhao et al., 2022) collect questions that require both table understanding and numeric reasoning to answer. Others, instead, present large-scale unified benchmarks for mathematical reasoning (Mishra et al., 2022b,a; Chen et al., 2023). NumGLUE (Mishra et al., 2022b) is a multi-task benchmark with the goal of evaluating the performance of models on eight different tasks. Mishra et al. 2022a push this direction further and presents Lila, which consists of 23 mathematical reasoning tasks, spanning a wide range of mathematics topics, linguistic complexity, question formats, and background knowledge requirements.
A.5 Other Quantitative Problems Numbers are an integral part of our daily lives, and we humans reason with numbers in a variety of tasks, such as understanding news, reports, elections, and markets. This has led many in the community to question whether AI systems can effectively perform quantitative reasoning in everyday scenarios. To this end, various benchmarks have been developed to evaluate the capabilities of AI systems in this area.
Diagrams, such as figures, charts, and plots, are essential media that convey large amounts of information in a concise way. FigureQA (Kahou et al., 2018), DVQA (Kafle et al., 2018), MNS (Zhang et al., 2020c), PGDP5K (Hao et al., 2022), and GeoRE (Yu et al., 2021a), are released to investigate models’ abilities to reason about quantitative relationships among entities grounded in diagrams. NumerSense (Lin et al., 2020), instead, examines whether and to what extent existing pre-trained language models can induce numerical commonsense knowledge. EQUATE (Ravichander et al., 2019) formalizes aspects of quantitative reasoning in a natural language inference framework. Quantitative reasoning can appear frequently in specific domains like finance, science, and programming. For instance, the ConvFinQA (Chen et al., 2022c) targets numerical reasoning over financial reports in a conversational question answering format. ScienceQA (Lu et al., 2022a) involves numerical reasoning in scientific domains, while P3 (Schuster et al., 2021) studies the function inference ability of deep learning models to find a valid input which makes the given program return True.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section on page 10.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations Section on page 10.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Finally, we outline directions for future research and highlight the potential for further exploration in this field. Our goal with this paper is to provide a comprehensive and useful resource for readers interested in the development of deep learning for mathematical reasoning. To aid in this effort, we have created a reading list that will be continually updated in a GitHub repository at https://github. com/lupantech/dl4math. Limitations
One limitation of our survey work is that it is focused on the intersection of mathematical reasoning and deep learning over the past decade, which may not encompass the entire field and its history. Additionally, our evaluation of existing benchmarks and methods is based on a curated set of papers and may not fully represent the state of the art in the field. Furthermore, due to the fast-paced nature of the field, our survey may not reflect the latest developments and advancements which may have come out close to or after the survey was conducted. Despite these limitations, our survey still provides a valuable overview of the current state and key trends in the field of mathematical reasoning and deep learning, and can serve as a valuable resource for researchers and practitioners working in this field