## 3 Neural Network Models for Arithmetic Reasoning
### Generic Large Language Models  
One of the most exciting discoveries of the past few years has been to realize that large language models (LLMs) trained in an autoregressive way can exhibit a surprising level of competence in a variety of tasks \"out of the box\", that is, without having been explicitly trained to solve such problems. This also seems to be the case for numerical reasoning; for example, GPT-3 is able to carry out basic arithmetic tasks such as two-digit addition without any additional fine-tuning, with performance becoming progressively stronger moving from the zero-shot to one-shot to few-shot setting [69]. However, the numerical knowledge of LLMs is often superficial and inaccurate: the calculation accuracy of GPT-3 rapidly decreases as the number of digits increases [69] and further growing the model size does not necessarily improve the extrapolation capabilities [70; 71]; the text-to-text T5 model fails in simple numerical tasks, such as number comparison and number sorting, when probed outside the interpolation range [72]; and the mathematical performance of the popular ChatGPT model is significantly below that of an average mathematics graduate student [51]. In general, even the largest and most recent models falter when required to perform multi-step mathematical reasoning, such as those included in the MATH benchmark. It has also been shown that the performance of LLMs on mathematical calculations strongly correlates with term frequency in the training data [73], suggesting that these gigantic models might obtain a seemingly high performance from surface-level memorization, rather than an understanding of arithmetic procedures.  
One possibility to improve the numerical reasoning in LLMs is to fine-tune them using domain-specific datasets. For example, GPT-style models can solve many of the problems in the Mathematics benchmark once trained on math datasets [70], and fine-tuned BERT, T5, or LLaMA models also achieve significant improvements on more challenging benchmarks [74; 75; 76; 77]. Yet, fine-tuned LLMs still fail, even in simple numerical tasks that probe extrapolation capabilities, and sometimes performance is even worse than the original architectures [72].  
Performance can also be improved by combining generative LLMs with post-processing verifiers, which are trained to judge the correctness of model-generated solutions: at test time, a fixed number of candidate solutions is sampled and the one that is ranked highest by the verifier is selected as the final output [37]. In this specific case, the system was also further trained to rely on a calculator for solving arithmetic problems, by injecting calculation annotations into the training set; at test time, the calculator overrides sampling when the model produces the annotations. The idea of producing annotations that can be subsequently processed by a software calculator is also pursued in more sophisticated approaches that train LLMs to generate computer code to solve complicated algorithmic tasks, rather than directly asking for the solution. In one recent demonstration of this method [23], the authors exploited an LLM pre-trained on text and fine-tuned on computer code [78] to generate computer programs that solve challenging (university-level) problems from the MATH benchmark. The performance gain was significant, though it should be emphasized that in these cases, the task of producing the final solution is partially delegated to an external tool (e.g., a Python interpreter), which can also take advantage of numerical libraries to perform mathematical operations such as solving equations or taking limits.