## 3 Neural Network Models for Arithmetic Reasoning
### Generic Large Language Models
#### 3.3.2 Injecting Numerical Semantics into Word Embeddings  
It turns out that the way numbers are represented in their \"surface form\" has a strong influence on the model performance in numerical reasoning tasks. Although some initial investigations suggested that standard embedding techniques could capture number semantics fairly well [87], subsequent studies pointed out that these representations are in fact inadequate at dealing precisely with numbers [88] and that commonly used sub-word tokenization techniques disrupt magnitude information [89]. For example, in GPT-3, a number like 1598 is tokenized as \"15\" and \"98\", while another format like 1,598 is split as three different tokens: \"1\", \", and \"598\".  
\begin{table}
\begin{tabular}{l l} \hline \hline USER: & What is the result of the following expression? \\  & 3 * 7 + 9 * 8 + 9 = \\ ASSISTANT: & 106 \\  & What is the result of the following expression? Please report also \\ USER: & the intermediate calculation steps. \\  & 3 * 7 + 9 * 8 + 9 = \\  & First, perform the multiplication operations: \\  & 3 * 7 = 21 \\ ASSISTANT: & 9 * 8 = 72 \\  & After that, add the results along with the 9: \\  & 21 + 72 + 9 = 102 \\  & So, the result of the expression 3 * 7 + 9 * 8 + 9 = is 102. \\ \hline \hline \end{tabular}
\end{table}
Table 3: Example of interaction with GPT-4. The model gives a wrong response when directly asked for the result of a simple arithmetic problem, while it produces the correct answer when forced to also report the intermediate calculation steps [_test performed on 1 December 2023_].  
Simple tricks to improve model performance are to adopt a character-level encoding of numbers, or to replace numbers with their corresponding scientific notation [90]. Another effective workaround consists of using more meaningful surface representations, for example, by explicitly providing the 10-base number decomposition (e.g., \"832\" becomes \"8 100 3 10 2\") [89; 91]. More advanced approaches have tried to augment the standard embeddings of number words by explicitly injecting semantic information representing quantities (for a recent survey, see [92]). One of the first attempts at learning better numeral embeddings [93] proposed to first map numbers into a log-space (to compress larger values) and train either a self-organizing map or a Gaussian mixture model with the goal of creating latent vectors encoding \"number prototypes\". A similarity function is then used to project the input number into the closest prototype. Another method forced the cosine similarity between word embeddings of two numbers to reflect their actual distance on the number line [94]. Further refinements combine the prototype-based approach with the scientific notation encoding [95]. Another interesting technique exploits the Online Encyclopedia of Integer Sequences, which includes notable series of numbers that are of particular mathematical interest, to train more meaningful number embeddings [96]. The rationale was to learn number embeddings by looking at number co-occurrence across well-structured, regular number sequences, to see if this would lead to the emergence of encodings representing useful relational properties of the integers. For example, the authors discovered that one specific neuron in the embedding vector was positively activated for even numbers and negatively activated for odd numbers, suggesting the emergence of a localistic encoding of an \"evenness\" feature. Another recent study has also explored the adequacy of different types of encoding schemes to solve a variety of linear algebra problems over matrices, concluding that the optimal encoding might be problem dependent [97].  
Overall, all these approaches for augmenting numerical embeddings allow to improve processing of small and medium-sized numbers, especially on linguistic tasks such as math word problems; however, none of them allows accurate manipulation (e.g., arithmetic) to be performed over larger quantities or numbers that are not contained in the training vocabulary.