# Recall Mechanism and Multi-Head Attention for Numerical Reasoning
## 1 Introduction  
Numerical reasoning refers to the ability to understand and work with numbers, make sense of quantitative information, and perform mathematical operations. In numerical reasoning, a person or machine solves a question that typically involves the use of numerical data in a given document. Numerical reasoning encompasses a range of mathematical operations such as arithmetic, algebra, ratios, etc. While simple numerical reasoning tasks, such as math word problems (MWPs) [1], typically involve a few arithmetic operations, more complex tasks--such as those in the financial domain--require analyzing heterogeneous data sources like financial reports to answer intricate questions and the solution [2, 3]. In healthcare, numerical reasoning aids in interpreting patient data (such as medical images and clinical readings), diagnosing conditions, and making treatment decisions. Similarly, information technology may involve extracting patterns and insights from large datasets [4]. An example of numerical reasoning in the financial domain is illustrated in Figure 1. Here, the task is to determine the percentage change in cash flow hedges from 2010 to 2011 based on a given passage. Solving such problems requires identifying relevant operands and deducing the correct operators from the passage, based on the question. Even competitive models like ChatGPT-4 struggle with numerical reasoning exams [5]. FINQA is the first dataset of its kind that handles complex QA tasks based on real financial documents. It consists of 8281 question-answer pairs, collaboratively constructed by \(11\) finance professionals using earnings reports from _S&P_ 500 companies. Answering questions in the FINQA dataset involves many common financial analysis calculations, such as addition, subtraction, multiplication, division, averaging, summation, comparison and so on [2]. The encoder-decoder architecture has emerged as a widely adopted deep neural network architecture for solving numerical reasoning tasks due to its ability to derive mathematical equations from textual inputs. As depicted in Figure 2, this architecture typically employs an encoder to transform a question and a document into vector representations, which are then processed by a decoder to produce the corresponding equation. More recent models, such as FINQA [2], ELASTIC [6], and SoarGraph [7], leverage pre-trained language models like RoBERTa to encode problems and documents, significantly improving performance. In terms of the decoder, early approaches, such as the deep neural solver (DNS) [8], utilized LSTMs, while later advancements introduced tree decoders to generate prefix traversal sequences of expression trees [9]. In the financial domain, FINQA proposes a retrieval-based generator QA framework that first retrieves supporting facts from financial reports and then generates executable reasoning programs to answer questions [2]. Despite these advancements, two significant challenges persist in numerical reasoning: (1) information loss as network depth increases, leading to the gradual dissipation of embedding vectors for questions and passages, and (2) suboptimal performance in multi-step reasoning tasks, leaving substantial room for accuracy improvement. To address these issues, we propose a model with a recall mechanism and multi-head attention for numerical reasoning (RMMANR). RMMANR consists of two main components: an encoder and a decoder. The encoder leverages RoBERTa to encode the question and passage into contextual embeddings. The decoder, which consists of four modules (RM Module, Selector, MA Module, and Program Solver), generates reasoning steps and answers based on contextual information from the encoder. The main contribution of our work is three-fold:  
* We propose a recall mechanism for numerical reasoning, which refocuses on the embedding information of questions and passages before generating operators and  
Figure 1: An example of numerical reasoning. operands.