# Recall Mechanism and Multi-Head Attention for Numerical Reasoning
## 3 Methods
### Decoder
#### 3.4.2 RM Module  
The RM Module applies a recall mechanism to refocus on the embedding information from the question and passage before generating operators and operands, ensuring the initially inputted embeddings are not lost as the model deepens. As illustrated in Figure 3,the RM Module is composed of three modules: Reasoning Module, Recall Module, and Generator Module. The Recall Module receives initial guidance vectors \(g_{initial\_op}\) from the Reasoning Module, recalls relevant information \(h^{q}\), and outputs \(g_{final\_op}\), which directs the operator generator to produce the operator \(op_{t}\). Concurrently, the Recall Module receives initial guidance vectors \(g_{initial\_oe}\) from the Reasoning Module, recalls relevant information \(h\), and outputs \(g_{final\_oe}\). Next, the operator generator pauses and passes \(g_{final\_op}\) and \(op_{t}\) to the operand generator, which generates the first operand \(oe_{t}^{1}\). Once the operator and operands of the subprogram \(r_{t}\) are completed, the RM Module then moves on to generate the the operator and operands of the next subprogram \(r_{t+1}\). After the array of mathematical operators \(OP_{RM}\) and the array of mathematical operators generate \(OE_{RM}\), they are passed to the Selector. Reasoning Module: As shown in Figure 5, the inputs for the Reasoning Module are the embedding of the previously generated symbol \(s_{t-1}\) and contextual vector \(h^{q}\) from the encoder. The Reasoning Module first calculates the context vector \(c\) by the attention weights \(a_{i}\) and the normalized vectors of \(h_{i}^{q}\):  
\[c=\sum_{i}a_{i}h_{i}^{q} \tag{3}\]  
\[a_{i}=\frac{exp(score(e_{s_{t-1}},h_{i}^{q}))}{\sum_{j}exp(score(e_{s_{t-1}}, h_{j}^{q}))} \tag{4}\]  
\[score(e_{s_{t-1}},h_{i}^{q})=e_{s_{t-1}}^{T}W_{1}W_{2}h_{i}^{q} \tag{5}\]  
where \(W_{1}\in\mathbb{R}^{h*h}\), \(W_{2}\in\mathbb{R}^{h*h}\), \(h=1024\), and both weight parameters are trainable. The encoding information from the encoder is summarized into \(c\) based on the previously generated symbol \(s_{t-1}\). Next, the Reasoning Module uses the GRU [31] to obtain the initial guidance output \(g_{initial\_op}\):  
\[g_{initial\_op},H_{t}=GRU(Relu(W_{3}[c:E_{op,cons,num}(s_{t-1})]),H_{t-1}) \tag{6}\]  
where \":\" denotes concatenation. \(Relu\) is the activation function, and \(W_{3}\in\mathbb{R}^{h*2h}\) is a trainable parameter. \(H_{t-1}\in\mathbb{R}^{h}\) represents the GRU's hidden state from the previous step, and \(H_{0}\) is initialized to zero. The Reasoning Module outputs the vector \(g_{initial\_op}\). The process of obtaining \(g_{initial\_oe}\) is the same as that of obtaining \(g_{initial\_op}\), with the main difference being that \(g_{initial\_oe}\) is obtained based on \(h\). Though \(g_{initial\_op}\) can guide the Generator Module to produce \(op\) and \(oe\). In the RM Module, \(g_{initial\_op}\) is not used directly to generate operators or operands. Recall Module: As the network deepens, the information from input questions and passages tends to dissipate in many models. Similar to how humans solve math problems by revisiting the question and passage for verification, the RM Module applies a recall mechanism to refocus on the embedding information before generating operators and operands. This ensures that important question and passage information is retained throughout the model's depth. When predicting operators, the Recall Module recalls the contextual vector \(h^{q}\). \(h^{q}\) is the key/K of the concern layer, and \(g_{initial\_op}\) is the query/Q of the concern layer.