# Recall Mechanism and Multi-Head Attention for Numerical Reasoning
## 3 Methods
### Decoder
#### 3.4.3 MA Module  
Many current models struggle with multi-step reasoning tasks [2; 6]. Multi-head attention allows the model to focus on different parts of the input sequence simultaneously. This enables parallel processing of multiple reasoning steps, which is especially beneficial for complex numerical reasoning tasks. By analyzing different possible solutions, multi-head attention optimizes the solution process for multi-step problems. As shown in Figure 4, the MA Module consists of two modules: a Multi-head Attention Reasoning Module and a Generator Module. The Multi-head Attention Reasoning Module processes the contextual embeddings, and the Generator Module generates the array of mathematical operators \(OP_{MA}\), and the array of mathematical operators generates \(OE_{MA}\).  
Multi-head Attention Reasoning Module: The input to the Multi-head Attention Reasoning Module consists of the previously generated symbol \(s_{t-1}\) and the embedded context vector \(h\) from the encoder. The reasoning module first calculates the context vector \(c\) using the multi-head attention mechanism, which allows it to effectively focus on different segments of the input. In the multi-head attention mechanism, attention weights and context vectors are calculated separately for each attention head. The specific steps are as follows:  
* Calculate the attention weights for each attention head: For the \(k\)th attention head, calculate the attention weight \(a_{i}^{(k)}\) from the previously generated symbol \(s_{t-1}\) and the context vector \(h\): \[a_{i}^{(k)}=\frac{exp(score(e_{s_{t-1}},h_{i}^{(k)}))}{\sum_{j}exp(score(e_{s_ {t-1}},h_{j}^{(k)}))}\] (14) where the scoring function for the \(k\)th attention head is: \[score(e_{s_{t-1}},h_{i}^{(k)})=e_{s_{t-1}}^{T}W_{6}^{(k)}W_{7}^{(k)}h_{i}^{(k)}\] (15) Here, \(W_{6}^{(k)}\in\mathbb{R}^{h*h}\) and \(W_{7}^{(k)}\in\mathbb{R}^{h*h}\) are trainable parameters of the \(k\)th attention head, where \(h=1024\).
* Calculate the context vector for each attention head: Different heads provide diverse attention perspectives, enabling the model to focus on multiple levels of information simultaneously, thereby enhancing its modeling capability. Sum the weighted vectors of attention heads to obtain the context vector \(c^{(k)}\) for the \(k\)th attention head :  
\[c^{(k)}=\sum_{i}a_{i}^{(k)}h_{i}^{(k)} \tag{16}\]
* Merge the context vectors of multiple attention heads: Combine the context vectors \(c^{(k)}\) from all attention heads and obtain the final context vector \(c\) through a linear transformation: \[c=W_{c}\Big{[}c^{(1)},c^{(2)},\ldots,c^{headNum}\Big{]}\] (17) where \(W_{c}\) is the linear transformation matrix after merging, and \(headNum=4\) is the number of attention heads. Next, the reasoning module uses a GRU to compute the initial guided output \(g_{initial\_op}\): \[g_{initial\_op},H_{t}=GRU(\text{Relu}(W_{8}[c:E_{op,cons,num}(s_{t-1})]),H_{t-1})\] (18) where \":\" denotes concatenation, Relu is the activation function, and \(W_{8}\in\mathbb{R}^{hsh}\) is a trainable parameter. \(H_{t-1}\in\mathbb{R}^{hsh}\) represents the hidden state of the GRU in the previous step, and \(H_{0}\) is initialized to zero. Generator Module: In the MA Module, \(g_{initial\_op}\) directly guides the Generator Module to generate operators and operands. The Generator Model consists of an operator generator and an operand generator. The guidance vector \(g_{initial\_op}\) enters the operator generator. The operator generator uses softmax to calculate the most likely \(i\)-th operator: \[P(op_{i}|g_{initial\_op},E_{op_{t-1}})=softmax(E_{op_{i}}^{T}R_{initial\_op})\] (19) \[R_{initial\_op}=Relu(W_{op}g_{initial\_op})\] (20) The guidance vector \(g_{initial\_oe}\) enters the operand generator. The operand generator uses softmax to calculate the most likely \(i\)-th operand: \[P(oe_{i}|g_{initial\_oe},E_{oe_{i-1}})=softmax(E_{oe_{i}}^{T}R_{initial\_oe})\] (21) \[R_{final\_oe}=Relu(W_{argu}g_{initial\_oe})\] (22)