# Recall Mechanism and Multi-Head Attention for Numerical Reasoning
## 5 Experiment Results and Analysis
### Error Analysis  
To better understand the limitations of our model in predicting mathematical operations, we analyze common error patterns. The errors can be categorized into the following three types:  
Operator prediction error: One common source of error is the incorrect selection of mathematical operations. For instance, as shown in Example 1 in Figure 8, in a case where the task required computing the percentage change in deferred tax assets, the correct sequence involved a subtraction operation, but the model mistakenly predicted an addition operation. Such errors often arise when the model fails to grasp the underlying mathematical relationship, particularly in scenarios involving changes or differences where subtraction is required instead of addition. \begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline
**Model** & \multicolumn{2}{l}{**The Entire Dataset**} & \multicolumn{2}{l}{**=1(3717)**} & \multicolumn{2}{l}{**=2(3013)**} & \multicolumn{2}{l}{\(\geq\)3(512)**} \\  & \multicolumn{1}{l}{**Eex**} & \multicolumn{1}{l}{**Prog**} & \multicolumn{1}{l}{**Eex**} & \multicolumn{1}{l}{**Prog**} & \multicolumn{1}{l}{**Eex**} & \multicolumn{1}{l}{**Prog**} & \multicolumn{1}{l}{**Eex**} & \multicolumn{1}{l}{**Prog**} & \multicolumn{1}{l}{**Eex**} & \multicolumn{1}{l}{**Prog**} \\  & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} & \multicolumn{1}{l}{**Acc**} \\ \hline FINQANet \(\dagger\) & 65.05 & 63.52 & 73.70 & 71.25 & 62.34 & 59.65 & 28.57 & 23.80 \\ ELASTIC \(\dagger\) & 68.96 & 65.21 & 76.30 & 75.66 & 66.01 & 66.01 & 31.78 & 31.10 \\ RMNR & 70.58 & 66.61 & 78.96 & 76.61 & 65.04 & 58.68 & **32.14** & **27.38** \\ MANR & 70.21 & 65.74 & 76.49 & 74.01 & **68.22** & **60.88** & 30.95 & 25.00 \\ RMMANR & **71.63** & **67.31** & **80.34** & **77.68** & 66.01 & 59.41 & 30.95 & 25.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results on different program steps. \(\dagger\) denotes that the results are sourced from the original ELASTIC [6] paper. Each model employs RoBERTa-large for encoding. Bold values indicate the best performance for each metric. Operand prediction error: Another prevalent issue is the misidentification of numerical values within the input text. As shown in Figure 8, in Example 2 involving the ratio of collateral pledged to short-term borrowings, the model incorrectly selected 19.3 instead of the correct value 2.3. This type of error may stem from: (1) Numerical mismatch: The model might prioritize selecting numbers that appear more prominently in the text, overlooking the correct values required by the problem. (2) Contextual misunderstanding: The model may fail to correctly associate numerical values with temporal constraints, leading to the selection of numbers from incorrect time periods or categories. Operand and operator prediction error: As shown in Example 3 in Figure 8, in a case requiring the calculation of the percentage change in the fair value of non-vested shares, the correct sequence should involve a multiplication step followed by subtraction and division. However, the model only predicted multiplication and division, neglecting the intermediate subtraction operation. This suggests that the model struggles with multi-step reasoning, particularly in percentage-based calculations, where computing the absolute difference before division is crucial. Figure 8: Examples of prediction error. These errors indicate three major challenges: (1) inaccurate selection of mathematical operations, (2) incorrect numerical value extraction, and (3) incorrect operands and operators. To mitigate these issues, future work can focus on the following:  
* Enhancing the model's ability to understand mathematical relationships by refining operation selection mechanisms. * Improving number extraction accuracy through better numerical alignment techniques.