# FA-RCNet: A Fused Feature Attention Network for Relationship Classification
## 4 Experiments
### Confusion Matrix Analysis  
This paper focuses on comparing the prediction results of the FA-RCNet model with the real results using the confusion matrix method on the SemEval-2010 Task8 dataset. Confusion matrices were calculated for 10 classifications (without differentiation of orientation) and 19 classifications (with differentiation of entity orientation), respectively. The diagonal areas of the confusion matrix show the results of correct model predictions, while the values in the other areas show the results of the error distribution.  
Figure 6 shows the confusion matrix for the results of the 10 classifications (without differentiation of direction). The graph shows that although the classification accuracy of the 'I-A' relationship has improved, it is still the smallest of all the categories. The most frequent error in all classifications is the classification of non-\"other\" as \"other\". The reason for this is that the model does not identify the type of relationship between the two entities or even that a relationship exists between the two entities. This paper argues that the reason for this phenomenon is that the semantics between each category is discrete and that to improve this phenomenon, we need to introduce external features or semantic logical reasoning.  
Figure 7 shows the confusion matrix for the results of the 19 classifications (differentiated directions). In this matrix, it is possible to show the recognition accuracy in different directions for the same relationship. The figure shows that the datum in the E-D2 (entity-destination<\(e_{2}\), \(e_{1}\)>) direction is 0, as explained in the previous section. Because there are 8000 data items in the training set of the dataset, only one datum has a relationship of E-D2, and there are 2717 data items in the test set, and only one datum has a relationship with E-D2. Deficiencies in the data set resulted in an accuracy of 0% in this direction. After excluding the E-D2 direction, I-A1 (instrument-agency<\(e_{1}\), \(e_{2}\)>) had the lowest accuracy among the other directions, which explains the low accuracy of the I-A category in the 10 classification results regardless of direction. On the one hand, this is because the focus in the features of this text is more on the role of the two entity features, weakening the links between the different words in the whole sentence. On the other hand, a part of the global features corresponding to (CLS) in the shallow information is also lost. This leads to suboptimal performance of the model in the more dependent categories of relationships between words. In future work, we will focus on the role of shallow global features in classification tasks.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **P** & **R** & **F1** \\ \hline Base & 66.86\% & 72.37\% & 69.51\% \\ Base + Attention module & 67.34\% & 72.43\% & 69.79\% \\ Base + Feature fusion module & 67.6\% & 72.20\% & 69.79\% \\ Base + Feature fusion module + Attention module & **67.80**\% & **72.23**\% & **69.95**\% \\ \hline \hline \end{tabular}
\end{table}
Table 10: Ablation experiments for each module on the KBP37 dataset.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **P** & **R** & **F1** \\ \hline Base & 82.43\% & 86.04\% & 84.19\% \\ Base + Attention module & 86.75\% & 90.81\% & 88.73\% \\ Base + Feature fusion module & 87.35\% & 91.25\% & 89.26\% \\ Base + Feature fusion module + Attention module & **89.29**\% & **91.38**\% & **90.33**\% \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation experiments for each module on the SemEval-2010 Task8 dataset.  
Figure 6: 10 Confusion matrix of classified (non-differentiated direction) results.  
Figure 7: A total of 19 confusion matrices of classified (differentiated direction) results.