# CRSAtt: By Capturing Relational Span and Using Attention for Relation Classification
## 4 Experiment
### Experimental Results  
This section will show the experimental results of our method on two publicly available datasets. On the SemEval-2010 Task 8 dataset, we used precision, recall, and micro f1 as our parameter metrics, and performed a full comparison experiment with previous methods. Additionally, to demonstrate the specific performance of our method on each relation class, we compared it in more detail with two, more powerful, current state-of-the-art models. To verify that our model has good generalizability, we conducted a comparison test on the KBP37 dataset using micro f1 as a parameter indicator. Table 7 shows the experimental results of our model with previous models on the SemEval-2010 Task 8 dataset. Our models are divided into two types, and one is the base model with SpanBert as the feature extraction architecture. The other is the advanced model with spanbert_large as the feature extraction architecture. Previous models include GLFN, TRE; BERTEM+MTB; R-BERT; Att-RCNN; LGCNN; Bi-SDP-Att; MALNet; BERT with Entity, convolution, and max-pooling; and D-BERT. The experimental results prove that our method outperforms all other methods, where the values in bold represent the most advanced results in this metric. It can be observed that our base model achieves state-of-the-art results in both recall and f1 metrics. The advanced model with spanbert_large as the feature extraction framework substantially outperforms the existing models in all three metrics. Take the BERT with Entity, convolution, and max-pooling model as an example, where precision improves by 0.63 percentage points, recall improves by 0.44 percentage points, and f1 improves by 0.6 percentage points. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyper-Parameters** & **Description** & **Value** \\ \hline dw & Size of BERT Output Vector & 786 \\ lm & Max Sequence Length after Tokenization & 384 \\ e & Epochs & 50 \\ lr & Learning Rate & \(2\times 10^{-5}\) \\ dr & Dropout Ratio & 0.2 \\ tnbs & Train batchsize & 32 \\ tbs & Test batchsize & 16 \\ alpha & Focal loss \(\alpha\) & [1,1,1,1,\(\cdots\),1,1,1,1,1] \\ gamma & Focal loss \(\gamma\) & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Hyperparameter settings for our model on the KBP37 dataset. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Precision** & **Recall** & **f1** \\ \hline GLFN [34] & - & - & 86.2 \\ TRE [25] & 88.0 & 86.2 & 87.1 \\ BERTEM + MTB [26] & - & 89.5 \\ R-BERT [10] & 87.03 & 90.10 & 88.54 \\ Att-RCNN [55] & - & - & 86.6 \\ LGCNN [36] & - & - & 85.5 \\ B-SDP-Att [7] & - & - & 85.1 \\ MALNet [22] & - & - & 86.3 \\ BERT with entity, convolution, and max-pooling [28] & 89.44 & 90.59 & 89.95 \\ D-BERT [27] & - & - & 90.1 \\ CRSAIt (parameter) & 88.78 & **91.60** & 90.17 \\ CRSAIt (parameter,large) & **90.07** & 91.03 & **90.55** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparison of our model with existing methods on the SemEval-2010 Task 8 dataset. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyper-Parameters** & **Description** & **Value** \\ \hline dw & Size of BERT Output Vector & 786 \\ lm & Max Sequence Length after Tokenization & 384 \\ e & Epochs & 50 \\ lr & Learning Rate & \(2\times 10^{-5}\) \\ dr & Dropout Ratio & 0.1 \\ tnbs & Train batchsize & 32 \\ tbs & Test batchsize & 16 \\ alpha & Focal loss \(\alpha\) & [1,0,4,0,6,0,2,0,4,0,4,0,4,0,4,4,0,4,0,4,0,4,0,4,0,Table 8 specifically shows the score comparison of our base model with R-BERT and BERT-ECM (BERT with entity, convolution, and max-pooling) on the SemEval-2010 Task 8 dataset for precision, recall, and f1. We find that R-BERT only performs better in the Message-Topic relation, and in comparison with BERT-ECM, we find that BERT-ECM is slightly better than our model in precision; however, in recall and f1, our model outperforms BERT-ECM, especially in the recall metric. We attribute this to our proposed attention-based prediction structure.