# A Keyword Detection and Context Filtering Method for Document Level Relation Extraction
## 4 Experiment and Results
### Case Study  
In order to further investigate the effectiveness of this model compared to the ATLOP model, we divided the dataset into different groups in terms of corpus length and tested the ATLOP model and our model on them separately. From the results in Figure 5, it can be seen that the SAM-LSTM model in this paper consistently outperforms the ATLOP model. It can also be seen that the performance of our model decreases more slowly when the text length rises. This indicates that the method in this paper can preserve the key feature information even in longer difficult cases through a powerful self-attentive mechanism, while being less affected by irrelevant information. We also note that the time consumption of our model does not increase significantly when \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Year** & **CDR** & **GDA** \\ \hline EoG [6] & 2019 & 63.6 & 81.5 \\ LSR [7] & 2020 & 64.8 & 82.2 \\ DHG [17] & 2020 & 65.9 & 83.1 \\ GLRE [9] & 2020 & 68.5 & - \\ SciBERT\({}_{base}\)[31] & 2019 & 65.1 & 82.5 \\ Kw-BioBERT [22] & 2021 & **76.4** & - \\ ATLOP-SciBERT\({}_{large}\)[14] & 2021 & 69.4 & 83.9 \\
**SAM-LSTM-SciBERT\({}_{large}\)** & & 73.5 & **84.7** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results (%) on the biomedical datasets CDR and GDA. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Ign F1** & **F1** \\ \hline w/o Memory & 59.45 & 60.89 \\ w/o Self-Attention & 58.88 & 60.48 \\ w/o Balanced loss & 59.32 & 61.22 \\ w/o SAM-LSTM & 57.60 & 59.52 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Ablation study of SAM-LSTM on Dev. faced with long difficult cases. This is mainly influenced by the length of the sequence due to the nature of the LSTM itself. However, the processing time of our method increased by a factor of 1.5 on average compared to ATLOP. We believe that this is mainly due to our strategy of successfully matching keywords for each pair of entities, which generates a relatively large amount of computation when there are more entities in the document. In addition, to explore whether the models in this paper can actually capture key features, experiments are designed to compare the contextual feature extraction ability of the SAM-LSTM model with that of the ATLOP model. The analysis continues using the corpus mentioned in Section 1. While classifying the relation between the two entities \"[PERSON]\" and \"Bendigo\" in the text, the contextual features used by the two models for classification are first obtained. Then, the cosine similarity is computed between the two and the word embeddings of the manually selected keywords \"born\" and \"died\". The experimental results in Table 6 show that the similarity of our model is up to 8.13% higher than that of ATLOP. This supports why our model performs better: ATLOP uses the attention relation of the pre-trained model to weight the average word embedding of the full text, so that the key features are diluted by the large number of irrelevant vectors within the document, which deviates from the features originally needed for the downstream classification task. In contrast, the self-attentive memory unit used in this paper can capture and record these features while processing full-text information, which can achieve more favorable results for classification. To further validate the performance of our model in difficult cases, we chose a text containing distant relational dependencies as an example. As shown in Figure 6, when our model classifies the relationship between \"Tang Dynasty\" and 'Chang'an', it can be seen that the subject 'Tang Dynasty' is separated from the object 'Chang'an' by 13 sentences. However, we can see from the similarity visualization that our model still accurately filters the keyword \"capital\" in 15 sentences and finally outputs features that are highly similar to it. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Keyword** & **ATLOP** & **SAM-LSTM** \\ \hline “born” & 0.6360 & 0.6813 \\ “died” & 0.5669 & 0.6482 \\ \hline \hline \end{tabular}
\end{table}