# CRSAtt: By Capturing Relational Span and Using Attention for Relation Classification
## 2 Related Work
[19] proposed an RNN-based framework to model long-range relational patterns, and experiments demonstrated the capability of the RNN-based approach in remote pattern modeling. Ref. [20] proposed a deep-learning relation classification model based on the shortest dependency path. They not only used a Bidirectional RCNN but also considered the problem of the directionality of relationships between entities. Ref. [9] proposed the use of a Bidirectional Long Short-Term Memory network (BLSTM) to model sentences containing complete, sequential information about all words, achieving state-of-the-art performance at the time. Ref. [21] added the attention mechanism to Bi-LSTM and proposed the AttBLSTM model to capture the most important semantic information in sentences. Ref. [22] designed an attention-based BLSTM layer for converting semantic information into high-level features, and also proposed a new filtering mechanism to reduce noise. All of the above methods use recurrent neural networks or their variants of long and short-term memory networks as feature extraction frameworks, and all use the Linear() function to map the probability distribution of relational categories. Although recurrent neural networks are better able to process sequential data and extract global information compared with convolutional neural networks, they still have the disadvantages of being unsuitable for long sequences and prone to the problem of gradient disappearance. Table 2 shows a summary of our model and the recurrent-neural-network-based models. Attention mechanism: Ref. [23] added the attention mechanism to CNNs. Two levels of attention mechanism are employed; the first one is applied to attention between individual word pairs in the input sequence, and the second one is applied to attention on the blending layer for the target category. Ref. [24] proposed a Bert-based model to perform relation extraction without combining lexical and syntactic features, achieving SOTA performance and providing a baseline for follow-up. Ref. [25] proposed a transformer-based relation extraction method TRE, which replaces the explicit linguistic features required by previous methods with implicit features captured in a pre-trained linguistic representation. Ref. [26] added an additional MTB (Matching The Blanks) task to the pre-training process of BERT to improve the performance of relation extraction during the pre-training phase. Ref. [27] introduced a dependency-based attention mechanism in the BERT architecture to learn high-level syntactic features. The dependency relation between each word and the target entity is considered, while different levels of semantic information are obtained by using the BERT middle layer to fuse multi-grain features for the final relation classification. Ref. [28] used a fine-tuned BERT model to extract the semantic representation of sequences and then used segmental convolution to obtain the semantic information affecting the relation classification. The closest to our work is RBERT [10], which also uses the pre-trained model BERT as a feature extractor to extract high-quality semantic features, and adds special symbols $ and # in data pre-processing as a way to highlight entity vectors and facilitate the classification of entity relations. The above methods all use Transformer or BERT based on attention mechanism as the feature extraction framework, \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Feature Extraction** & **Classification Method** & **Year** \\ \hline RNN+PI [19] & RNN & Linear + SoftMax & 2015 \\ BRCNN [20] & LSTM + Convolution layers & Linear + coarse-grained & 2016 \\ BLSTM [9] & BLSTM & SoftMax & 2015 \\ Att-BLSTM [21] & BLSTM + Attention Layer & Linear + SoftMax & 2016 \\ MALNet [22] & BLSTM + Attention Filter & Linear +SoftMax & 2020 \\  & Layer & Multi-Class Attention+ & - \\ CRSAtt(Our) & SpanBert & SoftMax & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of our model and the recurrent-neural-network-based models.