# FA-RCNet: A Fused Feature Attention Network for Relationship Classification
## 3 Methodology
### Feature Fusion and Attention Layers
#### 3.3.1 Feature Fusion Layer  
The word vectors output from the embedding layer are trained by BERT to produce a hidden layer output \(\text{H}_{i}\) at each layer. The mask matrix consists of 0s and 1s and is generated automatically by reading the entity identifiers \"$\" and \"#\" in the sentence. The number of the position corresponding to the vector between two markers is 1, indicating retention, and the number of all positions outside the vector corresponding to the two entities is 0, indicating discard. Figure 3 shows the Mask operation flow. Hidden out is the output of the hidden layer of the BERT model. The mask matrix is the mask matrix for entity feature extraction. The entity token is the set of entity vectors obtained after the mask matrix. Figure 3: The mask matrix is automatically generated by reading the entity markers, and the mask matrix is multiplied with the hidden layer output vector to extract the entity’s corresponding feature vector. Figure 2: Embedded process. One entity may correspond to more than one word, so one entity may correspond to more than one hidden layer output. Taking the first hidden layer output as an example, the hidden layer outputs corresponding to two entities in a sentence are extracted from H1 by Mask matrix. This process is expressed mathematically as Equation (1):  
\[\left[h_{i}\cdots h_{j},h_{k}\cdots h_{m}\right]=\left[ \begin{array}{ccc}H_{1- 1}&\cdots&H_{1-n}\end{array} \right]\times\left[ \begin{array}{c}m_{1}\\ \vdots\\ m_{n}\end{array} \right] \tag{1}\]  
where \(h_{i}\)-\(h_{j}\) and \(h_{k}\)-\(h_{m}\) can be represented as the set of vectors corresponding to entity I and entity II in the first hidden layer. \(H_{1-1}\) and \(H_{1-n}\) denote the first and last elements in the output of the hidden layer. \(m_{1}\) and \(m_{n}\) denote the first and last elements of the mask matrix. The set of vectors obtained by extraction is averaged to obtain the feature vectors corresponding to the two entities in the first hidden layer. The 24 hidden layer outputs of the BERT model are masked and averaged separately. The 24 feature vectors are obtained for each of the two entities. In order to make full use of the deep semantic information and the shallow semantic information in the hidden layer output, the sum operation is applied to these 24 feature vector pairs. This allows the semantic information contained in the output of each hidden layer to be combined. The combined vectors are all applied to the word vector generation process. This process can be expressed mathematically as Equations (2) and (3):  
\[E_{1}=\sum_{a=1}^{24}\left(\frac{1}{j_{a}-i_{a}+1}\sum_{b=i}^{j}h_{b}\right) \tag{2}\]  
\[E_{2}=\sum_{a=1}^{24}\left(\frac{1}{m_{a}-k_{a}+1}\sum_{b=k}^{m}h_{b}\right) \tag{3}\]  
where \(E_{1}\) is the feature sum corresponding to entity I; \(E_{2}\) is the feature sum corresponding to entity II; \(i_{a}\)-\(j_{a}\) is the starting position of the feature vector corresponding to entity I at layer a; and \(k_{a}\)-\(m_{a}\) are the starting and ending positions of the feature vector corresponding to entity II at layer a. In the previous relational classification models, only the hidden state output of the last layer or the last two layers of the BERT model was used for relational classification tasks, such as R-bert, etc. It is shown that the deeper feature vectors contain more semantic information and are more suitable for relational classification tasks, so the hidden state of the last layer is output separately as the first feature enhancement module. The classification task requires not only the semantic features of the two entities but also the global features of the whole sentence. In this paper, the global features of the whole sentence are output separately as the second feature enhancement module. An activation function is added to the fused features \(E_{1}\), \(E_{2}\), the last layer of the hidden state outputs \(e_{1}^{24}\), \(e_{2}^{24}\), and the global feature vector \(e_{0}\), respectively. A fully connected layer is also connected.