# FA-RCNet: A Fused Feature Attention Network for Relationship Classification
## 3 Methodology
### Feature Fusion and Attention Layers
#### 3.3.2 Attention Layers  
The attention modules used in existing relational classification models are all attention mechanisms based on encoder and decoder structures. The weight of each feature is determined by querying the similarity between the features. For the relational classification task, most of the features are extracted using the attention mechanism for all words in a sentence. However, the classification task is the discrimination of the relationship between two entities. Using the above attention mechanism weakens the relationship weights between two entities. To solve the above problem, the feature amplification attention module is proposed in this paper. The mean and variance in a feature vector are calculated and the importance of the feature vector is determined by the scale factor in the BN [49; 50] mechanism. The feature values are also normalized in the linear part of the nonlinear function. The feature values are then scaled by a weight scaling factor \(\frac{\gamma_{i}}{\sum_{j=0}\gamma_{j}}\). The purpose is to allow positive features to be feature-enhanced and negative features to be suppressed. This process can be expressed mathematically as Equation (7):  
\[x_{i}^{\prime}=BN(x_{i})=\gamma\frac{x_{i}-\mu_{B}}{\sqrt{\sigma_{B}^{2}+ \epsilon}}+\beta \tag{7}\]  
where \(\gamma\) and \(\beta\) are trainable affine transform parameters. \(x_{i}\) is the input to the BN layer. \(x_{i}^{\prime}\) is the output of the BN layer. \(\mu_{B}\) is the mean of the elements in a feature vector. \(\sigma_{B}\) is the variance of the elements in a feature vector with a constant \(\epsilon>0\) to ensure that it does not divide by zero. Figure 4 shows the feature amplification attention module.  
In feature amplification attention, the feature vector of a single entity and the global feature vector containing the contextual information of the whole sentence are fed into the attention module, and each element of the feature vector is mapped to a single feature value. Through the attention mechanism, not only the important features in the global feature can be amplified to make the contextual information richer, but also the relationship between two entities can be better fitted. This not only preserves more positive semantic information  
Figure 4: Channel Attention Modules.  
but also enhances the sensitivity of the model to the directionality of the relationship due to the further enrichment of contextual information.  
Meanwhile, an attention residual module is designed in this paper. The input and output of the attention module are added with the purpose of further increasing the weights of the effective features while retaining the original features. In this way, all features can be involved in the subsequent learning and updating of the network. Another role of the residual network is to effectively avoid the problems of gradient explosion and gradient disappearance when the number of layers of the neural network increases. The above process can be expressed as Equations (8)-(10):  
\[T_{1-\text{out}} =\text{Sigmoid}\Bigg{(}\frac{\gamma_{T_{1}-i}}{\sum_{j=0}\gamma _{j}}BN(T_{1})\Bigg{)}+T_{1} \tag{8}\] \[T_{2-\text{out}} =\text{Sigmoid}\Bigg{(}\frac{\gamma_{T_{2}-i}}{\sum_{j=0}\gamma _{j}}BN(T_{2})\Bigg{)}+T_{2}\] (9) \[e_{0-\text{out}} =\text{Sigmoid}\Bigg{(}\frac{\gamma_{e_{0}-i}}{\sum_{j=0}\gamma _{j}}BN(e_{0})\Bigg{)}+e_{0} \tag{10}\]  
where \(\frac{T_{i}}{\sum_{j=0}\gamma_{j}}\) is the weight scaling factor. \(T_{1}\) and \(T_{2}\) are the fusion vectors of the two entities output by the feature fusion module. \(e_{0}\) is the feature vector corresponding to [CLS]. Sigmoid is the nonlinear activation function. \(T_{1-Att}\), \(T_{2-Att}\), and \(e_{0-Att}\) are the fusion vectors of the two entities with [CLS] corresponding to the attentional output. \(T_{1}\), \(T_{2}\), and \(e_{0}\) correspond to the attentional input, respectively.