# Sentence Representation Method Based on Multi-Layer Semantic Network
## 3 Methods
### Design of Semantic Representation Network Based on Multi Attention
#### 3.2.2 Multi-Layer Semantic Information Extraction and Enhancement  
This part is the core of the multi-layer semantic representation network. Its purpose is to obtain as many layers of sentence semantic information as possible. As shown in Figure 4, this paper constructs a multi-layer attention network using the multi-head attention mechanism. It emphasizes the sentence word order by adding a location mask. Finally, the fused multi-layer semantic vector is used as the input of the representation layer. 1. Multi-layer semantic information extraction  
The single-layer attention network can only capture the semantic information of a single layer. Influenced by [PERSON] [36], we build a semantic extraction network based on the multi-attention mechanism to capture more semantic information. For the layer \(\tau\) in the multilayer attention network, the attention weight vector \(a_{\tau}\in R^{N\times 2u}\) of the layer is calculated first; the calculation method of attention weight vector \(a_{\tau}\) is shown in Formula (8). \[a_{\tau}=softmax(\alpha_{\tau}\sigma(\beta_{\tau}H+\overline{\alpha}_{\tau})+ \overline{\beta}_{\tau})^{T} \tag{8}\]  
where \(\tau=1,\ 2,\cdots,\lambda\), \(\lambda\) denotes the number of layers of multilayer attention network, \(\alpha_{\tau}\in R^{d_{\kappa}\times 2u}\), \(\beta_{\tau}\in R^{2u\times d_{\varrho}}\), \(\overline{\alpha}_{\tau}\in R^{d_{\delta}\times 1}\), \(\overline{\beta}_{\tau}\in R^{2u\times 1}\) is the parameter of layer \(\tau\), \(T\) is a transposition, \(\sigma\) is activation function, and \(H\in R^{N\times 2u}\) is the embedded representation of sentence \(S\). Although the attention mechanism can capture important semantic information in a sentence, the attention mechanism's calculation method to obtain semantic information is based on the word bag model. It does not pay attention to each word's order. Suppose the self-attention layer mainly captures the phrase components in the sentence. As long as the sentence phrases still have strong internal relations, the sentence is disordered; the results obtained after ordering are consistent with those obtained before the disorder. However, based on the attention weight vector, there are great differences in the understanding of sentences. Therefore, word order \(a_{\tau}\), information is added to strengthen the core semantic relationship of sentences. 2. Word order information enhancement  
Word order information can be a deterministic function of position [37] or obtained by network representation learning. This paper obtains word order information by calculating the relative distance between words considering the network structure to not increase the network complexity. Firstly, the absolute distance \(D_{ij}=|i-j|\) of any two words in the word vector sequence \(x\) corresponding to sentence \(S\) is calculated, where \(i,j=1,\ 2,\ldots,L\), \(L\) are the number of participles contained in the sentence. Then, for the distance, \(D_{ij}\) carries on the deviation standardization processing, obtains the word order information \(P_{ij}\), ensuring that even in complex sentences (containing more  
Figure 4: Multi-layer semantically encoded network structure diagram. words) \(D_{ij}\) will not calculate the model too large and cause the calculation to be too slow. The calculation formula of \(P_{ij}\) is as follows:  
\[P_{ij}=\frac{D_{ij}-D_{min}}{Dmin_{max}} \tag{9}\]  
where \(D_{max}\) is the maximum absolute distance between any two words, and \(D_{min}\) is the minimum absolute distance between any two words. Then, the word order information is used to weight the attention weight vector \(a_{\tau}\), that is, Formula (9) is replaced into Formula (8) to calculate the weighted semantic weight \(a_{\tau}\), The calculation formula of \(a_{\tau}\)' is shown in (10).