Multi-scale fusion self attention mechanism

1 INTRODUCTION . Attention mechanism is a model widely used in natural language processing tasks . Attention determines where the model needs attention by constructing an attention matrix . With the in-depth study of attention model , Vaswani et al . ( 2017 ) proposed a more advanced self attention mechanism . The self attention model dynamically constructs the attention matrix by calculating the correlation degree between words . Compared with the traditional attention mechanism , the self attention model can construct different attention matrices for different inputs and retain more information . Although the current self-attention model has achieved relatively successful results , we have found a serious problem , that is , self-attention can directly calculate dependencies between words . Although this attention matrix can pay good attention to the relevant information between one-to-one words . However , in real life , the language environment is very complicated . There are often many phrases in sentences . These phrases may contain many words , but the words that make up the phrase often can not fully express the meaning of the phrase itself . At this time , we need to be able to extract the one-to-many relationship between words and phrases . for instance : In Figure 1 , we give two examples . The two parts framed in green in the sentence are two phrases in the sentence . The meaning expressed by phrases such as “ in order to ” and “ Cambridge University ” in the two sentences can not get the corresponding meaning from any word of the two phrases . Therefore , these phrases need to be regarded as a whole to construct the attention matrix , so as to ensure that the information extracted by the model is correct . In previous research work , Yao et al . ( 2019 ) conducted pooling operation on entity words in relation extraction task to achieve the effect of phrase information coding . However , this operation has great limitations , that is , we need to know the position and length of phrases in advance , which is very difficult for general tasks . In this paper , we propose to use convolution kernels of different sizes to extract phrase level information in sentences , and learn whether to interact with the sampled information adaptively by the model itself . At the same time , not all scale information is required by the model , and the conflict between phrase information and its constituent words should be taken into account when selecting . For example , as shown in sentence 1 of Figure 1 , when calculating the weight of attention matrix , if the relevance of ( go , in order to ) is large , the attention relevance of ( go , in ) ( go , order ) ( go , to ) should be reduced accordingly , and vice versa . Therefore , we design a unique matrix sparsity strategy , which can better adapt to our multi-scale fusion self-attention model . The experimental results show that our model has a better effect on the relationship extraction task , and achieves a better level than the baseline on the GLUE data set . The main contributions of this paper include : 1 . Based on the traditional self attention mechanism , the phrase level representation is extracted through sampling at different scales , and the attention matrix is constructed by using the representation , which improves the deficiency that the attention model can only extract one-to-one information between words . 2 . On the basis of integrating multi-scale information , in order to better guide the model for information selection , a sparsity strategy of attention matrix is proposed , which can better select the information that needs to be focused when constructing attention matrix . 2 RELATED WORK . The initial attention mechanism is widely used in natural language processing tasks as a model to integrate information . The initial attention mechanism is often used as a model to learn the association between hidden vectors after Recurrent Neural Network ( RNN ) . Note that the emergence of the model breaks the limitation that the traditional encoder decoder structure depends on an internal fixed length vector during encoding and decoding ( Zhou et al . ( 2016 ) ) Du et al . ( 2018 ) Found in many experiments that the original single one-dimensional vector can no longer meet the requirements of extracting information diversity , so they proposed to build a 2-D attention matrix to adapt to more complex situations . Among them , they believe that each dimension in such a 2-D level attention matrix represents a different focus direction Vaswani et al . ( 2017 ) put forward the self attention mechanism in the transformer model . The author takes the attention model as the main structure of the model , improves the parallelism of the model , and changes the invariable characteristics of the previous attention matrix . The self attention model creates more contextual representations by designing different attention matrices for different samples . Correia et al . ( 2019 ) considered that the complexity of the model will increase with the increase of sentence length in the operation process of self attention mechanism , which makes it difficult for the model to deal with long text information . Therefore , the author puts forward some suggestions α- Entmax method to construct a sparse attention matrix . By constructing such a sparse attention matrix model , the parameters are reduced , which can adapt to longer text data . Li et al . ( 2021 ) aiming at the problem of insufficient information extraction in the traditional attention mechanism , proposed a method of using the hard attention mechanism to extract the important information of the model . At the same time , the author also proposed a method of adding negative information to the attention matrix to enrich the diversity of the model . 2.1 SELF ATTENTION . Before introducing our work , we first briefly introduce the self attention model of Vaswani et al . ( 2017 ) in transformer . Formally , We assume that H is a vectorized representation of a sentence with length n. The attention mechanism starts by projecting the tokens into three subspaces : the query subspace Q , key subspace K , and value subspace V , with corresponding projection matrices WQ , WK , WV . It then computes the n × n attention matrix A as follows . A = softmax ( ( Q ·KT ) / √ dk ) ( 1 ) Where dk is a scaling factor , T represents transpose operation , and K is used to calculate the correlation score between each item and the query . Then , the normalized attention weight is calculated using softmax , and the obtained attention matrix A is used to weight the value of each item in each query context . Finally , the obtained attention matrix A is multiplied by vector V to obtain the final sentence representation . 3 METHOD . In this section , we propose a general attention model , multi-scale fusion self attention mechanism , which can pay attention to the phrase level information in the sample . Our multi-scale fusion self attention model consists of two parts : multi-scale attention module and dynamic sparse module . After obtaining the corresponding coded representation of each word of the input sentence , the multi-scale attention module obtains the attention matrix at different scales . The dynamic sparse module constructs a dynamic coefficient strategy to construct a sparse attention matrix according to the summarized attention characteristics . Finally , the constructed attention matrix is multiplied by the value matrix to obtain a sentence representation with phrase level information.The multi-scale attention module is shown in Figure 2 . 3.1 MULTI-SCALE ATTENTION MODULE . Given a sentence S of length n , where Si ( 1≤i≤n ) is the word in the sentence , we can get their vector representation after encoding . The encoder can be Bert or LSTM . Take Bert as an example : H = Bert ( S ) ( 2 ) Where H ∈ Rn∗d is the vectorized representation of the sentence , where d is the encoding length , and Hi is the encoding representation of each words in the sentence . In order to obtain vector representation of different scales , we use convolutional neural network to further extract information from sentence representation . Convolutional neural network ( CNN ) can not integrate global information because it is limited by the scale of receptive field of convolution kernel , but it also makes a great improvement for Convolutional neural network to extract local informations , We use this to control the length of phrase level information to be extracted by setting the scale k of convolution kernel , so as to achieve the effect of integrating phrase level information . Convolution is an operation between the weight vector wc and the input vector of the input sequence H. The weight matrix wc is regarded as a convolution filter . We assume that the length of the filter is k , so wc ∈ Rk∗d . Convolution involves multiplying the dot product of wc by each k-gram in sequence H to obtain another sequence Hconv : Hjconv = wc Hj−k+1 : j ( 3 ) Where means multiply element by element . In order to ensure that the length of Hconv is consistent with H , we padded the convolution operation . At the same time , due to the ability to capture different features , it is usually necessary to use multiple filters ( or feature maps ) in convolution . Therefore , we use d filters ( Wc = w1c , w 2 c , ·· , w d c ) , and the convolution operation can be expressed as : Hijconv = w i c Hj−k+1 : j ( 0 ≤ i < d ) ( 4 ) The convolution result is matrix Hconv= [ H1conv , H 2 conv , ··· , H d conv ] ∈ Rd∗n . H integrates the phrase level information with K as the length . In the process of constructing the attention matrix , we refer to the construction method of Vaswani et al , and make corresponding modifications according to the characteristics of our task . The formula is as follows : Q = H ·WQ ( 5 ) K = H ·WK ( 6 ) Kconv = H T conv ·WK ( 7 ) ATT = softmax ( ( Q ·KT ) / √ dk ) ( 8 ) ATTconv = softmax ( ( Q ·KTconv ) / √ dk ) ( 9 ) The above formula is our method to construct a multi-scale attention matrix , where WQ and WK are trainable parameter matrixs with the shape of d * d. Kconv is the key matrix calculated by Hconv integrating phrase level information . The difference from the construction of Vaswani et al is that we calculated ATTconv , which is obtained by multiplying the query matrix with word as the minimum granularity and the key matrix Kconv with phrase as the minimum granularity . Its purpose is to calculate the degree of attention between words and phrases on different scales . 3.2 DYNAMIC SPARSE MODULE . In constructing the multi-scale attention model , we note that although there is phrase information in sentences , not all phrases with length k can be expressed as phrases . At the same time , we should also consider the differences between the traditional attention matrix and the multi-scale attention matrix , that is , if a word chooses to pay attention to a phrase , it should reduce the attention to each word in the phrase . Therefore , we design a dynamic sparse module for our multi-scale attention module , which can be dynamically adjusted to ensure that the model can select more suitable phrase information.The dynamic sparse module is shown in Figure 3 . The method is as follows . Q1 = H ·WQ1 ( 10 ) K1 = H ·WK1 ( 11 ) ATTcut = softmax ( ( Q1 ·KT1 ) / √ dk ) ( 12 ) WQ1 and WK1 are two trainable parameter matrices with the shape of d * d. in the above formula , we choose to reconstruct a network for calculating similarity . Different from formula 8 , this formula will be used to construct a sparse attention matrix . We set a threshold µ for att matrix to select whether the information is truncated . If the value of this position is greater than the threshold , it will be set to 1 . If not , it will be set to 0 . The formula is as follows : Maskqj = 1 if ( ATT qjcut > µ ) else 0 ( 13 ) Because the size of the threshold will closely affect the location of the matrix ’ s attention , here we set the threshold to 1 / n , so we get a mask matrix composed of 0 and 1 . Because this mask matrix only uses word level attention , we also need to build a mask matrix of multi-scale attention matrix . In order to realize the difference between traditional attention matrix and multi-scale attention matrix , we choose to reverse build the mask matrix of multi-scale attention matrix according to Mask matrix . The formula is as follows : Maskconv = E −Mask ( 14 ) Where E is a matrix with all values of one , so we can obtain the corresponding sparse multi-scale attention matrix according to the two mask matrices . The formula is as follows : ATTM = ATT Mask ( 15 ) ATTMc = ATTconv Maskconv ( 16 ) ATTfin = [ ATTM , ATTMc ] ( 17 ) ATTfin is the final attention matrix , which is obtained by splicing the traditional attention matrix and our multi-scale attention matrix in the last dimension , so it ’ s shape is n * 2n . ATTfin represents the correlation degree between the i-th word and the j-th word . When 1 ≤ j ≤ n , its value represents the attention value between words , and when n < j ≤ 2n , it represents the attention value between the i-th word and the ( j-n ) -th phrase . V = [ H HTconv ] ·WV ( 18 ) S = ATTfin · V ( 19 ) The calculation method of the value matrix is shown in formula 16 . For the trainable parameter matrix with the shape of d * d at WV , we splice H and Hconv in the first dimension and obtain the value matrix V with the shape of 2n * d through linear transformation.Finally , we multiply the obtained attention matrix ATTfin and value matrix V to obtain the final sentence representation . After obtaining the representation , we can use pooling or other dimensionality reduction methods to realize the final flat representation .

Summary: The manuscript presents a multi-scale self-attention method for NLP tasks. The aim is to better extract phrase- and word-level features. The main contribution of the proposed method is to apply different kernel sizes for feature extraction and multi-scale attention fusion. Additionally, a mechanism called dynamic sparse module is applied to adjust the weights of the obtained attention matrix.