Multi-scale fusion self attention mechanism

1 INTRODUCTION . Attention mechanism is a model widely used in natural language processing tasks . Attention determines where the model needs attention by constructing an attention matrix . With the in-depth study of attention model , Vaswani et al . ( 2017 ) proposed a more advanced self attention mechanism . The self attention model dynamically constructs the attention matrix by calculating the correlation degree between words . Compared with the traditional attention mechanism , the self attention model can construct different attention matrices for different inputs and retain more information . Although the current self-attention model has achieved relatively successful results , we have found a serious problem , that is , self-attention can directly calculate dependencies between words . Although this attention matrix can pay good attention to the relevant information between one-to-one words . However , in real life , the language environment is very complicated . There are often many phrases in sentences . These phrases may contain many words , but the words that make up the phrase often can not fully express the meaning of the phrase itself . At this time , we need to be able to extract the one-to-many relationship between words and phrases . for instance : In Figure 1 , we give two examples . The two parts framed in green in the sentence are two phrases in the sentence . The meaning expressed by phrases such as “ in order to ” and “ Cambridge University ” in the two sentences can not get the corresponding meaning from any word of the two phrases . Therefore , these phrases need to be regarded as a whole to construct the attention matrix , so as to ensure that the information extracted by the model is correct . In previous research work , Yao et al . ( 2019 ) conducted pooling operation on entity words in relation extraction task to achieve the effect of phrase information coding . However , this operation has great limitations , that is , we need to know the position and length of phrases in advance , which is very difficult for general tasks . In this paper , we propose to use convolution kernels of different sizes to extract phrase level information in sentences , and learn whether to interact with the sampled information adaptively by the model itself . At the same time , not all scale information is required by the model , and the conflict between phrase information and its constituent words should be taken into account when selecting . For example , as shown in sentence 1 of Figure 1 , when calculating the weight of attention matrix , if the relevance of ( go , in order to ) is large , the attention relevance of ( go , in ) ( go , order ) ( go , to ) should be reduced accordingly , and vice versa . Therefore , we design a unique matrix sparsity strategy , which can better adapt to our multi-scale fusion self-attention model . The experimental results show that our model has a better effect on the relationship extraction task , and achieves a better level than the baseline on the GLUE data set . The main contributions of this paper include : 1 . Based on the traditional self attention mechanism , the phrase level representation is extracted through sampling at different scales , and the attention matrix is constructed by using the representation , which improves the deficiency that the attention model can only extract one-to-one information between words . 2 . On the basis of integrating multi-scale information , in order to better guide the model for information selection , a sparsity strategy of attention matrix is proposed , which can better select the information that needs to be focused when constructing attention matrix . 2 RELATED WORK . The initial attention mechanism is widely used in natural language processing tasks as a model to integrate information . The initial attention mechanism is often used as a model to learn the association between hidden vectors after Recurrent Neural Network ( RNN ) . Note that the emergence of the model breaks the limitation that the traditional encoder decoder structure depends on an internal fixed length vector during encoding and decoding ( Zhou et al . ( 2016 ) ) Du et al . ( 2018 ) Found in many experiments that the original single one-dimensional vector can no longer meet the requirements of extracting information diversity , so they proposed to build a 2-D attention matrix to adapt to more complex situations . Among them , they believe that each dimension in such a 2-D level attention matrix represents a different focus direction Vaswani et al . ( 2017 ) put forward the self attention mechanism in the transformer model . The author takes the attention model as the main structure of the model , improves the parallelism of the model , and changes the invariable characteristics of the previous attention matrix . The self attention model creates more contextual representations by designing different attention matrices for different samples . Correia et al . ( 2019 ) considered that the complexity of the model will increase with the increase of sentence length in the operation process of self attention mechanism , which makes it difficult for the model to deal with long text information . Therefore , the author puts forward some suggestions α- Entmax method to construct a sparse attention matrix . By constructing such a sparse attention matrix model , the parameters are reduced , which can adapt to longer text data . Li et al . ( 2021 ) aiming at the problem of insufficient information extraction in the traditional attention mechanism , proposed a method of using the hard attention mechanism to extract the important information of the model . At the same time , the author also proposed a method of adding negative information to the attention matrix to enrich the diversity of the model . 2.1 SELF ATTENTION . Before introducing our work , we first briefly introduce the self attention model of Vaswani et al . ( 2017 ) in transformer . Formally , We assume that H is a vectorized representation of a sentence with length n. The attention mechanism starts by projecting the tokens into three subspaces : the query subspace Q , key subspace K , and value subspace V , with corresponding projection matrices WQ , WK , WV . It then computes the n × n attention matrix A as follows . A = softmax ( ( Q ·KT ) / √ dk ) ( 1 ) Where dk is a scaling factor , T represents transpose operation , and K is used to calculate the correlation score between each item and the query . Then , the normalized attention weight is calculated using softmax , and the obtained attention matrix A is used to weight the value of each item in each query context . Finally , the obtained attention matrix A is multiplied by vector V to obtain the final sentence representation . 3 METHOD . In this section , we propose a general attention model , multi-scale fusion self attention mechanism , which can pay attention to the phrase level information in the sample . Our multi-scale fusion self attention model consists of two parts : multi-scale attention module and dynamic sparse module . After obtaining the corresponding coded representation of each word of the input sentence , the multi-scale attention module obtains the attention matrix at different scales . The dynamic sparse module constructs a dynamic coefficient strategy to construct a sparse attention matrix according to the summarized attention characteristics . Finally , the constructed attention matrix is multiplied by the value matrix to obtain a sentence representation with phrase level information.The multi-scale attention module is shown in Figure 2 . 3.1 MULTI-SCALE ATTENTION MODULE . Given a sentence S of length n , where Si ( 1≤i≤n ) is the word in the sentence , we can get their vector representation after encoding . The encoder can be Bert or LSTM . Take Bert as an example : H = Bert ( S ) ( 2 ) Where H ∈ Rn∗d is the vectorized representation of the sentence , where d is the encoding length , and Hi is the encoding representation of each words in the sentence . In order to obtain vector representation of different scales , we use convolutional neural network to further extract information from sentence representation . Convolutional neural network ( CNN ) can not integrate global information because it is limited by the scale of receptive field of convolution kernel , but it also makes a great improvement for Convolutional neural network to extract local informations , We use this to control the length of phrase level information to be extracted by setting the scale k of convolution kernel , so as to achieve the effect of integrating phrase level information . Convolution is an operation between the weight vector wc and the input vector of the input sequence H. The weight matrix wc is regarded as a convolution filter . We assume that the length of the filter is k , so wc ∈ Rk∗d . Convolution involves multiplying the dot product of wc by each k-gram in sequence H to obtain another sequence Hconv : Hjconv = wc Hj−k+1 : j ( 3 ) Where means multiply element by element . In order to ensure that the length of Hconv is consistent with H , we padded the convolution operation . At the same time , due to the ability to capture different features , it is usually necessary to use multiple filters ( or feature maps ) in convolution . Therefore , we use d filters ( Wc = w1c , w 2 c , ·· , w d c ) , and the convolution operation can be expressed as : Hijconv = w i c Hj−k+1 : j ( 0 ≤ i < d ) ( 4 ) The convolution result is matrix Hconv= [ H1conv , H 2 conv , ··· , H d conv ] ∈ Rd∗n . H integrates the phrase level information with K as the length . In the process of constructing the attention matrix , we refer to the construction method of Vaswani et al , and make corresponding modifications according to the characteristics of our task . The formula is as follows : Q = H ·WQ ( 5 ) K = H ·WK ( 6 ) Kconv = H T conv ·WK ( 7 ) ATT = softmax ( ( Q ·KT ) / √ dk ) ( 8 ) ATTconv = softmax ( ( Q ·KTconv ) / √ dk ) ( 9 ) The above formula is our method to construct a multi-scale attention matrix , where WQ and WK are trainable parameter matrixs with the shape of d * d. Kconv is the key matrix calculated by Hconv integrating phrase level information . The difference from the construction of Vaswani et al is that we calculated ATTconv , which is obtained by multiplying the query matrix with word as the minimum granularity and the key matrix Kconv with phrase as the minimum granularity . Its purpose is to calculate the degree of attention between words and phrases on different scales . 3.2 DYNAMIC SPARSE MODULE . In constructing the multi-scale attention model , we note that although there is phrase information in sentences , not all phrases with length k can be expressed as phrases . At the same time , we should also consider the differences between the traditional attention matrix and the multi-scale attention matrix , that is , if a word chooses to pay attention to a phrase , it should reduce the attention to each word in the phrase . Therefore , we design a dynamic sparse module for our multi-scale attention module , which can be dynamically adjusted to ensure that the model can select more suitable phrase information.The dynamic sparse module is shown in Figure 3 . The method is as follows . Q1 = H ·WQ1 ( 10 ) K1 = H ·WK1 ( 11 ) ATTcut = softmax ( ( Q1 ·KT1 ) / √ dk ) ( 12 ) WQ1 and WK1 are two trainable parameter matrices with the shape of d * d. in the above formula , we choose to reconstruct a network for calculating similarity . Different from formula 8 , this formula will be used to construct a sparse attention matrix . We set a threshold µ for att matrix to select whether the information is truncated . If the value of this position is greater than the threshold , it will be set to 1 . If not , it will be set to 0 . The formula is as follows : Maskqj = 1 if ( ATT qjcut > µ ) else 0 ( 13 ) Because the size of the threshold will closely affect the location of the matrix ’ s attention , here we set the threshold to 1 / n , so we get a mask matrix composed of 0 and 1 . Because this mask matrix only uses word level attention , we also need to build a mask matrix of multi-scale attention matrix . In order to realize the difference between traditional attention matrix and multi-scale attention matrix , we choose to reverse build the mask matrix of multi-scale attention matrix according to Mask matrix . The formula is as follows : Maskconv = E −Mask ( 14 ) Where E is a matrix with all values of one , so we can obtain the corresponding sparse multi-scale attention matrix according to the two mask matrices . The formula is as follows : ATTM = ATT Mask ( 15 ) ATTMc = ATTconv Maskconv ( 16 ) ATTfin = [ ATTM , ATTMc ] ( 17 ) ATTfin is the final attention matrix , which is obtained by splicing the traditional attention matrix and our multi-scale attention matrix in the last dimension , so it ’ s shape is n * 2n . ATTfin represents the correlation degree between the i-th word and the j-th word . When 1 ≤ j ≤ n , its value represents the attention value between words , and when n < j ≤ 2n , it represents the attention value between the i-th word and the ( j-n ) -th phrase . V = [ H HTconv ] ·WV ( 18 ) S = ATTfin · V ( 19 ) The calculation method of the value matrix is shown in formula 16 . For the trainable parameter matrix with the shape of d * d at WV , we splice H and Hconv in the first dimension and obtain the value matrix V with the shape of 2n * d through linear transformation.Finally , we multiply the obtained attention matrix ATTfin and value matrix V to obtain the final sentence representation . After obtaining the representation , we can use pooling or other dimensionality reduction methods to realize the final flat representation .

Summary: This work proposes a multi-scale fusion self-attention module to help extract phase-level at different scales. It utilizes convolution operations with kernels of different sizes to achieve this end. The authors conduct experiments on the relation extraction and GLUE tasks to demonstrate its effectiveness.