Year: 2021

Abstract:
Self attention is widely used in various tasks because it can directly calculate the dependency between words, regardless of distance. However, the existing self attention lacks the ability to extract phrase level information. This is because the self attention only considers the one-to-one relationship between words and ignores the one-to-many relationship between words and phrases. Consequently, we design a multi-scale fusion self attention model for phrase information to resolve the above issues. Based on the traditional attention mechanism, multi-scale fusion self attention extracts phrase information at different scales by setting convolution kernels at different levels, and calculates the corresponding attention matrix at different scales, so that the model can better extract phrase level information. Compared with the traditional self attention model, we also designed a unique attention matrix sparsity strategy to better select the information that the model needs to pay attention to, so that our model can be more effective. Experimental results show that our model is superior to the existing baseline model in relation extraction task and GLUE task.

heading: 1 INTRODUCTION
text: Attention mechanism is a model widely used in natural language processing tasks. Attention determines where the model needs attention by constructing an attention matrix. With the in-depth study of attention model, Vaswani et al. (2017) proposed a more advanced self attention mechanism. The self attention model dynamically constructs the attention matrix by calculating the correlation degree between words. Compared with the traditional attention mechanism, the self attention model can construct different attention matrices for different inputs and retain more information.
Although the current self-attention model has achieved relatively successful results, we have found a serious problem, that is, self-attention can directly calculate dependencies between words. Although this attention matrix can pay good attention to the relevant information between one-to-one words. However, in real life, the language environment is very complicated. There are often many phrases in sentences. These phrases may contain many words, but the words that make up the phrase often cannot fully express the meaning of the phrase itself. At this time, we need to be able to extract the one-to-many relationship between words and phrases. for instance:
In Figure 1, we give two examples. The two parts framed in green in the sentence are two phrases in the sentence. The meaning expressed by phrases such as “in order to” and “Cambridge University” in the two sentences can not get the corresponding meaning from any word of the two phrases. Therefore, these phrases need to be regarded as a whole to construct the attention matrix, so as to ensure that the information extracted by the model is correct. In previous research work, Yao et al. (2019) conducted pooling operation on entity words in relation extraction task to achieve the effect
of phrase information coding. However, this operation has great limitations, that is, we need to know the position and length of phrases in advance, which is very difficult for general tasks.
In this paper, we propose to use convolution kernels of different sizes to extract phrase level information in sentences, and learn whether to interact with the sampled information adaptively by the model itself. At the same time, not all scale information is required by the model, and the conflict between phrase information and its constituent words should be taken into account when selecting. For example, as shown in sentence 1 of Figure 1, when calculating the weight of attention matrix, if the relevance of (go, in order to) is large, the attention relevance of (go, in) (go, order) (go, to) should be reduced accordingly, and vice versa. Therefore, we design a unique matrix sparsity strategy, which can better adapt to our multi-scale fusion self-attention model.
The experimental results show that our model has a better effect on the relationship extraction task, and achieves a better level than the baseline on the GLUE data set.
The main contributions of this paper include:
1. Based on the traditional self attention mechanism, the phrase level representation is extracted through sampling at different scales, and the attention matrix is constructed by using the representation, which improves the deficiency that the attention model can only extract one-to-one information between words.
2. On the basis of integrating multi-scale information, in order to better guide the model for information selection, a sparsity strategy of attention matrix is proposed, which can better select the information that needs to be focused when constructing attention matrix.

heading: 2 RELATED WORK
text: The initial attention mechanism is widely used in natural language processing tasks as a model to integrate information. The initial attention mechanism is often used as a model to learn the association between hidden vectors after Recurrent Neural Network (RNN). Note that the emergence of the model breaks the limitation that the traditional encoder decoder structure depends on an internal fixed length vector during encoding and decoding (Zhou et al. (2016))
Du et al. (2018) Found in many experiments that the original single one-dimensional vector can no longer meet the requirements of extracting information diversity, so they proposed to build a 2-D attention matrix to adapt to more complex situations. Among them, they believe that each dimension in such a 2-D level attention matrix represents a different focus direction
Vaswani et al. (2017) put forward the self attention mechanism in the transformer model. The author takes the attention model as the main structure of the model, improves the parallelism of the model, and changes the invariable characteristics of the previous attention matrix. The self attention model creates more contextual representations by designing different attention matrices for different samples.
Correia et al. (2019) considered that the complexity of the model will increase with the increase of sentence length in the operation process of self attention mechanism, which makes it difficult for the model to deal with long text information. Therefore, the author puts forward some suggestions α- Entmax method to construct a sparse attention matrix. By constructing such a sparse attention matrix model, the parameters are reduced, which can adapt to longer text data.
Li et al. (2021) aiming at the problem of insufficient information extraction in the traditional attention mechanism, proposed a method of using the hard attention mechanism to extract the important information of the model. At the same time, the author also proposed a method of adding negative information to the attention matrix to enrich the diversity of the model.

heading: 2.1 SELF ATTENTION
text: Before introducing our work, we first briefly introduce the self attention model of Vaswani et al. (2017) in transformer.
Formally, We assume that H is a vectorized representation of a sentence with length n. The attention mechanism starts by projecting the tokens into three subspaces: the query subspace Q, key subspace
K, and value subspace V , with corresponding projection matrices WQ, WK , WV . It then computes the n × n attention matrix A as follows.
A = softmax((Q ·KT )/ √ dk) (1)
Where dk is a scaling factor, T represents transpose operation, and K is used to calculate the correlation score between each item and the query. Then, the normalized attention weight is calculated using softmax, and the obtained attention matrix A is used to weight the value of each item in each query context. Finally, the obtained attention matrix A is multiplied by vector V to obtain the final sentence representation.

heading: 3 METHOD
text: In this section, we propose a general attention model, multi-scale fusion self attention mechanism, which can pay attention to the phrase level information in the sample. Our multi-scale fusion self attention model consists of two parts: multi-scale attention module and dynamic sparse module. After obtaining the corresponding coded representation of each word of the input sentence, the multi-scale attention module obtains the attention matrix at different scales. The dynamic sparse module constructs a dynamic coefficient strategy to construct a sparse attention matrix according to the summarized attention characteristics. Finally, the constructed attention matrix is multiplied by the value matrix to obtain a sentence representation with phrase level information.The multi-scale attention module is shown in Figure 2.

heading: 3.1 MULTI-SCALE ATTENTION MODULE
text: Given a sentence S of length n, where Si (1≤i≤n) is the word in the sentence, we can get their vector representation after encoding. The encoder can be Bert or LSTM. Take Bert as an example:
H = Bert(S) (2)
Where H ∈ Rn∗d is the vectorized representation of the sentence, where d is the encoding length, and Hi is the encoding representation of each words in the sentence. In order to obtain vector representation of different scales, we use convolutional neural network to further extract information from
sentence representation. Convolutional neural network (CNN) cannot integrate global information because it is limited by the scale of receptive field of convolution kernel, but it also makes a great improvement for Convolutional neural network to extract local informations, We use this to control the length of phrase level information to be extracted by setting the scale k of convolution kernel, so as to achieve the effect of integrating phrase level information.
Convolution is an operation between the weight vector wc and the input vector of the input sequence H. The weight matrix wc is regarded as a convolution filter. We assume that the length of the filter is k, so wc ∈ Rk∗d. Convolution involves multiplying the dot product of wc by each k-gram in sequence H to obtain another sequence Hconv :
Hjconv = wc Hj−k+1:j (3)
Where means multiply element by element. In order to ensure that the length of Hconv is consistent with H, we padded the convolution operation. At the same time, due to the ability to capture different features, it is usually necessary to use multiple filters (or feature maps) in convolution. Therefore, we use d filters (Wc = w1c , w 2 c , ··, w d c ), and the convolution operation can be expressed as:
Hijconv = w i c Hj−k+1:j(0 ≤ i<d) (4)
The convolution result is matrix Hconv=[H1conv ,H 2 conv ,···,H d conv] ∈ Rd∗n. H integrates the phrase level information with K as the length. In the process of constructing the attention matrix, we refer to the construction method of Vaswani et al, and make corresponding modifications according to the characteristics of our task. The formula is as follows:
Q = H ·WQ (5) K = H ·WK (6)
Kconv = H T conv ·WK (7) ATT = softmax((Q ·KT )/ √ dk) (8)
ATTconv = softmax((Q ·KTconv)/ √ dk) (9)
The above formula is our method to construct a multi-scale attention matrix, where WQ and WK are trainable parameter matrixs with the shape of d*d. Kconv is the key matrix calculated by Hconv integrating phrase level information. The difference from the construction of Vaswani et al is that we calculated ATTconv , which is obtained by multiplying the query matrix with word as the minimum granularity and the key matrix Kconv with phrase as the minimum granularity. Its purpose is to calculate the degree of attention between words and phrases on different scales.

heading: 3.2 DYNAMIC SPARSE MODULE
text: In constructing the multi-scale attention model, we note that although there is phrase information in sentences, not all phrases with length k can be expressed as phrases. At the same time, we should also consider the differences between the traditional attention matrix and the multi-scale attention
matrix, that is, if a word chooses to pay attention to a phrase, it should reduce the attention to each word in the phrase. Therefore, we design a dynamic sparse module for our multi-scale attention module, which can be dynamically adjusted to ensure that the model can select more suitable phrase information.The dynamic sparse module is shown in Figure 3. The method is as follows.
Q1 = H ·WQ1 (10) K1 = H ·WK1 (11)
ATTcut = softmax((Q1 ·KT1 )/ √ dk) (12)
WQ1 and WK1 are two trainable parameter matrices with the shape of d * d. in the above formula, we choose to reconstruct a network for calculating similarity. Different from formula 8, this formula will be used to construct a sparse attention matrix. We set a threshold µ for att matrix to select whether the information is truncated. If the value of this position is greater than the threshold, it will be set to 1. If not, it will be set to 0. The formula is as follows:
Maskqj = 1 if(ATT qjcut > µ) else 0 (13)
Because the size of the threshold will closely affect the location of the matrix’s attention, here we set the threshold to 1 / n, so we get a mask matrix composed of 0 and 1. Because this mask matrix only uses word level attention, we also need to build a mask matrix of multi-scale attention matrix. In order to realize the difference between traditional attention matrix and multi-scale attention matrix, we choose to reverse build the mask matrix of multi-scale attention matrix according to Mask matrix. The formula is as follows: Maskconv = E −Mask (14) Where E is a matrix with all values of one, so we can obtain the corresponding sparse multi-scale attention matrix according to the two mask matrices. The formula is as follows:
ATTM = ATT Mask (15) ATTMc = ATTconv Maskconv (16)
ATTfin = [ATTM , ATTMc] (17)
ATTfin is the final attention matrix, which is obtained by splicing the traditional attention matrix and our multi-scale attention matrix in the last dimension,so it’s shape is n * 2n. ATTfin represents the correlation degree between the i-th word and the j-th word. When 1 ≤ j ≤ n, its value represents the attention value between words, and when n < j ≤ 2n, it represents the attention value between the i-th word and the (j-n)-th phrase.
V = [ H HTconv ] ·WV (18)
S = ATTfin · V (19)
The calculation method of the value matrix is shown in formula 16. For the trainable parameter matrix with the shape of d * d at WV , we splice H and Hconv in the first dimension and obtain the value matrix V with the shape of 2n * d through linear transformation.Finally, we multiply the obtained attention matrix ATTfin and value matrix V to obtain the final sentence representation. After obtaining the representation, we can use pooling or other dimensionality reduction methods to realize the final flat representation.

heading: 4 EXPERIMENT AND EVALUATION
text: We evaluated the models of multiple benchmark data sets under multiple tasks. Experiments show that our multi-scale self attention mechanism achieves better results than the traditional self attention mechanism in sentence level relationship extraction task and GLUE task.

heading: 4.1 RELATION EXTRACTION
text: Traditional methods usually deal with relationship extraction task (RE) in supervised learning mode to extract the relationship between two entities mentioned in a sentence. Each sentence begins by manually labeling two entities. Then a model is needed to predict the relationship between these
annotation entities. As there are many efforts to adopt models for this setting (Zeng et al. (2014); Zhang et al. (2015); Lin et al. (2016)) In order to verify the effectiveness of our method, we selected two relational extracted data sets for experiments, namely semeval 2010 task 8 (Hendrickx et al. (2010)) and wiki80. Semeval 2010 task 8 dataset contains 10 distinguishable relationships (causality, tool agent, product producer, content container, entity source, entity destination, component whole, member set, message subject, and others). The first nine relationships have two directions, while the remaining relationships have no direction, so the total number of relationships is 19. The data set consisted of 10717 annotated sentences with 8000 and 2717 training and test samples, respectively.
Wiki80 comes from fewrel (Han et al. (2018)), which is a large-scale few shot data set. It contains 80 relationships and 56000 instances from Wikipedia and Wiki data. Since Wiki80 is not an official benchmark, we directly report the results on the validation set.
In the experiment, we choose Bert as the encoder of our model. The experimental results show that our model is superior to the existing limit model in performance. Table 1 shows the experimental results of the model on semeval 2010 task 8 and Wiki80 data set.
From the Table 1, we can see that our model is written as ”BERT-MS”. Because we embed our model as a fine-tuning model behind the Bert model, that is, after the sentence is pre trained, the language model obtains the vector representation of each word, and then connected to our multiscale self-attention model, so as to obtain the sentence representation that can extract phrase level
information by using the information of different scales. The next step is to get the flat sentence representation through polling or other dimensionality reduction methods, and finally get the final classification result through a fully connected neural network. Compared with the traditional Bert model, we only add a multi-scale self-attention model layer after the Bert model, and our model does not make any other changes. Similarly, the model in subsequent experiments also follows this model architecture.
Compared with the previous methods that only use entity information Soares et al. (2019) or pool entity and context respectively as the guidance information of relationship classification Zhu et al. (2020), our multi-scale self-attention mechanism can better integrate the importance of entity words and context information in sentences, and better extract the interactive information between entity words and phrase in sentences, So as to improve the accuracy of model classification.
In the experiment, we use Adam optimizer to optimize the model, and the learning rate is 3e-5. It can be seen from the results that our model can better adjust the Bert model to obtain better results than the baseline.

heading: 4.2 GLUE
text: GLUE The General Language Understanding Evaluation (GLUE) benchmark is a collective task for multiple natural language understanding. The glue benchmark includes the data sets shown in Table 2, and the descriptions of these datasets were initially summarized by Wang et al. (2018)
We used Bert-base as the encoder of our model. The batch size is set to 32 and the appropriate learning rate is selected in (5e-5, 4e-5, 3e-5 and 2e-5). The final results are shown in Table 2.
We compare the Nystromformer model of Xiong et al. (2021), CHARFORMERTall model of Tay et al. (2021)and BERT-base model. Compared with the network constructed by character level information and block information composed of characters (Tay et al. (2021)), our model selects the network constructed by word and phrase information, which can better find the hidden information in sentences and has stronger interpretability. Due to the existing baseline model, it can be determined that our multi-scale self-attention model can integrate the information of different scales, so as to achieve superior experimental results.

heading: 4.3 SELECTION OF K VALUE
text: In the above, our experiments are based on k = 3, that is, the model can only extract phrase level information with length of 3. In this section, we will conduct experiments for different k values to verify the impact of k value on the performance of the model. We chose semeval 2010 task 8 to validate our experiment. Table 4 shows our experimental results. In the experiment, we use the
fusion method for phrase information with different lengths, that is, if we choose (k = 3, k = 5), it means that we need to calculate the convolution information with 3 as the convolution kernel size and 5 as the convolution kernel size at the same time. In the sparse strategy, the two choose the same sparse matrix, and then the model chooses which scale should pay more attention to. It can be seen from the table that when (k = 3, k = 5), the model can get good experimental results on semeval 2010 task 8, which shows that more phrase level information in the sentence is generally concentrated in these two lengths. When k = 7, the effect is not good, which also shows that there are few phrases of this length in this data set. This experiment does not mean that using the same strategy in all tasks can get the best effect. Different data sets have different adaptive k values, which need to be adjusted according to the characteristics of the data set to achieve the best experimental effect.

heading: 5 CONCLUSION
text: We improve the existing self attention mechanism and propose a self attention mechanism that can integrate phrase level information, which overcomes the disadvantage that the traditional attention mechanism can not effectively extract phrase level information. Due to the universality of this method, it can be extended to other application scenarios of self attention mechanism, such as translation tasks. Future work will include optimizing our model and understanding how to select features to improve its performance.
ACKNOWLEDGE
This work was supported by the Innovation Foundation of Science and Technology of Dalian under Grant No.2018J12GX045 and National Key R&D Program of China under Grant No.2018AAA0100300.

heading: A APPENDIX
text: A.1 ATTENTION VISUALIZATION

Review summaries:
The manuscript presents a multi-scale self-attention method for NLP tasks. The aim is to better extract phrase- and word-level features. The main contribution of the proposed method is to apply different kernel sizes for feature extraction and multi-scale attention fusion. Additionally, a mechanism called dynamic sparse module is applied to adjust the weights of the obtained attention matrix.

This work proposes a multi-scale fusion self-attention module to help extract phase-level at different scales. It utilizes convolution operations with kernels of different sizes to achieve this end. The authors conduct experiments on the relation extraction and GLUE tasks to demonstrate its effectiveness.