Multi-scale fusion self attention mechanism
Authors: Qibin Li
Nianmin Yao
Jian Zhao
Yanan Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Attention
multi-scale
phrase information
sparsity scheme

Abstract: Self attention is widely used in various tasks because it can directly calculate the dependency between words, regardless of distance. However, the existing self attention lacks the ability to extract phrase level information. This is because the self attention only considers the one-to-one relationship between words and ignores the one-to-many relationship between words and phrases. Consequently, we design a multi-scale fusion self attention model for phrase information to resolve the above issues. Based on the traditional attention mechanism, multi-scale fusion self attention extracts phrase information at different scales by setting convolution kernels at different levels, and calculates the corresponding attention matrix at different scales, so that the model can better extract phrase level information. Compared with the traditional self attention model, we also designed a unique attention matrix sparsity strategy to better select the information that the model needs to pay attention to, so that our model can be more effective. Experimental results show that our model is superior to the existing baseline model in relation extraction task and GLUE task.

Under review as a conference paper at ICLR 2022
MULTI-SCALE FUSION SELF ATTENTION MECHANISM
Anonymous authors Paper under double-blind review
ABSTRACT
Self attention is widely used in various tasks because it directly calculate the dependency between words, regardless of distance. However, the existing self at- tention lacks the ability to extract phrase level information. This because the self attention only considers the one-to-one relationship between words and ig- nores the one-to-many relationship between words and phrases. Consequently, we design a multi-scale fusion self attention model for phrase information to resolve the above issues. Based on the traditional attention mechanism, multi-scale fusion self attention extracts phrase information at different scales by setting convolu- tion kernels at different levels, and calculates the corresponding attention matrix at different scales, that the model can better extract phrase level information. Compared with the traditional self attention model, we also designed a unique attention matrix sparsity strategy better select the information that the model needs to pay attention to, so that our model can be more effective. Experimental results show that our model is superior to the existing baseline model in relation extraction task and GLUE task.
INTRODUCTION
Attention mechanism model widely used in natural language processing tasks. Attention de- termines where the model needs attention by constructing attention matrix. With the in-depth study attention model, Vaswani et al. (017)) proposed a more advanced self attention mechanism. The self attention model dynamically constructs the attention matrix by calculating the correlation degree between words. Compared with the traditional attention mechanism, the self attention model can construct different attention matrices for different inputs and retain more information.
Although the current self-attention model has achieved relatively successful results, we have found a serious problem, that is, self-attentic this attention matrix can pay good attention to the relevant information between one-to-one words. However, in real life, the language environment is very complicated. There often many phrases in sentences. These phrases may contain many words, but the words that make up the phrase often cannot fully express the meaning of the phrase itself. At this time, we need be able to extract the onc-to-many relationship between words and phrases. for instance:
directly caleulate dependencies between words. Although
It is embarrassing the extremes he will go to in order to impress his boss.
He has accepted a research professorship at Cambridge University.
Figure 1: Examples of phrases in sentences.
In Figure 1, we give two examples. The two parts framed in green in the sentence are two phrases in the sentence. The meaning expressed by phrases such as "in order to" and "Cambridge University" in the two sentences can not get the corresponding meaning from any word of the two phrases. Therefore, these phrases need to be regarded ensure that the information extracted by the model is correct. In previous research work, Yao et al. (019) conducted pooling operation entity words in relation extraction task to achieve the effect
whole to construct the attention matrix,
to

---

Under review as a conference paper at ICLR 2022
of phrase information coding. However, this operation has great limitations, that is, need to know the position and length of phrases in advance, which is very difficult for general tasks.
In this paper, propose to use convolution kernels of different sizes to extract phrase level infor- mation in sentences, and learn whether to interact with the sampled information adaptively by the model itself. At the same time, not all scale information is required by the model, and the conflict be- tween phrase information and its constituent words should be taken into account when selecting. For example, as shown in sentence of Figure 1, when calculating the weight of attention matrix, if the relevance of (go, in order to) is large, the attention relevance of (go, in) (go, order) (go, to) should be reduced accordingly, and vice versa. Therefore, we design unique matrix sparsity strategy, which can better adapt to our multi-scale fusion self- anff-attention model.
The experimental results show that our model has a better effect the relationship extraction task, and achieves better level than the baseline on the GLUE data set.
The main contributions of this paper include:
1. Based on the traditional self attention mechanism. the phrase level representation is extracted through sampling at different scales, and the attention matrix is constructed by using the representa- tion, which improves the deficiency that the attention model can only extract one-to-one information between words.
2. On the basis of integrating multi-scale information, in order to better guide the model for in- formation selection, a sparsity strategy of attention matrix is proposed, which can better select the information that needs to be focused when constructing attention matrix.
2 RELATED WORK
The initial attention mechanism is widely used natural language processing tasks as a model to integrate information. The initial attention mechanism is often used model to learn the association between hidden vectors after Recurrent Neural Network (RNN). Note that the emergence of the model breaks the limitation that the traditional encoder decoder structure depends an internal fixed length vector during encoding and decoding (Zhou et al. (0066)
Du al. 2018) Found in many experiments that the original single one-dimensional vector can longer meet the requirements of extracting information diversity, so they proposed to build a 2-D attention matrix adapt more complex situations. Among them, they believe that each dimension in such a 2-D level attention matrix represents a different focus direction
Vaswani et al. (017)) put forward the self attention mechanism in the transformer model. The author takes the attention model as the main structure of the model, improves the parallelism of the model, and changes the invariable characteristics of the previous attention matrix. The self attention model creates more contextual representations by designing different attention matrices for different samples.
Correia et al. (2019) considered that the complexity of the model will increase with the increase of sentence length in the operation process of self attention mechanism, which makes difficult for the model to deal with long text information. Therefore, the author puts forward some suggestions a- Entmax method to construct a sparse attention matrix. By constructing such a sparse attention matrix model, the parameters are reduced, which can adapt longer text data.
Li et al. (202)) aiming at the problem insufficient information extraction in the traditional atten- tion mechanism, proposed a method of using the hard attention mechanism to extract the important information of the model. At the same time, the author also proposed a method of adding negative information to the attention matrix to enrich the diversity of the model.
2.1
SELF ATTENTION
Before introducing our work, we first briefly introduce the self attention model Vaswani et al. 2017) in transformer.
Formally, We assume that is a vectorized representation of sentence with length n. The attention mechanism starts by projecting the tokens into three subspaces: the query subspace Q, key subspace
2

---

Under review as a conference paper at ICLR 2022
conv
ATT
multi-scale attention
Convolution
Kcomv
go to in order
to
Q
ATT
Attention
Vector representation
Figure 2: General architecture for multi-scale attention module.
K, and value subspace V with corresponding projection matrices WΩ, W WV It then computes the attention matrix A as follows.
A = softmar((Q) k²
(1)
Where de is a scaling factor, T represents transpose operation, and used to calculate the corre- lation score between each item and the query. Then, the normalized attention weight is calculated using softmax, and the obtained attention matrix A is used to weight the value of each item in each query context. Finally, the obtained attention matrix A is multiplied by vector obtain the final sentence representation.
3
METHOD
In this section, we propose a general attention model, multi-scale fusion self attention mechanism, which can pay attention to the phrase level information in the sample. Our multi-scale fusion self attention model consists of two parts: multi-scale attention module and dynamic sparse module. After obtaining the corresponding coded representation each word the input sentence, the multi-scale attention module obtains the attention matrix different scales. The dynamic sparse module constructs dynamic coefficient strategy to construct a sparse attention matrix according to the summarized attention characteristics. Finally, the constructed attention matrix is multiplied by the value matrix to obtain a sentence representation with phrase level information. The multi-scale attention module is shown in Figure 2.
3.1 MULTI-SCALE ATTENTION MODULE
Given sentence S of length where S; (ISisn) is the word in the sentence, can get their vector representation after encoding. The encoder can Bert or LSTM. Take Bert as an example:
Bert(S)
Where H Rnd is the vectorized representation of the sentence, where d is the encoding length, and H; is the encoding representation each words in the sentence. order to obtain vector represen- tation of different scales, convolutional neural network further extract information from
3

---

Under review as a conference paper at ICLR 2022
MASK
ATT
ATT,
Gate
ottention
ATT
Vector representation
Figure 3: General architecture for dynamic sparse module.
sentence representation. Convolutional neural network (CNN) cannot integrate global information because is limited by the scale of receptive field of convolution kernel, but also makes a great improvement for Convolutional neural network extract local informations, We this to control the length of phrase level information to be extracted by setting the scale k of convolution kernel, so as achieve the effect of integrating phrase level information.
Convolution is an operation between the weight vector w. and the input vector of the input sequence H. The weight matrix We is regarded convolution filter. We assume that the length of the filter is k, so We Rked Convolution involves multiplying the dot product of we by each k-gram in sequence H obtain another sequence Heone H Hoon We Hy-k+1.j
Where @ means multiply element by element. In order to ensure that the length of Heone is consis- tent with H. we padded the convolution operation. the same time, due to the ability to capture different features, it is usually necessary to use multiple filters (or feature maps) in convolution. Therefore, we use d filters (Wa=v) w/ wa we). and the convolution operation can be expressed as:
HU conn w O 11-1+115(0 i<d)
(4)
H2 The convolution result is matrix H H Comm comu:" . conn Rd H integrates the phrase level information with the length. In the process of constructing the attention matrix, we refer to the construction method of Vaswani et al, and make corresponding modifications according to the characteristics of our task. The formula is as follows:
H
Q= H WΩ
(5)
K= H W
(6)
Keonv 1TTT WK
ATT = so softmar((Q ( //
(8)
ATT ATT come = sof softmar((Q /// V de)
(9)
The above formula is our method to construct multi-scale attention matrix, where Wº and W trainable parameter matrixs with the shape of d&d. Koone is the key matrix calculated by Home integrating phrase level information. The difference from the construction of Vaswani et al is that we calculated ATT cTTonne which is obtained by multiplying the query matrix with word as the minimum granularity and the key matrix Keone with phrase as the minimum granularity. Its purpose is to calculate the degree of attention between words and phrases on different scales.
3.2 DYNAMIC SPARSE MODULE
In constructing the multi-scale attention model, we note that although there is phrase information in sentences, not all phrases with length k can expressed as phrases. At the same time, we should also consider the differences between the traditional attention matrix and the multi-scale attention
4

---

Under review as a conference paper at ICLR 2022
matrix, that is, if word chooses to pay attention to a phrase, it should reduce the attention to each word in the phrase. Therefore, we design a dynamic sparse module for our multi-scale attention module, which can be dynamically adjusted ensure that the model can select more suitable phrase information. The dynamic sparse module shown in Figure 3. The method is as follows.
Q1 И . we
(10)
K1 11 WKK
(11)
ATT = softmar (0000///1/////0/000000(( kf)/vdd))
(12)
w and W are two trainable parameter matrices with the shape of d d. in the above formula, we choose to reconstruct network for calculating similarity. Different from formula 8, this formula will be used to construct a sparse attention matrix. We set threshold μ for att matrix select whether the information is truncated. If the value of this position is greater than the threshold, it will be set to not, it will be set to O. The formula is follows:
Mask =1 :(ATTT > ())
else
(13)
Because the size of the threshold will closely affect the location of the matrix's 's attention, here set the threshold to n, so we get a mask matrix composed of and 1. Because this mask matrix only uses word level attention, we also need to build mask matrix of multi-scale attention matrix. In order to realize the difference between traditional attention matrix and multi-scale attention matrix, choose to reverse build the mask matrix of multi-scale attention matrix according Mask matrix. The formula is as follows:
Maskeonv F Mask
(14)
Where E is matrix with all values of one, attention matrix according to the two mask matrices. The formula is as follows:
obtain the corresponding sparse multi-scale
ATT ATT & Mask
(15)
ATTMe ATT cone ④ Mashiconv ATT fin ATTTM:ATTA..
(16)
(17)
ATT fin is the final attention matrix, which is obtained by splicing the traditional attention matrix and our multi-scale attention matrix in the last dimension,so it's shape is n 2n. ATT fin represents the correlation degree between the i-th word and the j-th word. When I < S n, its value represents the attention value between words, and when n < < 2n, it represents the attention value between the i-th word and the (j-n)-th phrase.
H1
V
WV
(18)
H S ATT fin V
(19)
The calculation method of the value matrix is shown in formula 16. For the trainable parameter matrix with the shape of d * d WV, we splice H and Hoone in the first dimension and obtain the value matrix with the shape of 2n through linear transformation.Fing obtained attention matrix ATT fin and value matrix V obtain the final sentence representation. After obtaining the representation, we can use pooling or other dimensionality reduction methods to realize the final flat representation.
we multiply the
EXPERIMENT AND EVALUATION
We evaluated the models of multiple benchmark data sets under multiple tasks. Experiments show that our multi-scale self attention mechanism achieves better results than the traditional self attention mechanism in sentence level relationship extraction task and GLUE task.
4.1 RELATION EXTRACTION
Traditional methods usually deal with relationship extraction task (RE) in supervised learning mode to extract the relationship between two entities mentioned in a sentence. Each sentence begins by manually labeling entities. Then a model is needed predict the relationship between these
5

---

Under review as a conference paper at ICLR 2022
SYS
SemEval
Wiki80
CNn et al. 2019)
71.1
63,9
TRE TRE All et al. 2019
87.
Spankel Jiang et al. 2019)
87.4
BERT Han et al. (0 1))
88.0
84.6
BERT--EnityySoaan
al. (2019)
88.3
86.6
AR semocul (Zhu et al. 2020)
88.5
Han al. (2020)
86.1
BERT-MS(ouu model)
88.9
87.0
Table 1: Fl value in SemEval 2010 task 8 and AUC value in Wiki80.
Corpus
Train
Test
Task
Metrics
Domain
Single-Sentence Tasks
CoLA
8.5K
IK
acceptability sentiment
Matthews corr
misc
SST-2
67k
1.8k
acc.
movie reviews
Similarity and Paraphrase Tasks
MRPC
3.7k
1.7k
paraphrase paraphrase
Fl
news
QQP
364k
391k
F1
social QA questions
Inference Tasks
MNLI
393k
20k
NLI matched
acc./mismatched acc.
misc.
QNLI
105k
5.4k
QA/NLI
acc.
Wikipedia
RTE
2.5k
3k
NLI
acc.
news, Wikipedia
Table 2: Task descriptions and statistics, All tasks are single sentence sentence pair classification.
SYS
MNLI-(m/mm)
QNLI
SST-2
CoLA
RTE
QQP
MRPC
BiLSTM+CoVe
65.4/65.7
70.8
81.9
18.5
52.7
84.9
78.7
BiLSTM+CoVe Ve+Attn
VettAAtn
68.1/68.6
72.9
80.7
8.3
56.0
83.4
80.0
BiLSTM-ELMo+AN
76.4/76.1
79.8
90.4
36.0
56.8
84.3
84.4
OpenAl GPT
82.1/81.4
87.4
91.3
45.4
56.0
70.3
82.3
Nystromformer
80.9/82.2
88.7
91.4
86.3
88.1
BERT-base
83.9/84.1
90.6
92.3
56.5
65.7
90.7
88.8
CHARFORMERT.ar
83.7/84.4
91.0
91.5
51.8
91.4
91.4
BERT-MS
83.6/84.4
91.4
93.1
58.5
65.7
91.0
91.5
Table 3: Results on GLUE tasks. We report Fl score for MRPC and QQP and accuracy for others.
annotation entities. As there many efforts adopt models for this setting (Zeng al. (2014):
Zhang et al. (2015); Lin et al. (2016)) In order to verify the effectiveness of our method, we selected relational extracted data for experiments, namely semeval 2010 task 8 (Hendrickx et al. (210)) and wiki80. Semeval 2010 task 8 dataset contains 10 distinguishable relationships (causality, tool agent, product producer, content container, entity source. entity destination, component whole, member set, message subject, and others). The first nine relationships have two directions, while the remaining relationships have no direction, so the total number of relationships is 19. The data set consisted of 10717 annotated sentences with 8000 and 2717 training and test samples, respectively,
Wiki80 comes from fewrel (Han al. (2018), which is large-scale few shot data set. contains 80 relationships and 56000 instances from Wikipedia and Wiki data. Since Wiki80 is not an official benchmark, we directly report the results on the validation set.
In the experiment, choose Bert the encoder of our model. The experimental results show that our model is superior to the existing limit model in performance. Table shows the experimental results of the model semeval 2010 task 8 and Wiki80 data set.
From the Table 1, we can see that our model written as "BERT-MS". Because we embed our model a fine-tuning model behind the Bert model, that is, after the sentence is pre trained, the language model obtains the vector representation of each word, and then connected to our multi- scale self-attention model,
obtain the sentence representation that can extract phrase level

---

Under review as a conference paper at ICLR 2022
information by using the information of different scales. The next step is to get the flat sentence representation through polling or other dimensionality reduction methods, and finally get the final classification result through fully connected neural network. Compared with the traditional Bert model, we only add multi-scale self-attention model layer after the Bert model, and our model does not make any other changes. Similarly, the model in subsequent experiments also follows this model architecture.
Compared with the previous methods that only use entity information Soares et al. (2199) pool entity and context respectively as the guidance information of relationship classification Zhu et al. (2020), our multi-scale self-attention mechanism can better integrate the importance entity words and context information in sentences, and better extract the interactive information between entity words and phrase in sentences, So to improve the accuracy of model classification.
In the experiment, we use Adam optimizer to optimize the model, and the learning rate is 3e-5. It can be seen from the results that our model can better adjust the Bert model to obtain better results than the baseline.
4.2 GLUE
GLUE The General Language Understanding Evaluation (GLUE) benchmark is a collective task for multiple natural language understanding. The glue benchmark includes the data sets shown in Table 2, and the descriptions of these datasets were initially summarized by Wang et al. (2018)
We used Bert-base the encoder of model. The batch size is set to 32 and the appropriate learning rate selected in (Se-5, 4e-5, 3e-5 and 2e-5). The final results are shown in Table 2.
We compare the Nystromformer model of ofoXi al. (0021), CHARFORMER model of Tay eet al. and BERT-base model. Compared with the network constructed by character level information and block information composed of characters (Tay al. (202T), ), model selects the network constructed by word and phrase information, which can better find the hidden information in sentences and has stronger interpretability. Due the existing baseline model, it can be determined that our multi-scale self-attention model can integrate the information of different scales, so as to achieve superior experimental results.
4.3 SELECTION OF K VALUE
In the above, our experiments are based on = that is, the model can only extract phrase level information with length of 3. In this section, we will conduct experiments for different k values to verify the impact of k value the performance of the model. We chose semeval 2010 task 8 to validate our experiment. Table 4 shows our experimental results. In the experiment, we use the
k
SemEval
88.9
k=5
88.6
k=7
88.4
k=3.k=5
89.1
k=5,k77
88.7
Table 4: Fl value of different k value.
fusion method for phrase information with different lengths, that is, if we choose = 3, 5), means that we need to calculate the convolution information with 3 as the convolution kernel size and the convolution kernel size at the same time. In the sparse strategy, the two choose the same sparse matrix, and then the model chooses which scale should pay more attention to. It can be seen from the table that when (k= 3, k == 5), the model can get good experimental results on semeval 2010 task 8, which shows that more phrase level information in the sentence is generally concentrated in these two lengths. When = 7, the effect is not good, which also shows that there are few phrases of this length in this data set. This experiment does not mean that using the same strategy in all tasks can get the best effect. Different data sets have different adaptive values, which need to be adjusted according to the characteristies of the data set achieve the best experimental effect.
7

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION
We improve the existing self attention mechanism and propose self attention mechanism that can integrate phrase level information, which overcomes the disadvantage that the traditional attention mechanism can not effectively extract phrase level information. Due to the universality of this method, be extended to other application scenarios of self attention mechanism, such as trans- lation tasks. Future work will include optimizing our model and understanding how to select features to improve its performance.
ACKNOWLEDGE
This work was supported by the Innovation Foundation of Science and Technology of Dalian under Grant N0.2018J12GX045 and National Key R&D Program of China under Grant No.2018AAA0103300.
REFERENCES
Christoph Alt, Marc Hübner, and Leonhard Hennig. Improving relation extraction by pre-trained language representations. In Proceedings AKBBC 2019, 2019.
G. M. Correia, V. Niculae, and Aft Martins. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Jinhua Du, Jingguang Han, Andy Way, and Dadong Wan. Multi-level structured self-attentions for distantly supervised relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing(EMNNLP) 2018.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-arr evaluation. In Proceedings of EMNLP., 2018.
Xu Han, Tianyu Gao, Yuan Yao, Demin Ye, Zhiyuan Liu, and Maosong Sun. Opennre: An open and extensible toolkit for neural relation extraction. In Proceedings of EMNLP-IJCNIPP: System Demonstrations, 2019.
Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. More data, more relations, more context and more openness: A review and outlook for relation extraction. In Proceedings of Asian Chapter of the Association for Computational Linguistics 2020, 2020.
I. Hendrickx, N. K. Su, Z. Kozareva, P. Nakov, and S. Szpakowicz. Semeval-2010 task 8: Multi- way classification semantic relations between pairs of nominals. Association for Computational Linguistics, 2010.
Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. Generalizing natural language analysis through span opaatoelation representations. In Proceedings of ACL2020, 2019.
Q. Li. N. Yao, Zhao, and Y. Zhang. Self attention mechanism of bidirectional information en- hancement. Applied Intelligence, pp. 1-9, 2021.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Neural relation extrac- tion with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2016.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. In Proceedings of ACL2019, 2019.
Yi Tay, Vinh Q Tran, Sebastian Ruder. Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Si- mon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106. 12672, 2021.
8

---

Under review as a conference paper at ICLR 2022
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in Neural Information Processing Systems 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystrómffrmer: A nyström-based algorithm for approximating self-attention. 2021.
Y. Yao, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, and M. Sun. Docred: A large-scale document-level relation extraction dataset. ACL2019, 2019.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 2335-2344, 2014.
Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. Bidirectional long short-term memory networks for relation classification. In Proceedings of the 29th Pacific Asia conference on lan- guage, information and computation, pp. 73-78, 2015.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. Attention- based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 207-212, Berlin, Germany, August 2016. Association for Computational Linguistics. doi:
1/P1666-//1111 URL Bol_ittts//cclaatiooooyy.oo
2034
Wei Zhu, Xipeng Qiu, Yuan Ni, and Guotong Xie. Autore: Improving bert based relation classifica- tion models via architecture search. arXiv preprint arXiv:2009. 106800 2020.
A APPENDIX
A.1 ATTENTION VISUALIZATION
This
This
new
new
airpor
airport
is
is
named
named
Kertajati
Kertajati
International
International
Airport
Airport
<EOS>
This-new-airport
new-airport-is
airport-is-named
ls-named-Kerta...
named-Kertajet..
KertajatiIntemm...
International-Ai
Airport
<EOS>
Figure 4: An example of the attention mechanism, the darker the line color in the figure, the greater the value of attention. We that model successfully pays attention the phrase level information of "Kertajati International Airport"


Reviews:
[{"review_id": "tlvRtCzAuB", "reviewer": "Reviewer_cTsu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nA simple method that uses convolutional model to enhance the localness of self-attention and phrase-level learning.\n\nWeaknesses:\n* Adding convolution into self-attention and capturing phrase information have been well studied before;\n* The proposed dynamic masking strategy has flaws;\n* Writing needs improvement;\n* Experimental details are not always clear;\n* Ablation study is incomplete and comparison should be enhanced.\n\n\nDetails:\n1)\tAdding convolutional models and capturing phrase information for self-attention has been well explored in the context of machine translation. Take the following two papers as an example:\n[1] Yang et al., Convolutional Self-Attention Networks [2] Hao et al., Multi-Granularity Self-Attention for Neural Machine Translation These existing studies reduce the novelty of this paper. Also, a direct comparison with these studies is required.\n2)\tThe masking strategy is essentially another attention layer, but its formulation indicates that no gradient will be back-propagated into W^{Q_1} and W^{K_1} as shown by Eq. 10, 11, 12, 13. The authors don’t explain how to optimize them.\n3)\tWriting needs improvement. In particularly, logistic should be improved.\n4)\tHow did you implement your convolutional model? Did you apply a vanilla convolution or depth separable convolution, or dynamic convolution?\n5)\tThe proposed model is used for fine-tuning after BERT encoding. One noticeable thing is that, although BERT outputs representation for each word, its encoding is fully contextualized. To large extent, BERT encoding contains sentence-level information rather than simply word-level. The authors need more stronger motivation to tell the necessity of convolution in self-attention.\n6)\tOne ablation is missing. What if you stack another vanilla attention layer for finetuning without convolution? This should be added for comparison.\n", "summary_of_the_paper": "This paper incorporates convolutional models into self-attention to explicitly handle word-to-phrase correlation, paired with a sparse masking strategy to balance between word-to-word attention and word-to-phrase attention. The model achieves good performance on GLUE and RE tasks.", "summary_of_the_review": "In short, the authors propose convolutional model for self-attention, which is lack of novelty. The method also has flaws, and the experiments are not very convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZEALDFyaNHK", "reviewer": "Reviewer_iGQt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Despite being motivated by the attention between phrase and word, the empirical and technical novelty is limited. \n1. The proposed method essentially adds additional transformations (and therefore parameters) in the transformer architecture, so the performance improvement is expected. It is unclear whether the proposed method is indeed better when it has the same number of parameters and operations as the baselines. \n2. The idea of combining convolution and transformer architecture has been widely adopted in prior works.\n3. It is not clear whether modeling phrases explicitly is indeed helpful, especially in a deep neural network. The model may already capture the phrases implicitly.\n\nTo improve the paper, the authors should try to\n1. Verify that the proposed architecture is indeed superior to alternative design when having similar number of parameters and operations (e.g. what's the performance w.r.t model size, can the proposed model be applied to the backbone network to achieve better performance?)\n2. Show that the superior performance is not simply the result of more parameters (e.g. what happens if we set k=1 and increase the number of filters)\n3. Show evidence about why modeling phrases explicitly is important and the proposed method indeed capture the phrase information\n4. Explain why existing model is not sufficient for capturing phrase information\n", "summary_of_the_paper": "This paper introduces a new network architecture based on a transformer. The basic idea includes\n1) Extract phrase information using a convolution operator and compute the attention between phrases and words\n2) Learn to predict a mask for word-word attention to turn off word-phrase attention when word-word attention is high\nEmpirical results show that the proposed method improves the baseline BERT model when being applied to the features extracted by BERT.\n", "summary_of_the_review": "This paper falls short of showing the benefit of the proposed method. It is unclear whether the new architecture is indeed better than existing alternatives both theoretically and empirically.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZAeiAAQvehM", "reviewer": "Reviewer_c4sd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the novelty is limited. Using convolution operation with different kernel sizes to extract information of different scales is quite general, e.g., googlenet [1]. It seems it merely transfers the idea from multi-scale context information modeling on some vision tasks to sequential modeling tasks. \nBesides, I think simply convolution operation with different kernel sizes can help to extract contextualized information. However, it seems it can not guarantee extract phase information. \nAs claimed in the paper, the dynamic sparse module is proposed to realize the end that if a word chooses to pay attention to a phrase, it should reduce the attention to each word in the phrase. However, this work simply sets the attentional values that are smaller than a threshold to 0 to realize this end. I do not why this mechanism can realize the above-mentioned end. Besides, the contribution of this point is not verified.\n\n[1] Szegedy et al., Going deeper with convolutions, in CVPR, 2015.", "summary_of_the_paper": "This work proposes a multi-scale fusion self-attention module to help extract phase-level at different scales. It utilizes convolution operations with kernels of different sizes to achieve this end. The authors conduct experiments on the relation extraction and GLUE tasks to demonstrate its effectiveness.", "summary_of_the_review": "I think the novelty is limited and some explanation is not clear.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8oSJcRzrfNe", "reviewer": "Reviewer_hsJZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution of the manuscript is minor and the novelty of the proposed method is marginal. The motivation of the proposed method is not convincing. There are many existing methods that can extract phrase or context information for NLP tasks. \n\nFor multi-scale feature extraction or attention fusion, there are also many existing studies. The proposed method just simply use different convolutional kernels for the task. It does not provide many new insights to the community.\n\nThere is not a deep analysis of the existing literature. The introduction and related work section should be completely rewritten. The information provided in the current shape is very shallow. Many state-of-the-art studies are missing. \n\nThe experimental results cannot fully support the proposed method. The authors should compare the proposed method on more NLP benchmarks with a comparison with the SOTA methods. For example, the most recent methods used for comparison were published in 2020 in Table 1.\n\nAs the proposed method is a generic module. It should be evaluated on various NLP tasks.", "summary_of_the_paper": "The manuscript presents a multi-scale self-attention method for NLP tasks. The aim is to better extract phrase- and word-level features. The main contribution of the proposed method is to apply different kernel sizes for feature extraction and multi-scale attention fusion. Additionally, a mechanism called dynamic sparse module is applied to adjust the weights of the obtained attention matrix.", "summary_of_the_review": "Overall, the proposed method is not good enough for publishing in top tier conferences such as ICLR. The contribution and novelty of the proposed multi-scale self-attention method are marginal. The experimental results cannot fully support the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-scale fusion self attention model for phrase information, which incorporates convolutional models into self-attention to explicitly handle word-to-phrase correlation. This is paired with a sparse masking strategy to balance between word-to-word attention and word-to-phrase attention. The model achieves good performance on downstream tasks.

While the proposed method is simple and looks effective, reviewers have expressed concerns with lack of novelty (see the suggested missing references), lack of clarity in the experimental details, and unclear writing. Unfortunately, there was no response from the authors, which makes me recommend rejection. We urge the authors to follow the reviewers' suggestions in a future iteration of their work.
Decision: Reject