SesameBERT: Attention for Anywhere

1 INTRODUCTION . In recent years , unsupervised pretrained models have dominated the field of natural language processing ( NLP ) . The construction of a framework for such a model involves two steps : pretraining and fine-tuning . During pretraining , an encoder neural network model is trained using large-scale unlabeled data to learn word embeddings ; parameters are then fine-tuned with labeled data related to downstream tasks . Traditionally , word embeddings are vector representations learned from large quantities of unstructured textual data such as those from Wikipedia corpora ( Mikolov et al. , 2013 ) . Each word is represented by an independent vector , even though many words are morphologically similar . To solve this problem , techniques for contextualized word representation ( Peters et al. , 2018 ; Devlin et al. , 2019 ) have been developed ; some have proven to be more effective than conventional word-embedding techniques , which extract only local semantic information of individual words . By contrast , pretrained contextual representations learn sentence-level information from sentence encoders and can generate multiple word embeddings for a word . Pretraining methods related to contextualized word representation , such as BERT ( Devlin et al. , 2019 ) , OpenAI GPT ( Radford et al. , 2018 ) , and ELMo ( Peters et al. , 2018 ) , have attracted considerable attention in the field of NLP and have achieved high accuracy in GLUE tasks such as single-sentence , similarity and paraphrasing , and inference tasks ( Wang et al. , 2019 ) . Among the aforementioned pretraining methods , BERT , a state-of-the-art network , is the leading method that applies the architecture of the Transformer encoder , which outperforms other models with respect to the GLUE benchmark . BERT ’ s performance suggests that self-attention is highly effective in extracting the latent meanings of sentence embeddings . This study aimed to improve contextualized word embeddings , which constitute the output of encoder layers to be fed into a classifier . We used the original method of the pretraining stage in the BERT model . During the fine-tuning process , we introduced a new architecture known as Squeeze and Excitation alongside Gaussian blurring with symmetrically SAME padding ( ” SESAME ” hereafter ) . First , although the developer of the BERT model initially presented several options for its use , whether the selective layer approaches involved information contained in all layers was unclear . In a previous study , by investigating relationships between layers , we observed that the Squeeze and Excitation method ( Hu et al. , 2018 ) is key for focusing on information between layer weights . This method enables the network to perform feature recalibration and improves the quality of representations by selectively emphasizing informative features and suppressing redundant ones . Second , the self-attention mechanism enables a word to analyze other words in an input sequence ; this process can lead to more accurate encoding . The main benefit of the self-attention mechanism method is its high ability to capture global dependencies . Therefore , this paper proposes the strategy , namely Gaussian blurring , to focus on local contexts . We created a Gaussian matrix and performed convolution alongside a fixed window size for sentence embedding . Convolution helps a word to focus on not only its own importance but also its relationships with neighboring words . Through such focus , each word in a sentence can simultaneously maintain global and local dependencies . We conducted experiments with our proposed method to determine whether the trained model could outperform the BERT model . We observed that SesameBERT yielded marked improvement across most GLUE tasks . In addition , we adopted a new evaluation set called HANS ( McCoy et al. , 2019 ) , which was designed to diagnose the use of fallible structural heuristics , namely the lexical overlap heuristic , subsequent heuristic , and constituent heuristic . Models that apply these heuristics are guaranteed to fail in the HANS dataset . For example , although BERT scores highly in the given test set , it performs poorly in the HANS dataset ; BERT may label an example correctly not based on reasoning regarding the meanings of sentences but rather by assuming that the premise entails any hypothesis whose words all appear in the premise ( Dasgupta et al. , 2018 ) . By contrast , SesameBERT performs well in the HANS dataset ; this implies that this model does not merely rely on heuristics . In summary , our final model proved to be competitive on multiple downstream tasks . 2 RELATED WORK . 2.1 UNSUPERVISED PRETRAINING IN NLP . Most related studies have used pretrained word vectors ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) as the primary components of NLP architectures . This is problematic because word vectors capture semantics only from a word ’ s surrounding text . Therefore , a vector has the same embedding for the same word in different contexts , even though the word ’ s meaning may be different . Pretrained contextualized word representations overcome the shortcomings of word vectors by capturing the meanings of words with respect to context . ELMo ( Peters et al. , 2018 ) can extract contextsensitive representations from a language model by using hidden states in stacked LSTMs . Generative pretraining ( Radford et al. , 2018 ) uses the ” Transformer encoder ” rather than LSTMs to acquire textual representations for NLP downstream tasks ; however , one limitation of this model is that it is trained to predict future left-to-right contexts of a unidirectional nature . BERT ( Devlin et al. , 2019 ) involves a masked language modeling task and achieves high performance on multiple natural language-understanding tasks . In BERT architecture , however , because the output data of different layers encode a wide variety of information , the most appropriate pooling strategy depends on the case . Therefore , layer selection is a challenge in learning how to apply the aforementioned models . 2.2 SQUEEZE AND EXCITATION . The Squeeze and Excitation method was introduced by Hu et al . ( 2018 ) , who aimed to enhance the quality of representations produced by a network . Convolutional neural networks traditionally use convolutional filters to extract informative features from images . Such extraction is achieved by fusing the spatial and channel-wise information of the image in question . However , the channels of such networks ’ convolutional features have no interdependencies with one another . The network weighs each of its channels equally during the creation of output feature maps . Through Squeeze and Excitation , a network can take advantage of feature recalibration and use global information to emphasize informative features and suppress less important ones . 2.3 LOCALNESS MODELING . The self-attention network relies on an attention mechanism to capture global dependencies without considering their distances by calculating all the positions in an input sequence . Our Gaussianblurring method focuses on learning local contexts while maintaining a high ability to capture longrange dependencies . Localness modeling was considered a learnable form of Gaussian bias ( Yang et al. , 2019 ) in which a central position and dynamic window are predicted alongside intermediate representations in a neural network . However , instead of using Gaussian bias to mask the logit similarity of a word , we performed Gaussian bias in the layer after the embedding layer to demonstrate that performing element-wise operations in this layer can improve the model performance . 2.4 DIAGNOSING SYNTACTIC HEURISTICS . A recent study ( McCoy et al. , 2019 ) investigated whether neural network architectures are prone to adopting shallow heuristics to achieve success in training examples rather than learning the underlying generalizations that need to be captured . For example , in computer vision , neural networks trained to recognize objects are misled by contextual heuristics in cases of monkey recognition ( Wang et al. , 2017 ) . For example , in the field of natural language inference ( NLI ) , a model may predict a label that contradicts the input because the word ” not ” , which often appears in examples of contradiction in standard NLI training sets , is present ( Naik et al. , 2018 ; Carmona et al. , 2018 ) . In the present study , we aimed to make SesameBERT robust with respect to all training sets . Consequently , our experiments used HANS datasets to diagnose some fallible structural heuristics presented in this paper ( McCoy et al. , 2019 ) . 3 METHODS . We focused on BERT , which is the encoder architecture of a multilayer Transformer ( Vaswani et al. , 2017 ) , featuring some improvements . The encoder consists of L encoder layers , each containing two sublayers , namely a multihead self-attention layer and a feed-forward network . The multihead mechanism runs through a scaled dot product attention function , which can be formulated by querying a dictionary entry with key value pairs ( Miller et al. , 2016 ) . The self-attention input consists of a query Q ∈ Rl×d , a key K ∈ Rl×d , and a value V ∈ Rl×d , where l is the length of the input sentence , and d is the dimension of embedding for query , key and value . For subsequent layers , Q , K , V comes from the output of the previous layer . The scaled dot product attention ( Vaswani et al. , 2017 ) is defined as follows : Attention ( Q , K , V ) = softmax ( QKT√ d ) · V ( 1 ) The output represents the multiplication of the attention weights A and the vector v , where A = softmax ( QK T √ d ) ∈ Rl×l . The attention weights Ai , j enabled us to better understand about the importance of the i-th key-value pairs with respect to the j-th query in generating the output ( Bahdanau et al. , 2015 ) . During fine-tuning , We used the output encoder layer from the pretrained BERT model to create contextualized word embeddings and feed these embeddings into the model . Although several methods have been developed for extracting contextualized embeddings from various layers , we believed that these methods had substantial room for improvement . Therefore , we used Squeeze and Excitation to solve the aforementioned problem . 3.1 SQUEEZE AND EXCITATION . In this study , we proposed the application of Squeeze and Excitation ( Hu et al. , 2018 ) ; its application to the output of the encoder layer was straightforward once we realized that the number of channels was equivalent to the number of layers . Therefore , we intended to use the term channels and layers interchangeably . First , we defined U : , : ,k as the output of the k-th encoder layer , for all 1 ≤ k ≤ n. We wanted to acquire global information from between the layers before feeding the input into the classifier ; therefore , we concatenated all the output from each encoder layer to form the feature maps U ∈ Rl×d×n . In the squeeze step , by using global average pooling on the kth layer , we were able to squeeze the global spatial information into a layer descriptor . In other words , we set the kth layer ’ s output of the squeeze function as Z : , : ,k. Z : , : ,k = fsq ( Uk ) = 1 l × d l∑ i=1 d∑ j=1 Ui , j , k ( 2 ) In the excitation step , we aimed to fully capture layer-wise dependencies . This method uses the layer-wise output of the squeeze operation fsq to modulate interdependencies of all layers . Excitation is a gating mechanism with a sigmoid activation function that contains two fully connected layers . Let W1 and W2 be the weights of the first and second fully connected layers , respectively , and let r be the bottleneck in the layer excitation that encodes the layer-wise dependencies ; therefore , W1 ∈ Rn× n r , and W2 ∈ R n r×n . The excitation function fex : s = fex ( z ) = σ ( ReLU ( z , W1 ) , W2 ) ( 3 ) where z is the vector squeezed from tensor Z . Finally , we rescaled the output Z : , : ,k by multiplying it by sk . The rescaled output is deonted as ũk . The scaling function fscale is defined as follows : ũk = fscale ( sk , U : , : ,k ) ( 4 ) We concatenated all rescaled outputs from all encoder layers to form our rescaled feature maps ũ . The architecture is shown in Figure 1 . We then extracted layers from the rescaled feature maps , or calculated a weighted average layer ũavg . ũavg = ∑n k=1 fscale ( sk , U : , : ,k ) ∑n k=1 sk ( 5 )

Summary: This paper proposes a novel BERT based neural architecture, SESAME-BERT, which consists of “Squeeze and Excitation” method and Gaussian blurring. “Squeeze and Excitation” method extracts features from BERT by calculating a weighted sum of layers in BERT to feed the feature vectors to a downstream classifier. To capture the local context of a word, they apply Gaussian blurring on output layers of the self-attention layer in BERT. The authors show their model’s performance on GLUE and HANS dataset.