SesameBERT: Attention for Anywhere
Authors: Ta-Chun Su
Hsiang-Chih Cheng
Venue:  (2020)
Keywords: Natural Language Processing
Deep Learning
Self Attention

Abstract: Fine-tuning with pre-trained models has achieved exceptional results for many language tasks. In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse language-understanding benchmarks. However, in many downstream tasks, information between layers is ignored by BERT for fine-tuning. In addition, although self-attention networks are well-known for their ability to capture global dependencies, room for improvement remains in terms of emphasizing the importance of local contexts. In light of these advantages and disadvantages, this paper proposes SesameBERT, a generalized fine-tuning method that (1) enables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring. Furthermore, we demonstrated the effectiveness of our approach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations. The experiments revealed that SesameBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.

Under review as a conference paper at ICLR 2020
SESAMEBERT: ATTENTION FOR ANYWHERE
Anonymous authors
Paper under double-blind review
ABSTRACT
Fine-tuning with pre-trained models has achieved exceptional results for many language tasks. In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse uugggugggundrsstaddinngg benchmarks. However, in many down- stream tasks, information between layers is ignored by BERT for fine-tuning. addition, although self-attention networks are well-known for their ability to cap- ture global dependencies, room for improvement remains in terms emphasizing the importance of local contexts. In light of these advantages and disadvantages, this paper proposes SesameBERT, generalized fine-tuning method that (1) en- ables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring. Furthermore, proach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations, The ex- periments revealed that SesTmeBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.
demonstrated the effectiveness of our ap-
INTRODUCTION
In recent years, unsupervised pretrained models have dominated the field of natural language pro- cessing (NLP). The construction of framework for such model involves two steps: pretraining and fine-tuning, During pretraining, an encoder neural network model is trained using large-scale unlabeled data learn word embeddings; parameters are then fine-tuned with labeled data related to downstream tasks.
Traditionally, word embeddings are vector representations learned from large quantities of unstruc- tured textual data such as those from Wikipedia corpora (Mikolov et al. 2013). Each word is repre- sented by an independent vector, even though many words are morphologically similar. To solve this problem, techniques for contextualized word representation (Peters et al. 2018 Devlin et al. 2019) have been developed; some have proven to be more effective than conventional word-embedding techniques, which extract only local semantic information of individual words. By contrast, pre- trained contextual representations learn sentence-level information from sentence encoders and can generate multiple word embeddings for word. Pretraining methods related to contextualized word representation, such as BERT (Devlin et al.) 2019), OpenAl GPT (Radford et al.) (2018), and ELMo (Peters al., 2018), have attracted considerable attention in the field of NLP and have achieved high accuracy in GLUE tasks such as single-sentence, similarity and paraphrasing, and inference tasks (Wang et al. 2019). Among the aforementioned pretraining methods, BERT, state-of-the-art network, is the leading method that applies the architecture of the Transformer encoder, which out- performs other models with respect the GLUE benchmark. BERT's performance suggests that self-attention is highly effective in extracting the latent meanings of sentence embeddings.
This study aimed to improve contextualized word embeddings, which constitute the output of en- coder layers to be fed into classifier. We used the original method of the pretraining stage in the BERT model. During the fine-tuning process, we introduced a new architecture known as Squeeze and Excitation alongside Gaussian blurring with symmetrically SAME padding ("SESAME" here- after). First, although the developer of the BERT model initially presented several options for its use, whether the selective layer approaches involved information contained in all layers unclear. In a previous study, by investigating relationships between layers, we observed that the Squeeze and Excitation method (Hu et al. (018) is key for focusing information between layer weights. This

---

Under review as a conference paper at ICLR 2020
method enables the network to perform feature recalibration and improves the quality of representa- tions by selectively emphasizing informative features and suppressing redundant ones. Second, the self-attention mechanism enables word analyze other words in input sequence; this process can lead to more accurate encoding. The main benefit of the self-attention mechanism method is its high ability capture global dependencies. Therefore, this paper proposes the strategy, namely Gaussian blurring, to focus on local contexts. We created a Gaussian matrix and performed convo- lution alongside fixed window size for sentence embedding. Convolution helps word focus on not only its own importance but also its relationships with neighboring words. Through such focus, each word in sentence can simultaneously maintain global and local dependencies.
We conducted experiments with our proposed method to determine whether the trained model could outperform the BERT model. We observed that SesameBERT yielded marked improvement across most GLUE tasks. In addition, we adopted a new evaluation set called HANS (McCoy et al. 2019), which was designed to diagnose the use of fallible structural heuristics, namely the lexical overlap heuristic, subsequent heuristic, and constituent heuristic. Models that apply these heuristics are guaranteed to fail in the HANS dataset. For example, although BERT scores highly in the given test set, it performs poorly in the HANS dataset; BERT may label example correctly not based on reasoning regarding the meanings of sentences but rather by assuming that the premise entails any hypothesis whose words all appear in the premise (Dasgupta et al. 2018). By contrast, SesameBERT performs well in the HANS dataset; this implies that this model does not merely rely on heuristics. In summary, our final model proved to be competitive on multiple downstream tasks.
RELATED WORK
2.1 UNSUPERVISED PRETRAINING IN NLP
Most related studies have used pretrained word vectors (Mikolov et al. 2013 Pennington et al. 2014) as the primary components NLP architectures. This is problematic because word vectors capture semantics only from word's surrounding text. Therefore, vector has the same embedding for the same word in different contexts, even though the word's meaning may be different.
Pretrained contextualized word representations overcome the shortcomings of word vectors by cap- turing the meanings of words with respect to context. ML (Peters et al. 2018) can extract context- sensitive representations from language model by using hidden states in stacked LSTMs. Genera- tive pretraining (Radford et al.) 2018) uses the "Transformer encoder" rather than LSTMs to acquire textual representations for NLP downstream tasks; however, one limitation of this model is that it is trained to predict future left-to-right contexts of unidirectional nature. BERT (Devlin et al., 2019) involves a masked language modeling task and achieves high performance on multiple natural language-understanding tasks. In BERT architecture, however, because the output data of different layers encode a wide variety of information, the most appropriate pooling strategy depends on the case. Therefore, layer selection is challenge in learning how to apply the aforementioned models.
2.2 SQUEEZE AND EXCITATION
The Squeeze and Excitation method was introduced by Hu et al. (0018), who aimed to enhance the quality of representations produced by a network. Convolutional neural networks traditionally use convolutional filters to extract informative features from images. Such extraction is achieved by fusing the spatial and channel-wise information of the image in question. However, the channels of such networks" convolutional features have no interdependencies with one another. The network weighs each of its channels equally during the creation of output feature maps. Through Squeeze and Excitation, network can take advantage of feature recalibration and use global information to emphasize informative features and suppress less important ones.
2.3 LOCALNESS MODELING
The self-attention network relies on an attention mechanism to capture global dependencies without considering their distances by calculating all the positions in an input sequence. Our Gaussian- blurring method focuses on learning local contexts while maintaining a high ability to capture long- range dependencies. Localness modeling considered learnable form of Gaussian bias (Yang
2

---

Under review as a conference paper at ICLR 2020
et al. 2019) in which central position and dynamic window predicted alongside intermediate representations in a neural network. However, instead of using Gaussian bias to mask the logit sim- ilarity of word, performed Gaussian bias in the layer after the embedding layer to demonstrate that performing element-wise operations in this layer can improve the model performance.
2.4 DIAGNOSING SYNTACTIC HEURISTICS
recent study (MeCoy et al., 2019) investigated whether neural network architectures are prone to adopting shallow heuristics to achieve success in training examples rather than learning the under- lying generalizations that need to be captured. For example, in computer vision, neural networks trained to recognize objects are misled by contextual heuristics in cases of monkey recognition Wang et al.] 2017). For example, in the field of natural language inference (NLI), a model may predict label that contradicts the input because the word "not", which often appears in examples of contradiction in standard NLI training sets, is present (Naik et al. 2018: Carmona et al. 2018). In the present study, aimed to make BeRT robust with respect to all training sets. Consequently, our experiments used HANS datasets to diagnose some fallible structural heuristics presented in this paper (McCoy et al. 2019).
METHODS
We focused on BERT, which is the encoder architecture of multilayer Transformer (Vaswani al, 2017), featuring some improvements. The encoder consists of L. encoder layers, each containing two sublayers, namely a multihead self-attention layer and a feed-forward network. The multihead mechanism runs through scaled dot product attention function, which can be formulated by query- ing a dictionary entry with key value pairs (Miller et al.) 2016). The self-attention input consists of query Q Rixd a key K Rxxd and value V E Rtdd where I is the length of the input sentence, and is the dimension of embedding for query, key and value. For subsequent layers, Q. K, V comes from the output of the previous layer. The scaled dot product attention (Vaswani al., 2017 is defined as follows:
QKT Vd ) V
Attention (Q, V) sootmax
(1)
The output represents the multiplication of the attention weights A and the vector v, where A softmar( QK"
vid E Rix! The attention weights AL, enabled us better understand about the importance the i-th key-value pairs with respect to the j-th query in generating the output Bahdanau al. 2015). During fine-tuning, We used the output encoder layer from the pretrained BERT model to create contextualized word embeddings and feed these embeddings into the model. Although several methods have been developed for extracting contextualized embeddings from var- ious layers, we believed that these methods had substantial room for improvement. Therefore, we used Squeeze and Excitation solve the aforementioned problem.
3.1 SQUEEZE AND EXCITATION
In this study, we proposed the application of Squeeze and Excitation (Hu et al. 2018);; ; its application to the output of the encoder layer was straightforward once we realized that the number of channels was equivalent to the number of layers. Therefore, we intended to use the term channels and layers interchangeably.
First, defined U.,., the output of the k-th encoder layer, for all 1 < A < We wanted to acquire global information from between the layers before feeding the input into the classifier; therefore, we concatenated all the output from each encoder layer to form the feature maps U Rlxdxn In the squeeze step. by using global average pooling on the kth layer, we were able to squeeze the global spatial information into a layer descriptor. In other words, we the kth layer's output of the squeeze function as Z...k.
Z Z...k . Uu)
U
x
3

---

Under review as a conference paper at ICLR 2020
Classifier
Layer fz=
Encoder
fp
Layer é
Encoder
:
Layer f
Encoder
Layer
Encoder
Sussts
1x1x LAGU
Imputs
Embedkdnng
Figure I: We extracted the output from each layer of the encoders and concatenated all the layers to form three-dimensional tensor U. We then performed Squeeze (U)(U) and Excitation (J (U)) to obtain the weight of each output layer. Finally, we fed the weighted average of all layers into the classifier. In this work employed 12 attention layers.
In the excitation step. we aimed fully capture layer-wise dependencies. This method uses the layer-wise output of the squeeze operation Jay to modulate interdependencies of all layers. Exci- tation is gating mechanism with sigmoid activation function that contains two fully connected layers. Let W1 and W2 be the weights of the first and second fully connected layers, respectively, and let r be the bottleneck in the layer excitation that encodes the layer-wise dependencies; therefore, Wi _=x and W2 IR X The excitation function fezi
s= fex o (ReLU(Z, W1), W2) where z is the vector squeezed from tensor Z.
Finally, we rescaled the output Z..,k by multiplying it by 8g. The rescaled output is deonted as un. The scaling function facale is defined follows:
fscute (5k, U k,,,k)
(4)
We concatenated all rescaled outputs from all encoder layers to form our rescaled feature maps û. The architecture shown in Figure ! calculated a weighted average layer Havg-
We then extracted layers from the rescaled feature maps, or
Σ Socale (Sk U U...k)
Uavg
(5)
Σk= 8k
3.2 GAUSSIAN BLURRING
Given an input sequence X = 22., Rixd_ the model transformed it into queries Q, keys K, and values V. where Q, K, and E glxd Multihead attention enabled the model to jointly attend to information from different representation subspaces at different positions. Thus, the three types of representations are split into subspaces of size to attend different information. For example, Q = (Q' Q² 2²) with Q' IRlx for all < i < h. In each subspace h, the element of in the output sequence O = (of of., is computed as follows:
of Attention(g K²) k")a
(6)
where of R
To capture the local dependency related to each word, we first used a predefined fixed window size create Gaussian blur g, where g E Rt
g(x; k) = exp(
20
where a refers to the standard deviation. Several Gaussian- strategies are feasible for apply- ing convolutional operations to attention outputs.
4

---

Under review as a conference paper at ICLR 2020
depthwise
Figure Diagram of one-dimensional Gaussian blur kernel, which was convoluted through the in- put dimension. This approach enabled the central word to acquire information concerning neighbor- ing words with weights proportional the Gaus- sian distribution.
Attendion Outpors Matrix
Gauasian Blur
Recalibrated Attention Outputs Matrix
Kernel
3.2.1 GAUSSIAN BLURRING ON ATTENTION OUTPUTS
The first strategy focuses on each attention output O², We restrict O s.j: to a local scope with fixed size i centered at the position and dimension j, where S j S d. and can be any odd number between 1 and 1, expressed as follows:
O
10 --
O O ++
(8)
We then enhance the localness of O Lj:: through a parameter-free convolution operation with g.
= O
The final attention output Oh, which the dot product between the Gaussian kernel and the corresponding input array elements at every position of O $.
0
(10)
where * defined as a convolution operation, as illustrated in Figure
More specifically, 3 the entry of Oh in the i-th row and j-th column, equals Mur(O%, (,)):
Σ
g(x: (ok,o,,)00 k))
kE(-A,A]
=
Σ
k( Σ Aipx
(11)
A.
3.2.2 GAUSSIAN BLURRING ON VALUES
Another option focuses values V. We applied the aforementioned method again but restrict V local scope. The final attention output Oh denoted as follows:
Atteetionn (Q", K (')(v' * =g)
(12)
The difference between the present method and the method of performing Gaussian blurring on attention outputs and values is that the method of performing Gaussian blurring on attention outputs and values places greater emphasis on the interaction of cross-query vectors, whereas the present method focuses on cross-value vectors. Finally, the outputs of the attention heads are concatenated to form the final output representation Ō:
Ô - 1 θ%)
(13)
where Ô E Igldd The multihead mechanism enables each head to capture distinct linguistic input properties ( et al. 2019). Furthermore, because our model is based on BERT, which builds an encoder framework with stack of 12 layers, were able to apply locality modeling to all lay- ers through Squeeze and Excitation. Therefore, we expected that the global information and local properties captured by all layers could be exploited.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
We evaluated the proposed SesameBERT model by conducting multiple classification tasks. For comparison with the results of previous study on BERT (Devlin et al.] 2019), we reimplemented the BERT model in TensorFlow in our experiments. | In addition, we set most of the parameters to be identical to those in the original BERT model, namely, batch size: 16, 32, learning rate: Se-5, 3c-5, 2c-5, and number of epochs: 3, 4. All of the results in this paper can be replicated in more than 12 hours by a graphics processing unit with nine GLUE datasets. We trained all of the models in the same computation environment with an NVIDIA Tesla V100 graphics processing unit.
4.1 GLUE DATASETS
GLUE benchmark is collection of nine natural ingguag--understandde tasks, including question- answering, sentiment analysis. identification of textual similarities, and recogmition of textual en- tailment (Wang al. 2019). GLUE datasets were employed because they are sets of tools used to evaluate the performance of models for a diverse set of existing NLU tasks. The datasets and metrics used for the experiments in this study are detailed in the appendix[A]
4.2 HANS DATASET
We used a new evaluation set, namely the HANS dataset, to diagnose fallible structural heuristics presented in previous study (McCoy al., 2019) based on syntactic properties. More specifically, models might apply accurate labels not based on reasoning regarding the meanings of words but rather by assuming that the premise entails any hypothesis whose words all appear in the premise (Dasgupta et al. 2018; Naik et al.] 2018). Furthermore, an instance that contradicts the lexical overlap heuristics MNLI is likely rare to prevent model from learning heuristics. Models may learn to assume that a label is contradictory whenever a negation word is contained in the premise but not the hypothesis (McCoy & Linzen| 2019). Therefore, whether a model scored well on given test set because it relied on heuristics can be observed. For example, BERT performed well on MNLI tasks but poorly on the HANS dataset; this finding suggested that the BERT model employs the aforementioned heuristics.
The main difference between the MNLI and HANS datasets their numbers of labels. The MNLI dataset has three labels, namely Entailment, Neutral, and Contradiction. In the HANS dataset, instances labeled as Contradiction or Neutral are translated into non-entailment. Therefore, this dataset has only two labels: Entailment and Non-entailment. The HANS dataset targets three heuris- ties, namely Lexical overlap, Subsequence, and Constituent, with more details in appendix B This dataset not only serves tool for measuring progress in this field but also enables the visualization of interpretable shortcomings in models trained using MNLI.
4.3
RESULTS
4.3.1 GLUE DATASETS RESULTS
This subsection provides the experiment results of the baseline model and the models trained using our proposed method. We performed Gaussian blurring on attention outputs in the experiment. In addition, employed batch size of 32, learning rates of 3c-5, and epochs over the data for all GLUE tasks. We fine-tuned the SesameBERT model through 9 downstream tasks in the datasets. For each task, we performed fine-tuning alongside Gaussian blur kernel sigmas 1e-2, 1e-1, 3e-1, and 5e-1 and selected that with the most favorable performance in the dev set. Because GLUE datasets do not distribute labels for test sets, we uploaded our predictions to the GLUE server for evaluation. The results are presented in Table [ GLUE benchmark is provided for reference. In most tasks, our proposed method outperformed the original BERT- model (Devlin et al. 2019). For example, in the RTE and AX datasets, Sesame BERT yielded improvements of 1.2% and 1.6%, respectively. We conducted experiments on GLUE datasets to test the effects of Gaussian blurring alongside BERT on the value layer and context layer. Table 2 shows the degrees of accuracy in the dev set. The performance of Gaussian blurring with respect to self-attention layers varied among cases.
Our code will be released upon acceptance.

---

Under review as a conference paper at ICLR 2020
Table 1: 'est results in relation the GLUE benchmark. The metrics for these tasks, shown in
appendix AA] were calculated using GLUE score. We compared our SesameBERT model with the
original BERT-Base model, ELMo (Peters al. 2018) and OpenAI GPT (Radford al. 2018). All
results were obtained from htt http://://///lueeeeemmmaaaaaa rk com/ Leaderboar
/
BiLSTM-ELLMo+Atn
OpenAl GPT
BERT- -ase
SesameBERT
CoLA
33.6
45.4
52.1
52.7
SST-2
90.4
91.3
93.5
94,2
MRPC
84.4
82.3
88.9
88.9
STS-B
72.3
80.0
85.8
85.5
QQP
63.1
70.3
71.2
70.8
MNLI-m
74.1
82.1
84.6
83.7
MNLI-mm
74.5
81.4
83.4
83.6
QNLI
79.8
88.1
90.5
91.0
RTE
58.9
56.0
66.4
67.6
AX
21.7
34.2
35.8
GLUE score
70.0
76.9
78.3
78.6
Table 2: Performance of Gaussian blurring alongside the BERT model. The results were tested on four GLUE datasets, with accuracy as the metric.
MRPC
RTE
QNLI
SST-2
BERT
86.7
65.3
88.4
92.7
Blur on Value layer
86.8
69.7
90.9
91.3
Blur on Context layer
86.5
70.4
90.8
92.0
Gong et al. (2019) demonstrated that different layers vary in terms of their abilities to distinguish and capture neighboring positions and global dependency between words. We evaluated the weights learned from all layers. These weights indicated that heavier weight represents greater importance. The results are shown in appendix C Because the lower layer represents word embeddings that are deficient in terms of context (Baosong Yang 2018), the self-attention model in the lower layer may need to encode representations with global context and may struggle to learn localness. Table shows the degree of accuracy predicted by each extracted attention output layer method. The results indicated that the lower layers had lower accuracy.
We performed three ablation studies. First, examined the performance of our method without blurring; we observed that Squeeze and Excitation helped the higher layer. This trend suggested that higher layers benefit more than do lower layers from Squeeze and Excitation. Second, an- alyzed the effect of Gaussian blurring on the context layer. The results revealed that the method with blurring achieved higher accuracy in lower layers. We assumed that capturing short-range de- pendencies among neighboring words in lower layers is an effective strategy. Even if self-attention models capture long-range dependencies beyond phrase boundaries in higher layers, modeling lo- calness remains helpful metric. Finally, we observed the direct effects of SesameBERT. Although our proposed architecture performed poorly in lower layers, it outperformed the other methods in higher layers. This finding indicated that in higher layers, using Squeeze and Excitation alongside Gaussian blurring helps self-attention models to capture global information all layers.
4.3.2 HANS DATASET RESULTS
We trained both BERT and SeS on the MNLI-m dataset to evaluate their classification ac- curacy. Similar to the results of another study (Devlin et al. 2019), BERT achieved 84.6% accuracy, which is higher than that of SesameBERT, the effects of two models on each type of heuristic. The results are presented Figure 3 we first examined heuristics for which the label was Entailment. We see that both models performed well; they assigned the correct labels almost 100% of the time, as we had expected them to do after adopting the heuristics targeted by HANS.
shown in Table In the HANS dataset, explored

---

Under review as a conference paper at ICLR 2020
Table Comparison of specified layers among various approaches in the RTE dataset. We dissected our models into two methods. SE-BERT refers to BERT with Squeeze and Excitation; BLUR-BERT refers BERT with Gaussian blurring.
Layers
BERT SE-BERT BLUR-BERT
SesameBERT
Dev Set Accuracy
First Hidden Layer
58.1
57.0
64.6
54.5
Second Hidden Layer
55.6
56.3
57.4
54.2
Second-to-Las Hidden
64.6
69.0
67.5
70.8
Last Hidden
65.3
67.9
68.6
70.4
Sum Last Four Hidden
65.0
69.3
68.2
69.7
Sum All 12 Layers
68.2
68.6
66.4
69.0
Weighted Average Layers
66.8
67.5
67.9
70.0
Next, we evaluated the heuristics labeled as Non-entailment. BERT performed poorly for all three cases, meaning that BERT assigned correct labels based heuristics instead of applying the correct rules of inference. By contrast, our proposed method performed almost three times as well as BERT in the case of "Lexical overlap".
E0M
subsecuence
lodcol overiop
lusical ovarlap
constiuent
CK
0.00
6.00
6.00
373
030 1.0
Accuracy
Accuracy
Figure 3: We compared BERT and SesameBERT for each case. Left: Results of heuristics-entailed cases. Right: Results of heuristics labeled as Nonentailment. BERT performed poorly in all three cases in Right; this indicated that the model had adopted shallow heuristics rather than learning the latent information that it intended to capture.
contrast to the results Left:
This paper argues that capturing local contexts for self-attention networks with Gaussian blurring can prevent models from easily adopting heuristies. Although our models performed poorly cases of "SSussquuence" and "Constituent", both of these heuristics may be hierarchical cases of the lexical overlap heuristic, meaning that the performance of this hierarchy would necessarily match the performance of our models (McCoy et al. 2019).
CONCLUSION
This paper proposes a fine-tuning approach named SesameBERT based on the pretraining model BERT improve the performance of self-attention networks. Specifically, we aimed find high- quality attention output layers and then extract information from aspects in all layers through Squeeze and Excitation. Additionally, we adopted Gaussian blurring to help capture local contexts. Experiments using GLUE datasets revealed that BEST outperformed the BERT baseline model. The results also revealed the weight distributions different layers and the effects of ap- plying different Gaussian-blurring approaches when training the model. Finally, used the HANS dataset to determine whether our models were learning what we wanted them to learn rather than using shallow heuristies. We highlighted the of lexical overlap heuristics the BERT model. SesameBERT could be further applied to prevent models from easily adopting shallow heuristics.
advantage
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
Derek F. Wong Fandong Meng Lidia S. Chao Tong Zhang Baosong Yang, Zhaopeng Tu. Modeling localness for self-attention networks. EMNLP, 2018.
V. Ivan Sanchez Carmona, Jeff Mitchell, and Sebastian Riedel. Behavior analysis of nli models: Uncovering the influence of three factors on robustness. NAACL, 2018.
Ishita Dasgupta, Demi Guo, Andreas Stuhlmuller, Samuel J. Gershman, and Noah D. Goodman. Evaluating compositionality in sentence embeddings. 2018.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Ticyan Liu. Efficient training of bert by progressively stacking. ICML, 2019.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-exciiation networks. CVPR, 2018.
Jian Li, Baosong Yang. Zi-Yi Dou, Xing Wang, Michael R. Lyu, and Zhaopeng Tu. Information aggregation for multi-head attention with by-ge-eem-..ee.. NAACL, 2019.
R. Thomas McCoy and Tal Linzen. Non-entailed subsequences challenge for natural language inference. SCiL, 2019.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. NIPS, 2013.
Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston, Key-value memory networks for directly reading documents. 2016.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. COLING, 2018.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. EMNLP, 2014.
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. NAACI,, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ICLR, 2019.
Jiany Wang, Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, and Alan Yuille. Visual concepts and compositional voting. Annals of Mathematical Sciences and Applications, 2017.
Baosong Yang. Longyue Wang, Derek Wong, Lidia S. Chao, and Zhaopeng Tu. Convolutional self-attention networks. NAACL, 2019.

---

Under review as a conference paper at ICLR 2020
A DESCRIPTIONS OF GLUE DATASETS
Table 4: Descriptions of GLUE tasks. The second and third column denote the sizes the corre-
sponding corpora. All tasks classification tasks, except for STS-B, which is regression task.
Corpus
#Train
#Test
Task
Metries
Domain
Single Sentence Tasks
CoLA
8,5k 67k
Ik
acceptability
Matthews correlation
misc
SST-2
1.8k
sentiment
Accuracy
movie reviews
Similarity/ Paraphrase Tasks
MRPC
3.7k
1.7k
paraphrase
Accuracy/F1
news misc
STS-B
7k
1.4k
sentence similarity
Pearson/Spearman corr.
QQP
364k
391k
paraphrase
Accuracy/F1
social QA
Inference Tasks
MNLI
3933
20k
NLI
Accuracy
misc Wikipedia
QNLI
105k
5.5k
QA/LII
Accuracy Accuracy
RTE
2.5k
3k
NLI
Wikipedia news, paper, ctc
AX
1.1k
NLI
Matthews correlation
B DESCRIPTION OF HANS DATASET
Table 5: Three types of heuristics targeted by the HANS dataset. The examples show incorrect entailment predictions that would result from targeting these heuristics.
Heuristic
Definition
Example
Lexical overlap
Assume that premise entails all hypotheses
The docter paid the actor.
constructed from words premise
WRONG doctor paid the actor.
Subsequence
Assume that premise entails all of its contigu- subsequences.
The doctor near the actor danced. The actor danced.
WRONG
Constituent
Assume that a premise entails all complete sub- trees in its parse tree.
If the artist slept, the actor ran. The artist slept.
WRONG
C LAYER WEIGHTS CALCULATED BY SQUEEZE AND EXCITATION
1.0
0.8
0.6
-0.2
-0.0
Layer
Layer
Layer 9
Layer 12
Figure 4: Evaluation of the weights calculated by Squeeze and Excitation for all layers, with the RTE dataset as an example.
10


Reviews:
[{"review_id": "rkeEfJhCKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes adding two mechanisms to the BERT architecture for NLU. The first is based on integrating information from all layers of the encoder via a method called Squeeze and Excitation. The second uses Gaussian blurring to encourage information sharing among neighboring words. The proposed method improves modestly on BERT on the GLUE suite of problems. It also substantially improves on BERT with respect to a class of examples that are designed to confound models that learn superficial heuristics based on word occurrence.\n\nI learn toward rejecting this paper. The method shows some performance gains over BERT on some GLUE tasks, but these are fairly small for the most part, and BERT outperforms the proposed method by a similar amount on a similar number of tasks. The strongest result is the HANS \"lexical_overlap\" case, where the proposed method has a clear advantage. I have no experience with these kinds of NLU models, so I can't say with confidence whether the architectural additions proposed are well-motivated, but to me it feels like there is not a strong justification for adding these particular features to the BERT architecture, and the results do not clearly demonstrate their utility except in the \"lexical_overlap\" case.\n\nDetails / Questions:\n* It seems to me that the GLUE results might be within the margin of error. Is it feasible to replicate training with different random seeds to see what the variance in the performance numbers might be? I suspect that a statistical analysis [1] might conclude that BERT and the proposed method are indistinguishable on the GLUE suite.\n\n* Were the proposed architectural additions conceived with the HANS \"counterexamples\" in mind (i.e. is there a specific reason to think that these types of methods would avoid the \"superficial\" reasoning that these examples are supposed to reveal)? Were other methods of adding context considered?\n\n* I suggest using the same x-axis scale on the two charts in Figure 3 to avoid confusion about the magnitudes of the differences.\n\nReferences:\n[1] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1-30.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJezoUU0tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes fine-tune methodologies for BERT-like models (namely, SeasameBERT).  This includes a method that considers all BERT layers and captures local information via Gaussian blurring. The methods were evaluated on several baseline datasets (e.g., GLUE, HANS)\n\nStrengths: \n\n* The paper is easy to follow. \n\n*  Squeeze-and-extraction was used to incorporate all hidden layers instead of the common-practice of averaging last 4-layers. I find it both logical and useful. \n\n* The suggested gaussian blurring method is able to capture local dependencies, which is missing in attention-based transformer layer.\n\n*  SesameBERT improves performance on some GLUE metrics and on HANS dataset. Also ablation analysis suggests squeeze-and-extraction is a good technique to extract features from BERT model compared to other common practices.  \n\n\nWeaknesses:\n\n* In my opinion, the paper novelty is not significant enough. Although useful, the suggested techniques are based on existing methods. \n\n*  Incorporate spatial/context-information is usually done by concatenating a location-based embedding with the original word embedding. I’m curious if the blurring Gaussian will be as useful compared to such version. \n\n* Since the suggested methods are generic, It can be more convincing to see results on recent models, and not only BERT. Currently, the results are not significantly better.  \n\n* The HANS DATASET RESULTS section seems rushed, will be good to elaborate more about HANS. also the first sentences of the section discusses GLUE results not HANS. \n\nTo conclude: The paper is easy to follow, suggests two nice methods for fine-tune BERT. But although useful, the suggested methods are not novel enough. The performance does not significantly improves, and the methods are applied only to BERT model. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1g_66GAKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel BERT based neural architecture, SESAME-BERT, which consists of “Squeeze and Excitation” method and Gaussian blurring. “Squeeze and Excitation” method extracts features from BERT by calculating a weighted sum of layers in BERT to feed the feature vectors to a downstream classifier. To capture the local context of a word, they apply Gaussian blurring on output layers of the self-attention layer in BERT. The authors show their model’s performance on GLUE and HANS dataset.\n\nStrengths\n*This paper claims the importance of the local context of a word and shows an effect of their method on the various datasets: GLUE, and HANS.\n\nWeaknesses\n* It seems like the self-attention layer can learn the local context information. Finding important words and predicts contextual vector representation of a word is what self-attention does.\nSo, if using local-context information, which is information in important near words, is an important feature for some downstream tasks, then the self-attention layer can learn such important near words by training the key, query, and value weight parameters to connect the near important words.\nIt would be nice if the authors provide some evidence that self-attention can't learn such a local-context feature.\n\n*In table 1, their experimental results show a slight improvement by using their method, but it's not significant.\n\n* On HANS dataset, they show using local-context can prevent models from easily adopting heuristics. How Gaussian blurring can prevent that problem? More explanation about the relation between local-context and adopting heuristics is required.\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a few architectural modifications to the BERT model for language understanding, which are meant to apply during fine-tuning for target tasks. 

All three reviewers had concerns about the motivation for at least one of the proposed methods, and none of three reviewers found the primary experimental results convincing: The proposed methods yield a small improvement on average across target tasks, but one that is not consistent across tasks, and that may not be statistically significant.

The authors clarified some points, but did not substantially rebut any of the reviewers concerns. Even though the reviewers express relatively low confidence, their concerns sound serious and uncontested, so I don't think we can accept this paper as is.
Decision: Reject