Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

1 INTRODUCTION . Sequence-to-Sequence ( Seq2Seq ) learning ( Sutskever et al. , 2014 ) has advanced the state of the art in various natural language processing ( NLP ) tasks , such as machine translation ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ; Wu et al. , 2019 ) , text summarization ( Wang et al. , 2019b ; Zhang et al. , 2020 ) , and grammatical error correction ( Kiyono et al. , 2019 ; Kaneko et al. , 2020 ) . Seq2Seq models are generally implemented with an encoder-decoder framework , in which a multi-layer encoder summarizes a source sequence into a sequence of representation and another multi-layer decoder produces the target sequence conditioned on the encoded representation . Recent studies reveal that fusing the intermediate encoder layers ( EncoderFusion ) is beneficial for Seq2Seq models , such as layer attention ( Bapna et al. , 2018 ) , layer aggregation ( Dou et al. , 2018 ; Wang et al. , 2019c ) , and layer-wise coordination ( He et al. , 2018 ) . Despite its effectiveness , not much is known about how fusing encoder layer representations work . The intuitive explanation is that fusing encoder layers exploits surface and syntactic information embedded in the lower encoder layers ( Belinkov et al. , 2017 ; Peters et al. , 2018 ) . However , other studies show that attending to lower encoder layers ( excluding the encoder embedding layer ) does not improve model performance ( Domhan , 2018 ) , which is conflicted with existing conclusions . It is still unclear why and when fusing encoder layers should work in Seq2Seq models . This paper tries to shed light upon behavior of Seq2Seq models augmented with EncoderFusion method . To this end , we propose a novel fine-grained layer attention to evaluate the contribution of ∗Work was done when Xuebo Liu and Liang Ding were interning at Tencent AI Lab . individual encoder layers . We conduct experiments on several representative Seq2Seq NLP tasks , including machine translation , text summarization , and grammatical error correction . Through a series of analyses , we find that the uppermost decoder layer pays more attention to the encoder embedding layer . Masking the encoder embedding layer significantly drops model performance by generating hallucinatory ( i.e . fluent but unfaithful to the source ) predictions . The encoded representation of the standard Seq2Seq models ( i.e . w/o fusing encoder layers ) may not have enough capacity to model both semantic and surface features ( especially at the encoder embedding layer ) . We call the problem described above the source representation bottleneck . Based on this observation , we simplify the EncoderFusion approaches by only connecting the encoder embedding layer to softmax layer ( SurfaceFusion ) . The SurfaceFusion approach shortens the path distance between source and target embeddings , which can help to learn better bilingual embeddings with direct interactions . Experimental results on several Seq2Seq NLP tasks show that our method consistently outperforms both the vanilla Seq2Seq model and the layer attention model . Extensive analyses reveal that our approach produces more aligned bilingual word embeddings by shortening the path distance between them , which confirm our claim . Our main contributions are as follows : • We introduce a fine-grained layer attention method to qualitatively and quantitatively evaluate the contribution of individual encoder layers . • We demonstrate that the encoder embedding layer is essential for fusing encoder layers , which consolidates conflicted findings reported by previous studies . • We propose a simple yet effective SurfaceFusion approach to directly exploit the encoder embedding layer for the decoder , which produces more expressive bilingual embeddings . 2 PRELIMINARIES . 2.1 SEQUENCE-TO-SEQUENCE LEARNING . Seq2Seq learning aims to maximize the log-likelihood of a target sequence y = { y1 , . . . , yJ } conditioned on a source sequence x = { x1 , . . . , xI } , which is formulated as : ŷ = arg max logP ( y|x ) . Typically , Seq2Seq learning can be implemented as various architectures ( Bahdanau et al. , 2015 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ; Wu et al. , 2019 ) , among which the Transformer ( Vaswani et al. , 2017 ) has advanced the state of the art . Without loss of generality , we introduce Transformer as the testbed in this paper . Transformer consists of an encoder E equipped with N identical layers to map the source sequence x into distributed representations , based on which a decoder D equipped with M identical layers generates the target sequence y : XN = E ( X0 ) N : = n=1 FFN ( ATT ( Xn−1 , Xn−1 , Xn−1 ) ) ( 1 ) YM = D ( Y0 , XN ) M : = m=1 FFN ( ATT ( ATT ( Ym−1 , Ym−1 , Ym−1 ) , XN , XN ) ) ( 2 ) where X0 denotes the sum of the word embeddings Xemb and position embeddings Xpos of x , Y0 denotes that of the shifted right y , FFN ( · ) denotes a position-wise feed-forward network , and ATT ( · ) denotes a multi-head dot-product attention network with three arguments–query , key and value . Residual connection ( He et al. , 2016 ) and layer normalization ( Ba et al. , 2016 ) are used in each sub-layer , which are suppressed in Equation 1 and 2 for clarity . Finally , the output representation YM of the decoder is projected into the probability P ( y|x ) , which is optimized during model training . 2.2 EXPERIMENTAL SETUP . To validate the universality of source representation bottleneck in Seq2Seq models , we conducted experiments on three representative tasks , which vary from the distance between input and output domains and the scale of training data : Machine translation takes a sentence in one language as input , and outputs a semantically-equivalent sentence in another language . We conducted experiments on three benchmarking datasets : smallscale WMT16 Romanian-English ( Ro-En ; 0.6M instances ) , medium-scale WMT14 English-German ( En-De ; 4.5M instances ) , and large-scale WMT14 English-French ( En-Fr ; 36.0M instances ) . The tokenized BLEU score ( Papineni et al. , 2002 ) was used for all the translation tasks . Text summarization takes a long-text document as input , and outputs a short and adequate summary in the same language . We used the CNN/Daily Mail corpus ( 0.3M instances ) . We evaluated with the standard ROUGE metric ( Lin , 2004 ) , i.e . Rouge-1 , Rouge-2 , and Rouge-L. Grammatical error correction takes a sentence with grammatical errors as input , and outputs a corrected sentence . We used CONLL14 datasets as the testbed ( 1.4M instances ) . The MaxMatch ( M2 ) scores ( Dahlmeier & Ng , 2012 ) were used for evaluation with precision , recall , and F0.5 values . The machine translation task has distant input/output domains ( i.e . in different languages ) , while the other tasks have similar input/output domains ( i.e . in the same language ) . We used Transformer ( Vaswani et al. , 2017 ) as the Seq2Seq model . Details of the datasets and model training are listed in Appendix A.1 . 3 BEHAVIOR OF ENCODERFUSION . In this section , we first formulate our research hypothesis of source representation bottleneck ( §3.1 ) that EncoderFusion expects to solve . In the following subsections , we propose a fine-grained layer attention model ( §3.2 ) to validate our hypothesis on well-designed experiments ( §3.3 ) . 3.1 SOURCE REPRESENTATION BOTTLENECK . Seq2Seq models learn more abstract features with the increase of layer level ( i.e . X0 → XN and Y0 → YM ) ( Belinkov et al. , 2017 ) . It has been extensively validated that a reasonable use of both the abstract representations ( at higher-level layers ) and the surface representations ( at lower-level layers ) is beneficial for various NLP ( Lu & Li , 2013 ; Hu et al. , 2014 ; Dou et al. , 2018 ; Peters et al. , 2018 ) and CV ( Long et al. , 2014 ; Pinheiro et al. , 2016 ; Lin et al. , 2017 ; Chen et al. , 2018a ) tasks . However , the Seq2Seq decoder only takes the abstract representations at uppermost layer XN as input ( Equation 2 ) , while ignores other usefully surface representations at other layers Xn ( n < N ) . Although XN has encoded surface features from low-level representations through layer-by-layer abstraction and residual connections , we hypothesize that its limited representation capacity may not sufficiently model those surface features from lower encoder layers , especially the embedding layer . We call such an issue as source representation bottleneck . 3.2 FINE-GRAINED LAYER ATTENTION . For each decoder layer , layer attention ( Bapna et al. , 2018 ; Peters et al. , 2018 ) assigns normalized scalar weights to all encoder layers , providing a direct way for evaluating the contributions made by each encoder layer . However , the capacity of a simple scalar weight is limited , leading to insufficient evaluation of the contributions . Motivated by fine-grained attention ( Choi et al. , 2018 ) that each element of a context vector receives an individual attention weight , we propose a fine-grained layer attention model to combine the advantages of both techniques . This allows us to more convincingly evaluate the contribution of individual encoder layer to the model performance . Besides , the nature of fine-grained attention enables us to give in-depth analyses of the representation power in §3.3 . Specifically , we replace the layer-agnostic source representation XN with the layer-aware representation Sm for each decoder layer Ym , which is calculated as : Sm = N∑ n=0 ŵm , n Xn , ŵm , n = [ ŵm , n,1 , . . . , ŵm , n , D ] , ŵm , n , d = exp ( wm , n , d ) ∑N n′=0 exp ( w m , n′ , d ) where denotes an element-wise multiplication , and wm , n , d denotes an element in the learnable attention weight W ∈ RM× ( N+1 ) ×D , where D is the dimensionality of the source representation . When n = 0 , we use the word embeddings Xemb without position embeddings as X0 , which has been empirically proved effective . We applied a regularization technique – DropConnect ( Wan et al. , 2013 ) to the attention weight W for a stable training , which randomly drops each wm , n , d with a probability p and divides W by 1− p. We set it to 0.3 for all the experiments . Table 2 lists the results . The proposed fine-grained layer attention model consistently outperforms the vanilla Transformer across Seq2Seq tasks , demonstrating the benefit of fusing surface features at lower-level layers . We evaluated several EncoderFusion methods in Table 1 , including layer aggregation ( Dou et al. , 2018 ) , layer-wise coordination ( He et al. , 2018 ) , and coarse-grained layer attention ( Bapna et al. , 2018 ) . Their results are respectively 34.05 , 34.19 , and 34.32 , which are all lower than that of fine-grained layer attention ( 34.45 ) . Based on these experimental results , we thus choose fine-grained layer attention as a representative of EncoderFusion in the following analyses . 3.3 BEHAVIOR CHANGES ACROSS ENCODER LAYERS . In this section , we investigate whether the surface features at lower encoder layers ( especially the encoder embedding layer ) contribute to the model performance via carefully designed experiments . Visualization of layer attention We first visualize the learned layer attention distribution in Figure 1 , in which each weight is the averaged attention weights over all dimensions . Generally , a higher weight denotes more contribution of an encoder layer to the corresponding decoder layer . Clearly , in all tasks higher decoder layers especially the uppermost ones pay more attention to the encoder embedding layer , which indicates that the surface representations potentially bring some additional useful features to the model performance . Voita et al . ( 2019 ) ; Wang & Tu ( 2020 ) reveal that the upper layers of decoder are responsible for the translation part while the lower layers for the language modeling part . Similarly , our results show that surface representations might play an important role in learning to translate source tokens . Among the Seq2Seq models , there are still considerable differences in the attention heatmaps . In the summarization model , almost all decoder layers focus more on the encoder embedding layer , while in the other two models the intermediate decoder layers pay more attention to the higher-level encoder layers . This is consistent with the findings of Rothe et al . ( 2019 ) , in which they reveal that the summarization task , as a typical extractive generation task , tends to use more surface features to generate extractive summaries . In contrast , both machine translation and error correction tasks require a large amount of syntactic and semantic information , which are generally embedded in higher-level encoder layers ( Peters et al. , 2018 ) . However , we still can not conclude that source representation bottleneck does exist in Seq2Seq models , since the surface features might act as a noise regularizer to improve the robustness of encoder output representations . To dispel the doubt , we further design two experiments to directly evaluate the effectiveness of surface features at the encoder embedding layer . Contribution of individual encoder layer In this experiment , we quantitatively analyze the behaviors change of a trained Seq2Seq model when masking a specific encoder layer ( i.e . turning its attention weight to zero and redistribute the other attention weights ) . Note that the masking operation does not affect the information flow of encoding calculation , i.e . keeping Equation 1 unchanged . Length -6 % 0 % 6 % Translation GEC Summarization Emb 1 2 3 4 5 6 Performance R el at iv e C ha ng e -4 % -2 % 0 % Translation GEC Summarization Emb 1 2 3 4 5 6 Length -6 % 0 % 6 % Translation GEC Summarization Emb 1 2 3 4 5 6 Performance R el at iv e C ha ng e -4 % -2 % 0 % Translation GEC Summarization Emb 1 2 3 4 5 6 Figure 2 ( a ) shows the contribution of individual encoder layer to model performance . As seen , masking the encoder embedding layer seriously harms the model performance in all tasks , which confirms our claim that the surface features in the embedding layer are essential to Seq2Seq models . Figure 2 ( b ) shows the results on the output length . Masking the encoder embedding layer consistently increases the length of generated output , which is especially significant for the summarization model . One possible reason is that the instances in translation and correction tasks have similar input/output lengths , while the summarization instances have distant input/output lengths . By analyzing the model outputs , we found that the Seq2Seq models tend to generate some hallucinatory ( i.e . fluent but unfaithful to the source ) predictions ( Lee et al. , 2019 ; Wang & Sennrich , 2020 ) when masking the embedding layer . Taking the correction task for an example , a right prediction “ anyone ” was replaced by the hallucinatory prediction “ friends of anyone ” in the masked model , in which the corresponding source contains no information related to “ friends ” . This issue becomes worse in the summarization task , since the hallucinatory prediction is more likely to be a sentence . The additional hallucinations will increase the output length and reduce the model performance . In addition , Lee et al . ( 2019 ) point out that even if hallucinations occur only occasionally , the Seq2Seq model may evidently lose user trust than other prediction problems , indicating the importance to fuse surface features at the embedding layer . More cases are studied in Appendix A.2 . Expressivity of attended dimensions in the encoder embedding layer As shown in Figure 1 , the uppermost decoder layer pays most attention to the encoder embedding layer ( i.e . the lower right corner ) . If the embedding layer acts as a noise regularizer , the layer dimensions would be randomly attended by the fine-grained model ; otherwise , the dimensions of higher attention weights should be distinguished from the other dimensions . Starting from this intuition , we reordered the dimensions of the encoder embedding layer according to the attention weights ŵM,0 , and split it into two equal sub-embedding matrices , i.e . more attended dimensions and less attended dimensions . We compared the expressivity of the two sub-embedding matrices by the commonly-used singular value decomposition ( Gao et al. , 2019 ; Wang et al. , 2019a ; Shen et al. , 2020 ) , in which higher normalized singular values denote that the embedding is more uniformly distributed , thus are more expressive . The singular values are normalized by dividing them by the largest value and their log scale values are reported for better clarity . Figure 3 depicts the singular value results . For comparison , we also report the values of the randomly selected dimensions . Clearly , the more attended dimensions are most expressive , while the less attended dimensions are least expressive . These results demonstrate that the fine-grained attention model indeed extracts useful surface information from the encoder embedding layer , which does not play the role of a noise regularizer . From the above experiments , we prove that the encoder embedding layer indeed provides useful surface information , which is not fully exploited by the standard Seq2Seq models .

Summary: The authors perform a thorough analysis of encoder fusion for Transformers: which encoder layer should the N-th decoder layer attend to? It turns out that the final decoder layers often attend to the encoder embeddings, leading the authors to provide them to the last decoder layer which leads to small improvements of performance on machine translation, summarization and grammar correction tasks. These are nice results, but the gains are small and the models are tested in the very basic configurations. These tasks and techniques, as well as some numbers used to claim state-of-the-art, are from a few years ago (e.g., SOTA on en-de translation is higher currently than the authors claim and higher than their number). It would be interesting to see if the presented conclusions hold for larger models - esp. for a T5 Transformer on masked language modeling, as this would be a more commonly used model in 2020. Unluckily, it is quite possible that increased activation size may negate the benefits of the authors' technique. It may well though make it even more important -- it would be really good to know! Lacking these experiments, we cannot recommend acceptance at this point.