Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
Authors: Xuebo Liu
Longyue Wang
Derek F. Wong
Liang Ding
Lidia S. Chao
Zhaopeng Tu
Venue: ICLR 2021 Poster (2021)
Keywords: Encoder layer fusion
Transformer
Sequence-to-sequence learning
Machine translation
Summarization
Grammatical error correction

Abstract: Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embeddings. Source code is freely available at https://github.com/SunbowLiu/SurfaceFusion.

Published as a conference paper at ICLR 2021
UNDERSTANDING AND IMPROVING ENCODER LAYER FUSION IN SEQUENCE-TO-SSQUENNE LEARNING
Xuebo Liu'; ; Longyue Wang", Derek F. Wong', Liang Ding", Lidia S. Chao' & Zhaopeng Tu?
CNLP"CT Lab, Department of Computer and Information Science, University of Macau Tencent AT Lab The University of Sydney nlp2ct. xuebo q i 1. com, (vinnylywang, u @tencent........................ com, [derekfw, lidiase @um. edu. com, 1din30978sydnee edu. au
ABSTRACT
Encoder layer fusion (EncoderFusion) a technique to fuse the encoder lay- ers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, is still not entirely clear why and when EncoderFusion should work. In this paper, main contribu- tion is to take step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention the encoder embedding layer across NLP tasks. Based on this obser- vation, we propose a simple fusion method, SurfaceFusion. by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NI.P benchmarks, including machine translation, text summarization, and grammatical error correction. It ob- tains the state-of-theaan art performance English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns expressive bilingual word embeddings by building closer relationship between relevant source and target embeddings. Source code is freely available at https::/gi com SunbowLin wtabbbowwi/SSufffaooooc wwt.:///uuuaauuuiiooooooooo
WMT16 Aemaanin-EEnglsse and WMT14 14
INTRODUCTION
Sequeenceto-Sequueee (Seq2Seq) learning (Sutskever et al. 2014) has advanced the state of the art in various natural language processing (NLP) tasks, such as machine translation (Bahdanau et al.) 2015 Vaswani et al. 2017 Wu al. 2019), text summarization (Wang et al. 2019b] Zhang et al. 2020), and grammatical error correction (Kiyono et al. 2019 Kaneko et al. 2020). Seq2Seq models
generally implemented with encoder-decoder Tramework, in which multi-layer encoder summarizes a source sequence into sequence of representation and another multi- decoder produces the target sequence conditioned on the encoded representation.
Recent studies reveal that fusing the intermediate encoder layers (EncoderFusion) is beneficial for Seq2Seq models, such as layer attention (Bapna al., 2018), layer aggregation (Dou et al., 2018 Wang al.) 2009c), and layer- wise coordination (He et al., 2018). Despite its effectiveness, not much is known about how fusing encoder layer representations work. The intuitive explanation that fusing encoder layers exploits surface and syntactic information embedded in the lower encoder layers (Be- linkov et al., 2017; Peters et al. 2018). However, other studies show that attending to lower encoder layers (excluding the encoder embedding layer) does not improve model performance (Domhan 2018), which is conflicted with existing conclusions. still unclear why and when fusing encoder layers should work in Seq2Seq models.
This paper tries to shed light upon behavior of Seq2Seq models augmented with EncoderFusion method. To this end, we propose novel fine--rriieed layer attention to evaluate the contribution of
Work was done when Xuebo Liu and Liang Ding were interning at Tencent AT Lah.

---

Published as a conference paper at ICLR 2021
individual encoder layers. We conduct experiments several representative Seq2Seq NLP tasks, including machine translation, text summarization, and grammatical eπror correction. Through a series of analyses, find that the uppermost decoder layer pays more attention the encoder embedding layer. Masking the encoder embedding layer significantly drops model performance by generating hallucinatory (i.e. fluent but unfaithful to the source) predictions. The encoded representation of the standard Seq2Seq models (i.e. w/o fusing encoder layers) may not have enough capacity to model both semantic and surface features (especially at the encoder embedding layer). We call the problem described above the source representation bottleneck.
Based on this observation, we simplify the EncoderFusion approaches by only connecting the encoder embedding layer to softmax layer (SurfaceFusion). The SurfaceFusion approach shortens the path distance between source and target embeddings, which can help to learn better bilingual embeddings with direct interactions. Experimental results on several Seq2Seq NLP tasks show that our method consistently outperforms both the vanilla Seq2Seq model and the layer attention model. Extensive analyses reveal that our approach produces more aligned bilingual word embeddings by shortening the path distance between them, which confirm our claim.
Our main contributions are as follows:
We introduce a fine-grained layer attention method to qualitatively and quantitatively evaluate the contribution of individual encoder layers.
We demonstrate that the encoder embedding layer is essential for fusing encoder layers, which consolidates conflicted findings reported by previous studies.
We propose a simple yet effective Surface Fusion approach to directly exploit the encoder embedding layer for the decoder, which produces more expressive bilingual embeddings.
2 PRELIMINARIES
2.1
SEQUENCE-TO-SEQUREE LEARNING
Seq2Seq learning aims to maximize the log- likelihooh of a target sequence y = (y ................................ ys} condi-
tioned on source sequence x {.1,.... )r). which is formulated as: ý = arg max log P(y x Typically, Seq2Seq learning can implemented as various architectures (Bahdanau et al.) 2015 Gehring et al. 2017 Vaswani et al., [017 Wu et al. [211)) among which the Transformer (Vaswani ct al.] (017) (17) has advanced the state of the art. Without loss of generality, we introduce Transformer as the testbed in this paper. Transformer consists of an encoder & equipped with N identical layers to map the source sequence x into distributed representations, based on which a decoder D equipped with M identical layers generates the target sequence y:
X =((X))
FFN (ATT(X" _x-111 xx-1))
(1)
M = X
FFN (ATT (ATT(Y'-- y ym-l), X. x))
(2)
where X denotes the sum of the word embeddings Xemb and position embeddings X pos of x, Y denotes that of the shifted right y, FFN(.) denotes a position-wise feed-forward network, and ATT(-) denotes a multi-head dot-product attention network with three arguments-query, key and value. Residual connection (He et al. 2016) and layer normalization (Ba et al., 2016) are used in each sub-layer, which are suppressed in Equation and 2 for elarity. Finally, the output representation Y of the decoder is projected into the probability P Pyy(), xx), which is optimized during model training.
2.2 EXPERIMENTAL SETUP
To validate the universality of source representation bottleneck in Seq2Seq models, we conducted experiments on three representative tasks, which vary from the distance between input and output domains and the scale of training data:
Machine translation takes sentence in one language input, and outputs a semantically-cquuveae sentence in another language. We conducted experiments on three benchmarking datasets: small- scale WM 6 Comanian-Engliin ERo-En; 0.6M instances), medium-scale WMT14 Enerish
2

---

Published as a conference paper at ICLR 2021
(En-De; 4.5M instances), and large-scale WMT14 Maglish--rencch (En-Fr; 36.0M instances). The tokenized BLEU score (Papineni et al. 2002) was used for all the translation tasks.
Text summarization takes long-text document input, and outputs short and adequate summary in the same language. We used the CNN/Daily Mail corpus (0.3M instances). We evaluated with the standard ROUGE metric (Lin 2004), i.e. Rouge-1, Rouge-2, and Rouge-L..
Grammatical error correction takes a sentence with grammatical errors as input, and outputs a corrected sentence. We used CONLL14 datasets the testbed (1.4M instances). The MaxMatch (M²) scores (Dahlmeier & Ng. 2012) were used for evaluation with precision, recall, and Fo.5 values.
The machine translation task has distant input/output domains (i.e. in different languages), while the other tasks have similar input/output domains (i.e. in the same language). We used Trans- former (Vaswani et al. 2017) the Seq2Seq model. Details of the datasets and model training are listed in Appendix A.I.
BEHAVIOR OF ENCODERFUSION
In this section, we first formulate our research hypothesis of source representation bottleneck ($ that EncoderFusion expects to solve. In the following subsections, we propose fine-grained layer attention model ($3.2) validate our hypothesis on well-designed experiments ($3.))
3.1 SOURCE REPRESENTATION BOTTLENECK
Seq2Seq models learn more abstract features with the increase of layer level (i.e. X X and Y Y Y (Belinkov et al. 2017 It has been extensively validated that reasonable use of both the abstract representations (at higher-level layers) and the surface representations (at lower-level layers) is beneficial for various NLP (Lu & Li 2013 Hu et al. 2014 Dou et al.) 2018 Peters et al.) 2018 and CV (Long et al. 2014 Pinheiro et al. 2016 Lin et al. 2017 Chen et al. (2118)) tasks.
However, the Seq2Seq decoder only takes the abstract representations at uppermost layer XN as input (Equation[). while ignores other usefully surface representations at other layers X" (n < N). Although X has encoded surface features from low-level representations through layer-by-layer abstraction and residual connections, we hypothesize that its limited representation capacity may not sufficiently model those surface features from lower encoder layers, especially the embedding layer. We call such an issue as source representation bottleneck.
3.2 FINE-GRAINED LAYER ATTENTION
For each decoder layer, layer attention (Bapna et al., 2018 Peters et al.) 2018) assigns normalized scalar weights to encoder layers, providing a direct way for evaluating the contributions made by each encoder layer. However, the capacity of a simple scalar weight is limited, leading to insufficient evaluation of the contributions.
Motivated by fine-grained attention (Choi al., 2018) that each element of context vector receives an individual attention weight, we propose a fine-grained layer attention model to combine the advantages of both techniques. This allows to more convincingly evaluate the contribution of individual encoder layer to the model performance. Besides, the nature of inn-grained attention enables to give in-depth analyses of the representation power in [3.3
Specifically, we replace the layer-agnostic source representation X with the layer-aware representa- tion Sm for each decoder layer " which is calculated as:
N
exp(wm....d)
S Pm
mm
X
W mLR
m.
exp soppummindd
n=U
where denotes an element-wise multiplication, and m.nd denotes an element in the learnable attention weight W (X++))) where D is the dimensionality of the source representation. When n = 0, we use the word embeddings X without position embeddings as X", which has been empirically proved effective. We applied regularization technique DropConnect (Wan et al.
3

---

Published as a conference paper at ICLR 2021
2013) to the attention weight W for stable training, which randomly drops each wm with probability p and divides W by 1 1- p. We set it to 0.3 for all the experiments.
Table lists the results. The proposed fine-grained layer attention model consistently outperforms the vanilla Transformer across Seq2Seq tasks, demonstrating the benefit of fusing surface features at lower-level layers.
We evaluated several EncoderFusion methods in
Table 1: Results of existing encoder layer fusion
Table I including layer aggregation (Dou et al.) 2018), layer-wise coordination (He et 20. 8).
methods on the WMT16 Ro-En translation task.
Model
BLEU
and coarse-grained layer attention (Bapna al.)
2018). Their results are respectively 34.05, 34.19, and 34.32, which are all lower than that of fine-grained layer attention (34.45). Based on these experimental results, we thus choose fine-grained layer attention representative of EncoderFusion following analyses.
Vanilla Transformer
33,80
Layer aggregation
34.05
Layer-wise coordination
34.19
Coarse-graiee -eeeee-rrnddd layer attention
34.32
Frine-graindd layer attention
34.45
3.3 BEHAVIOR CHANGES ACROSS ENCODER LAYERS
In this section, we investigate whether the surface features at lower encoder layers (especially the encoder embedding layer) contribute to the model performance via carefully designed experiments.
Decoder Layer
Decoder Layer 3
Decoder Layer
1
4
5
5
6
1
2
3
4
6
11
18 11 15 12 .11
14
.13 .12
1.14
21
19 .16 .14
5 5-16
18
.17
.16 .14 .13
5-12
.13
14
.14 .14
5-16
18
18
11
.12 .11
Encoder Layer
4.14.15 15
.1
14 .13
4.12
.13
.1
.15
.13 .11
4.14
15
.15
11 .10
.10
3.15
.4.
.13
11
.14
3.12
.13
14
11
.13
.11
3.11
14
113
11
.11
.12
2.14 .1 12
11
14
.15
2.14
.14
14
118
.13
.12
2.14
12
12
11
.15
.16
1.14 11 11
.13
14
.16
1.16
.15
.15
14
.13
.14
.16
.13
.11
14
.17
.18
Emb
.13 .11 .13 .15 17 .19
Emb
19
.18
.17
.11
21
.66
Emb
.13
.11
.10
.14
.18
.19
(a) Translation: Ro-En
(b) Summarization
(c) Correction
Figure I: Attention distribution that each decoder layer (z-axis) attending to encoder layers (y-axis).
Visualization of layer attention We first visualize the learned layer attention distribution in Fig- [ in which each weight is the averaged attention weights over all dimensions. Generally, higher weight denotes more contribution of an encoder layer to the corresponding decoder layer.
Clearly, in all tasks higher decoder layers especially the uppermost ones pay more attention the encoder embedding layer, which indicates that the surface representations potentially bring some additional useful features the model performance. Voita al. (2019); Wang & Tu (2020) reveal that the upper layers of decoder are responsible for the translation part while the lower layers for the language modeling part. Similarly, our results show that surface representations might play an important role in learning to translate source tokens.
Among the Seq2Seq models, there are still considerable differences in the attention heatmaps. In the summarization model, almost all decoder layers focus more on the encoder embedding layer, while the other two models the intermediate decoder layers pay more attention to the higher-level encoder layers. This is consistent with the findings of Rothe al. (2019), in which they reveal that the summarization task, as a typical extractive generation task, tends to use more surface features to generate extractive summaries. In contrast, both machine translation and error correction tasks require large amount of syntactic and semantic information, which are generally embedded in higher-level encoder layers (Peters et al., 2018).
4

---

Published as a conference paper at ICLR 2021
However, we still cannot conclude that source representation bottleneck does exist in Seq2Seq models, since the surface features might act as a noise regularizer to improve the robustness encoder output representations. To dispel the doubt, further design two experiments to directly evaluate the effectiveness of surface features at the encoder embedding layer.
Contribution of individual encoder layer In this experiment, we quantitatively analyze the be- haviors change of trained Seq2Seq model when masking specific encoder layer (i.e. turning its attention weight to zero and redistribute the other attention weights). Note that the masking operation does not affect the information flow of encoding calculation, i.c. keeping Equation [unchanged.
0%
6%
Relative Change
-4%
0%
Emb
3
-8%
-6%
Translation Summarization Correction
Translation Summarization
Correction
(a) Model performance
(b) Length of output
Figure 2: Relative changes of (a) model performance and (b) length of output when masking individual encoder layer in the trained Seq2Seq models. As seen, masking the embedding layer leads to significant drop of model performance and increase of output length.
Figure 2(a) shows the contribution of individual encoder layer to model performance. As seen, masking the encoder embedding layer seriously harms the model performance in tasks, which confirms our claim that the surface features in the embedding layer are essential to Seq2Seq models.
Figure 2(b) shows the results on the output length. Masking the encoder embedding layer consistently increases the length of generated output, which is especially significant for the summarization model. One possible reason is that the instances in translation and correction tasks have similar input/output lengths, while the summarization instances have distant input/output lengths.
By analyzing the model outputs, we found that the Seq2Seq models tend to generate some hallucina- tory (i.e. fluent but unfaithful to the source) predictions (Lee et al. 2019 Wang & Sennrich 2020) when masking the embedding layer. Taking the correction task for an example, a right prediction "anyone" replaced by the hallucinatory prediction "friends of anyone" in the masked model. in which the corresponding source contains no information related to "friends". This issue becomes worse in the summarization task, since the hallucinatory prediction more likely to be sentence.
The additional hallucinations will increase the output length and reduce the model performance. In addition, Lee al. (2019) point out that even if hallucinations occur only occasionally, the Seq2Seq model may evidently lose user trust than other prediction problems, indicating the importance to fuse surface features the embedding layer. More cases are studied in Appendix A.2.
Expressivity of attended dimensions in the encoder embedding layer As shown in Figure the uppermost decoder layer pays most attention to the encoder embedding layer (i.e. the lower right corner). If the embedding layer acts as noise regularizer, the layer dimensions would be randomly attended by the fine-grained model; otherwise, the dimensions of higher attention weights should be distinguished from the other dimensions.
Starting from this intuition, we reordered the dimensions of the encoder embedding layer according to the attention weights Dr, and split into two equal sub-embedding matrices, i.c. more attended dimensions and less attended dimensions. We compared the expressivity of the two sub-embedding matrices by the commonly-use
singular value decomposition (Gao et al. 2019 Wang et al. 2019a Shen et al.) 2020), in which higher normalized singular values denote that the embedding more uniformly distributed, thus are more expressive. The singular values are normalized by dividing them by the largest value and their log scale values reported for better clarity.
5

---

Published as a conference paper at ICLR 2021
More attended
More attended
More attended
Randomly selected
Randomly selected
Randomly selected
Log Elgenvalues
-1
Less attended
Less attended
Less attended
-2
-3
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
Index Singular Value
Index of Singular Value
Index of Singular Value
(a) Translation: Ro-En
(b) Summarization
(c) Correction
Figure 3: Log scale singular values of the three sub-embedding matrices in the fine-grained layer attention models. Higher log eigenvalues denote more expressivity the dimensions.
Figure deppicts depicts the singular value results. For comparison, we also report the values of the randomly selected dimensions. Clearly, the more attended dimensions are most expressive, while the less attended dimensions are least expressive. These results demonstrate that the fine- une-grained attention model indeed extracts useful surface information from the encoder embedding layer. which does not play the role of noise regularizer.
From the above experiments, we prove that the encoder embedding layer indeed provides useful surface information, which is not fully exploited by the standard Seq2Seq models.
OUR METHOD
In Section show that the uppermost decoder layer requires more surface features for better representation learning. One possible reason is that the uppermost decoder layer is used for predicting individual target token, which naturally benefits from more token-level surface features than sequence- level abstract features. To validate this assumption, we simplify fine-grained layer attention that only the uppermost decoder layer can attend the embedding layer and output layer of the encoder. Empirical results show that the simplified variant works on par with the original one, revealing that the surface features embed at the source embedding layer is expressive.
Although layer attention model partially alleviates source representation bottleneck, it potentially introduces unnecessary intermediate encoder representations. To address this gap, we propose to directly connect the decoder softmax layer and the encoder embedding layer with simple Surface Fusion method.
4.1 SURFACEFUSION
Seq2Seq learning aims to maximize the log-likelihood of target sequence y given source sequence x. practice, it factorizes the likelihood of the target sequence into individually token likelihoods:
y arg max II log P(y;) = arg max II log ((vll=s;.))
(3)
We rewrite P(y;) as a fused probability with the second condition term x:
log P(y,) + (P(,)lu<s,x)) P(ys(x))
(4)
where (b()) fusion method that will be deseribed later, and P(y /x) is probability conditioned on the source surface features. Specifically, we employ multi-head dot-product attention net- work (Vaswani al. 2017) with decoder output representation y as query, encoder output representations X as keys and encoder surface representations Xamb as values, to calculate a surface representation x)y,.
Then we use the pre-softmax weight E Rd of the vanilla model to transform the surface representation r(yı, x) E IRd into pre-softmax logit F(yj, x) E R Rol, The final surface constraint

---

Published as a conference paper at ICLR 2021
probability is calculated as:
exp(11, ((U)) p(y)(x)= Ewev, exp(1 x)/T)
(5)
where I.(( denotes index function take the logit of target token y, and denotes softmax temperature parameter to control the smoothness of the probability distribution P(yj [x). As approaches O, the distribution tends to be an one-hot distribution representing the token of the maximum probability. The distribution becomes uniform at higher T.
There are many variants of fusion methods (Gulcehre al., 2015 Sriram et al. 2017 Stahlberg et al. 2018). The aim of this paper is not to explore this whole space but simply to show that two fairly straightforward implementations works well and that SurfaceFusion helps for sequence to- sequence models:
Choices of fusion function +
Hard fusion:
$ hard = À log P(ujlye) x) (1 X) (()))x)
(6)
Soft fusion:
soft log(softmax (1; Øマ x) + log P(u,
where pre-defined interpolation weight, and Byy,lye, x) is the pre-softmax logit of the probability P(ujlwej, x). Compared to hard fusion, soft fusion removes the need for manually setting the hyperparameter A.
The proposed SurfaceFusion method is easy to use. There are only two additional hyperparameters. ie. (Equation 3 and т (Equation 5). We find that is sensitive to the corpus scale but insensitive to the relationship of input/output domain, which was set to 0.9 for the En-De, En-Fr and correction tasks, and 0.8 for the Ro-En and summarization tasks. For T. was set to 5 for soft fusion and 1 for hard fusion across different benchmarks. We kept other settings all the same with the vanilla models. In practice, we observed an additional 10% inference latency with the introduction of SurfaceFusion.
4.2 EXPERIMENTAL. RESULTS
Table 2: Results of the proposed SurfaceFusion methods on the Seq2Seq tasks. "FGLA" denotes fine-grained layer attention. The existing results are (Ghavvinieejad et al. (2019) for Ro- En, Ott et al. (2018) for En- De and En-Fr, Ott et al. (2019) for summarization, and Chollampat & Ng (0188) for correction. All reported scores are the higher the better.
Translation
Summarization
Correction
Ro-En
En-De
En-Fr
RG-1
RG-2
RG-L
Prec.
Recall
Fo.5
Existing
34.0
29.3
43.2
40.1
17.6
36.8
65.5
33.1
54.8
Vanilla
33.8
28.9
43.4
40.4
17.7
37.2
64.7
33.2
54.3
FGLA
34.5
29.1
43.5
40.8
18.0
37.5
67.7
31.9
55.3
Hard fusion
35.1
29.5
43.9
40.9
18.2
37.7
67.0
34.4
56.3
Soft fusion
34.0
29.0
43.6
41.0
18.3
37.9
66.8
35.0
56.6
Model performance Table 2 lists the results of the proposed approach on different tasks. In addition the vanilla Seq2Seq model ("Vanilla")) we also report the results of existing studies on the same datasets ("Existing") for better comparison. Our re- implementation of the vanilla models matches the results reported previous works, which we believe make the evaluation convincing.
Clearly, the proposed fusion approaches outperform the basclines (i.c. "Vanilla" and "FGLA") in all cases, while there are still considerable differences among model variations. Hard fusion performs better on the translation tasks, while soft fusion is superior on the summarization and correction tasks. Unlike hard fusion that performs at the probability level, soft fusion performs at the logit level to provide earlier and direct way for fusing surface features, which might be better solution for the tasks with a similar input/output domain.
7

---

Published as a conference paper at ICLR 2021
Closeness of word embeddings SurfaceFu- sion shortens the path distance between source and target embeddings, which can help to learn better bilingual embeddings with direct inter- actions. Table 3 shows the cosine similarities between the tied source and target embeddings on the Ro-En translation task.
Cosine similarities between aligned source and target word embeddings. "All" and "Non-Shared" denotes keeping or removing the aligned pair when the source and target words are the same, which are easier to be aligned.
Table 3:
All
Non-Shared
Vanilla
0.602
0.338
In the experiment, we first train an additional aligner (i.e. fast-align (Dyer al. 2013)) on the training corpus and use the alignment links to construct a word dictionary. The results calculated over
SurfaceFusion
0.650
0.417
the dictionary show that the relationship between the source and target embedding becomes much closer (i.e. high cosine similarities). This can help each other to learn better representations, and has
been validated to be beneficial for Seq2Seq models (Press & Wolf, 2017
Liu al., 2019).
Expressivity of word embeddings In this experiment, we quantitatively evaluate the expressivity of the word embeddings learned by different models using the singular value decomposition. The related experimental details and executions similar that of Figure
Figures T shows the results of the tied source and target embeddings the Ro-En translation task. The word em- beddings of the vanilla model have fast decaying singular values, which limits the representational power of embed- dings to a small sub-space. The SurfaceFusion model slows down the decaying and the singular values become more uniformly distributed, which demonstrates that the fused surface features remarkably enhance the representation learning of embeddings. This provides better starting point for the model to effectively extract surface and ab- stract features, which leads to an improvement of model performance.
SurtaceFusion
Log Elgenvalues
Vanilla
-2
100 200
300
400
500
Index Singular Value
Figure 4: Log scale singular values of the embeddings.
RELATED WORK
EncoderFusion Seq2Seq Lower encoder layers that embed useful surface features are far away from the training signals, which poses difficulty for deep Seq2Seq models to exploit such useful features. Although residual connections (He al.] 2016) have been incorporated to combine layers, these connections have been "shallow" themselves, and only fuse by simple, one-step operations (Yu cct al., 2018). In response to this problem, several approaches have been proposed to fuse the encoder layers with advanced methods, such layer attention (Bapna al.) 2018 Shen et al.) [018 Wang
et al. 2019c), layer aggregation (Dou et al. 2020p. and layer-wise coordination (He et al. promising results on different NLP tasks, not much is known about how the EncoderFusion works. In
2018
Wang et al. 20) Dou et al.) 2019] Li et al. 2018 Liu et al. (020)). Although these methods show
addition, some other studies show that exploiting low-layer encoder representations fail improve model performance (Domhan 2018).
In this paper, we consolidate the conflicting conclusions of existing studies by pointing out that the encoder embedding layer is the key, which can help Seq2Seq models to precisely predict target words. Based on this finding, we propose novel Surface Fusion to directly connecting the encoder embedding layer and the softmax layer, which consistently outperform current EncoderFusion approaches across different NLP tasks.
Variants of Feature Fusion Feature fusion aims to merge two sets of features into one, which is
frequently employed in CV tasks, such as semantic segmentation (Long et al.] 2014 Chen et al.
2018al Zhang al. 2018) and object detection (Pinheiro et al. 2016 Lin al. (017). Zhang et al.) (2018) shows that simply fusing surface and abstract features tends to be less effective due to the gap semantie levels.
For NLP tasks, researchers investigated fusion models for language understanding (Lu & Li 2013
Hu et al. 2014 Peters et al. 2018) and language generation (Gulcehre et al. 2015 Sriram et al. 2017
8

---

Published as a conference paper at ICLR 2021
Stahlberg al. 2018). Nguyen & Chiang (2019) propose to fuse features at representation-levll but we empirically find this kind of fusion method is not orthogonal to multi-layer models due to the large semantic gap. (uulcehre et al. 2013) combine the predictions produced by the Seq2Seq model and external LM predictions in a later fusion manner, which pose little impact to the original information flow. Stahhbeerg et al. ( improve upon it by removing the dependence hyper-parameter. In this work, we demonstrate the effectiveness of the two typical probability-lveel fusion methods on sequence-to-sequenee learning tasks. Unlike them that rely on external model, our approach only requires surface attention module that can be jointly trained with the vanilla Seq2Seq model.
the manually defined
CONCLUSION AND FUTURE WORK
In this paper, we investigate how encoder layer fusion works on solving the source representation bottleneck. Based series of experiments on different Seq2Seq tasks, we find that the encoder embedding layer is important to the success of EncoderFusion by exploiting the useful surface information. Based on this observation, we propose novel SurfaceFusion to directly connect the encoder embedding layer and softmax layer. Experiments show that SurfaceFusion consistently outperforms the conventional EncoderFusion in several datasets. Extensive analyses reveal that SurfaceFusion enhances the learning of expressive bilingual word embeddings for Seq2Seq models, which confirm our claim.
Future directions include validating our findings on more Seq2Seq tasks (e.g. dialogue and speech recognition) and model architectures (RNMT+ (Chen et al. (018b) and DynamieConv (Wu et al. 2019)). It is also worthwhile to explore more alternatives to EncoderFusion from the perspective of exploiting the embedding layer.
ACKNOWLEDGMENTS
This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. 0101/2019/A2), and the Multi-year Research Grant from the University of Macau (Grant No. MYRG20020-00054-FST), We thank the anonymous reviewers for their insightful comments.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In arXiv, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In EMNLP, 2018.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad. and James Glass. What do neural machine translation models learn about morphology? In ACL, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 201 8a.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances neural machine translation. In ACL, 2018b.
Hecyoul Choi, Kyunghyun Cho, and Yoshua Bengio. Fine-grained attention mechanism for neural machine translation. Neurocomputing, 2018.
Shamil Chollampatt and Hwee Tou Ng. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In AAAI, 2018.
Daniel Dahlmeier and Hwee Tou Ng. Better evaluation for grammatical error correction. In NAACL, 2012.

---

Published as a conference paper at ICLR 2021
Tobias Domhan. How much attention do you need? granular analysis of neural machine translation architectures. In ACL, 2018
Zi-Yi Dou, Zhaopeng Tu, Xing Wang. Shuming Shi, and Tong Zhang. Exploiting deep representations for neural machine translation. EMNLP, 2018.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. Dynamic layer aggregation for neural machine translation with routing-by-agreement. In AAAI, 2019.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of IBM model 2. In NAACL, 2013.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In JCML, 2017.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP, 2019.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. In arXiv, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie- Yan Liu. Layer-wise coordination between encoder and decoder for neural machine translation. In NIPS, 2018.
Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. Convolutional neural network architectures for matching natural language sentences. In NIPS, 2014.
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction, In ACL, 2020.
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. An empirical study of incorporating pseudo data into grammatical error correction. In EMNLP, 2019.
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. Hallucinations in neural machine translation. In OpenReview, 2019.
Jian Li, Xing Wang, Baosong Yang, Shuming Shi, Michael Lyu, and Zhaopeng Tu. Neuron interaction based representation composition for neural machine translation. In AAAI, 2020.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization
Branches Out, 2004.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Fenglin Liu, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. Layer-wise cross-view decoding for sequence-to-ssqueee learning. arXiv, 2020.
Xuebo Liu, Derek F. Wong, Yang Liu, Lidia Chao, Tong Xiao, and Jingbo Zhu. Shared-privane bilingual word embeddings for neural machine translation. ACL, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2014.
Zhengdong Lu and Hang Li. deep architecture for matching short texts. In NIPS, 2013.
10

---

Published as a conference paper at ICLR 2021
Toan Nguyen and David Chiang. Improving lexical choice in neural machine translation. In NAACL, 2019.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In ACL, 2002.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In ICLR, 2018.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Pedro H. O. Pinheiro, Tsung- Yi Lin, Ronan Collobert, and Piotr Dollár. Learning to refine object segments. In ECCV, 2016.
Ofir Press and Lior Wolf. Using the Output Embedding to Improve Language Models. In EACL, 2017.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. In arXiv, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Rethinking batch normalization in transformers. In ICML, 2020.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Yan Liu. Dense information flow for neural machine translation. NAACL, 2018.
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training seq2seq models together with language models. In arXiv, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014.
Felix Stahlberg, James Cross, and Veselin Stoyanov. Simple fusion: Return of the language model. In WMT, 2018
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL, 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML, 2013.
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. In arXiv, 2020.
Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adversarial training. In JCML, 2019a.
Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, and Jingming Liu. Denoising based sequence-to- sequence pre-training for text generation. In EMNLP, 2019b.
11

---

Published as a conference paper at ICLR 2021
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yingiao Li, and Jingbo Zhu. Multi-layer representa- tion fusion for neural machine translation. In COLING, 2018a.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In ACL, 2019c.
Wenxuan Wang and Zhaopeng Tu. Rethinking the value of transformer components. In COLING, 2020.
Xinyi Wang, Hicu Pham, Zihang Dai, and Graham Neubig. Switchout: An efficient data augmentation algorithm for neural machine translation. In EMNLP, 2018b.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. In arXiv, 2016.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. CVPR, 2018.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In ICML, 2020.
Zhenli Zhang, Xiangyu Zhang, Chao Peng, Dazhi Cheng, and Jian Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In ECCV, 2018
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 EXPERIMENTAL SETUP
Table 4: Statisties of the datasets and hyperparameters for the experiments. All the data have been tokenized and split into joint sub-word units (Sennrich et al. 2016). "Batch" denotes the number of source tokens and target tokens used in each training step. "DP" denotes the dropout value (Srivastaval et al. 2014). "LP" denotes the length penalty (Wu et al. 2016). "Base" and "Big" denote the two kinds model variants Transformer. We chose the checkpoint with best validation ppl for testing.
Vocab
#Sents
Training
Testing
Sre/Tgt
Train
Dev
Test
Model
Batch
Step
DP
Beam
LP
Ro-En 34,976
0.6M
2K
2K
Base
16K
60K
0.3
1.0
En-De
32,768
4.5M
3K
3K
Big
460K
30K
0.3
0.6
En-Fr 36,736
35.5M
6K
3K
Big
460K
80K
0.1
0.9
CNN/DM
50,264
0.3M
13K
11K
Base
64K
70K
0.1
2.0
CONLL
33,352
1.3M
5K
1K
Base
64K
80K
0.2
0.6
Machine translation For WMT16 Romanian-Fnglish, used the prepossessed date and existing result from Ghazvininejad et al. (2019). The validation set is newsdev2016 and the test set is newtest2016. For WMT English- German, the prepossessed data and existing result are derived from Out et al. (2018). The validation set is newstest2013 and the test set is newstest2014, For WMT14 English- French, we reported the existing result from Ott et al. (2018) and followed them to preprocess the data sets. The validation set is 2risstest201++0011 and the test set is newstest2014.
Text summarization For CNN/Daily Mail dataset, we used the existing result and preprocessing method Oft et al. (2019). During testing, the minimum length was set to 55 and the maximum length was set to 140, which were tuned on the development data. We also followed Paulus al. (2018) to disallow repeating the same trigram.
Grammatical error correction For CONLL14 benchmark, the preprocessing scrip and existing re- sult are given by Cholllampatt & Ng 2018). We applied the regularization technique SwitchOut (Wang) jet al., 2018b) in this task to prevent overfitting, which was set 0.8 for the source and 0.9 for the target.
more details of the benchmarks. is noted that other unmentioned hyperparameters keep the same with the original paper of Transformer (Vaswani et al. 2017). All the models are implemented by the open-source toolkit fairseq (Ott et al. 2019
A.2 CASE STUDY
Tables 6 and give the cases from the three tasks. We can see that the hallucination issue related to surface features consistently appear over the different Seq2Seq tasks. The most representative cases are those from the correction task, in which very similar input/output sequences still make such mistakes.
Another observation the prediction omission problem when masking the encoder output layer. The lack of abstract features leads to incomplete semantics of source representations, thus making Seq2Seq models omit generating a part of source, hurting the model performance. By looking the cases over the three tasks, find that the prediction omission is widespread in the prediction of modifiers, e.g. adjectives and adverbs.
sh
https://ftthub.cco///tttt//////
13

---

Published as a conference paper at ICLR 2021
Table 5: Examples from the Ro-En translation task. Red words good predictions, while blue words are bad predictions. Masking the embedding layer ("Mask Emb") of the fine- -raeggr layer attention model leads to hallucinatory predictions, prolonging the prediction length. While masking the output layer ("Mask Out") leads to prediction omissions, shortening the length.
Source
diseara voi merge acasa voi dormi linistit.
Reference
i will go home tonight and sleep well.
Hallucination
Vanilla
will go home and sleep quietly
Mask Emb
the device will go home and will sleep peacefully. i will go home and sleep quietly.
Mask Out
Source
radem adesea mult atunci cand vorbim.
Reference
we often laugh lot when we talk
Omission
Vanilla
we often laugh lot when we talk
Mask Emb
we often laugh lot when we talk
Mask Out
often laugh when talk.
Table 6: Examples from the CNN/DM summarization task.
Source
But able to carry just much power 400,000 volts It is designed
to be less obtrusive and will be used for clean energy purposes
Reference
But it able to carry just as much power - 400,000 volts It is designed to be less obtrusive and will be used for clean energy
Hallucination
Vanilla
But it able to carry just as much power 400,000 volts It is designed to be less obtrusive and will be used for clean energy.
Mask Emb
... is able carry just much power 400,000 volts. The design is
T-shape, with two hanging baskets either side
Mask Out
But it able to carry just as much power - 400,000 volts. It designed to be less obtrusive and will be used for clean energy.
Source
Opening statements in his trial are scheduled to begin Monday...
Reference
Opening statements are scheduled Monday in the trial of James
Holmes...
Vanilla
Prosecutors are not swayed, will seek the death penalty Opening statements in his trial scheduled to begin Monday Holmes says was suffering psychotic episode at the time
Omission
Mask Emb
Prosecutors are not swayed, will seek the death penalty Opening statements in his trial are scheduled to begin Monday Holmes says he was suffering psychotic episode at the time
Mask Out
Prosecutors not swayed and will seek the death penalty. Holmes says was suffering psychotic episode" at time
Table 7: Examples from the CONLL correction task.
Source
They can become anyone.
Reference
They can become anyone.
Hallucination
Vanilla
They can become anyone.
Mask Emb
They can become friends with anyone.
Mask Out
They can become anyone.
Source
In conclude. people should think carefully of what is the consequences of telling the relatives his her generic disorder issue conclusion. people should think carefully about what is consequences of telling the relatives his or her generic disorder issue.
Reference
Omission
Vanilla
conclusion, people should think carefully about what is the consequences of telling the relatives his or her generic disorder issue.
Mask Emb
In conclusion, people should think carefully about what is the consequences of telling the relatives his or her generic disorder issue..
Mask Out
In conclusion. people should think carefully about what is the consequences
of telling relatives his or generic issue.
14


Reviews:
[{"review_id": "tHgAVHwJypr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####################   Summary   ####################\n\nThis paper introduces the fine-grained layer attention to evaluate the contribution of individual encoder layers and investigate how encoder layer fusion works, where the decoder layers have access to information for various encoder layers as opposed to only the final encoder layer as in standard Transformer.\n\nBased on the observations that the encoder embedding layer is important to the success of encoder layer fusion and the uppermost decoder layer pays more attention to the encoder embedding layer, this paper proposes SurfaceFusion, which only connects the encoder embedding layer to the softmax layer of decoders, leading to quite substantial gains in metrics such as BLEU.\n\n#####################    Strengths   ####################\n1. This paper is really clearly written. The paper is easy to follow and understand.\n2. The proposed SurfaceFusion is well-motivated by a series of experiments on different Seq2Seq tasks.\n3. The experimental results are very solid.\n\n#####################   Concerns   ####################\n\nI hope the author can select several following concerns as feedback.\n\n1. In this work, the authors find that in the summarization model, almost all decoder layers focus more on the encoder embedding layer. In other words, compared with the translation model and correction model, the summarization model tends to use more surface features from the encoder embedding layer. Therefore, it is reasonable to speculate that the summarization model will receive more improvements from the proposed SurfaceFusion. However, the experimental results in Table 1 show that the translation and correction models achieve higher improvements than the summarization model. \n\n2. Figure 1 shows that the uppermost decoder layer also pays attention to other encoder layers, why SurfaceFusion only uses the encoder embedding layer? The authors claim that “although the layer attention model partially alleviates this problem, it potentially introduces unnecessary intermediate encoder representations.” Can you show some visualizations/experimental results to support this hypothesis that the intermediate encoder representations are unnecessary for the uppermost decoder layer?\n\n3. The experiments show that the encoder embedding layer is beneficial for all decoder layers, why the proposed SurfaceFusion does not consider connecting the encoder embedding layer to all decoder layers. A much wider empirical investigation would be appreciated.\n\n4. Along the same line, some relevant works are omitted [1][2][3][4][5][6], especially for the work from [1], which also considers the connections between the lower encoder layers and uppermost decoder layer, and boosts similar or even higher results on the same benchmarks (EN-DE and EN-FR) as this work. I would suggest the authors take a discussion and a comparison to that work.\n\n5. The margin of some gains in Table 1 is small. Statistical significant test and error range are highly appreciated.\n\nOverall, this paper is really clearly written, the proposed SurfaceFusion is novel and well-motivated, and experimental results are very solid. I recommend the authors to enrich this work by deepening their analysis. A much wider empirical investigation would be appreciated.\n\n[1] Layer-Wise Cross-View Decoding for Sequence-to-Sequence Learning.\n\n[2] Neuron interaction based representation composition for neural machine translation. In AAAI 2020.\n\n[3] Dynamic layer aggregation for neural machine translation with routing-by-agreement. In AAAI 2019.\n\n[4] Learning deep transformer models for machine translation. In ACL 2019.\n\n[5] Dense information flow for neural machine translation. In NAACL-HLT 2018.\n\n[6] Multi-layer representation fusion for neural machine translation. In COLING 2018.\n\n#####################    After Rebuttal   ####################\n\nI thank the authors for responding to the comments and have read them carefully.\nThe authors have addressed all my concerns in the rebuttal and I vote for acceptance of this submission.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aGqrDNXRRGf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a *fine-grained layer attention* (FGLA) to analyze Encoder Fusion method. \n\nWhile the introduction of FGLA allows for the analysis of the contribution of individual encoder layers, it changed the model’s architecture. Thus, when a model with FGLA is trained, it’s optimized differently such that the use of each separated layer in the encoder is modeled explicitly through attention weight. As a result, the contribution of each layer found in Seq2seq with FGLA might not be the same as in the standard Seq2seq. Therefore, I think the conclusion might not be appropriate for the standard Seq2Seq model. In other words, the source representation bottleneck hypothesis might not hold true for standard Seq2seq.\n\nCompared to the previous fusion method that uses a scalar per layer as attention weight (i.e., *Transparent Attention*) in the work of Bapna et. al., 2018, FGLA generalizes it by making a vector per layer. This generalization is straightforward and simple. The later analysis in the paper is carried out on FGLA. However, I find that the same analysis can be performed with Transparent Attention. The introduction of FGLA seems not well-motivated in this regard. In terms of interpretability, Transparent Attention is more interpretable since there is one scalar per layer, whereas FGLA has to report average weight per layer (Figure 1).\n\nFor experimental results, FGLA is not compared with other fusion methods. Thus it’s unclear how the proposed method positions itself in the previous work on Encoder Fusion.\n\nA previous work of ([Nguyen and Chiang, 2018](https://www.aclweb.org/anthology/N18-1031/)) proposed a similar way of fusing source embeddings directly to the output of the decoder. Their work can be seen as Surface fusion proposed in this paper.\n\nWhile the motivation for this kind of fusion might be obvious for machine translation, it’s less so for summarization and grammatical error correction. I think it’s worth spending some text/examples on why this is needed for those tasks.\n\n\n**References**\nImproving Lexical Choice in Neural Machine Translation. Nguyen and Chiang, NAACL 2018\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qz0OMIa4aj7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors perform a thorough analysis of encoder fusion for Transformers: which encoder layer should the N-th decoder layer attend to? It turns out that the final decoder layers often attend to the encoder embeddings, leading the authors to provide them to the last decoder layer which leads to small improvements of performance on machine translation, summarization and grammar correction tasks. These are nice results, but the gains are small and the models are tested in the very basic configurations. These tasks and techniques, as well as some numbers used to claim state-of-the-art, are from a few years ago (e.g., SOTA on en-de translation is higher currently than the authors claim and higher than their number). It would be interesting to see if the presented conclusions hold for larger models - esp. for a T5 Transformer on masked language modeling, as this would be a more commonly used model in 2020. Unluckily, it is quite possible that increased activation size may negate the benefits of the authors' technique. It may well though make it even more important -- it would be really good to know! Lacking these experiments, we cannot recommend acceptance at this point.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AwW_BLVg59", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting idea where the authors propose \"SurfaceFusion\", where they use the source embeddings learned by the encoder to modulate the output of the decoder at the final layer. The authors claim this is because the embeddings contain valuable information that is lost during encoder processing because the encoder lacks the capacity to represent both semantic and surface features. The authors then show through a series of experiments that attending over the encoder embeddings is useful, and propose a way to integrate the information from the embeddings directly into the last layer of the decoder, showing that this improves experimental results.\n\nFew comments:\n\n1) there are some grammatical and spelling errors, e.g.: \"analyses\", \"which is the EncoderFusion expected to solve\"\n2) while the premise and analysis are interesting, i am curious about the reasons behind the design choice of \"SurfaceFusion\". If the encoder source embeddings are very important, and layerwise/finegrained attention perform similarly well, why not simplify the approach and simply add/concat them to the output of the encoder (or somewhere else where it makes sense)? have you tried this instead of introducing additional hyperparams?\n3) what are the costs in terms of wall-clock time when introducing an additional softmax operation for every token in the decoder?\n\n---\nupdate: \n\nThanks for the clarifications. I've read the response and other reviews and have updated my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes fine-grained layer attention to evaluate the contribution of individual encoder layers. This departs from the standard transformer architecture where the decoder uses only the final encoder layer. This paper investigates how encoder layer fusion works, where the decoder layers have access to information for various encoder layers. The main finding of the paper is that the encoder embedding layer is particularly important. They propose SurfaceFusion, which only connects the encoder embedding layer to the softmax layer of decoders, leading to accuracy gains.

There was some disagreement among reviewers about this paper. Overall, I found this a simple but effective contribution with interesting findings that can help future research in seq2seq models. Some of the weaknesses (discussing other relevant works, discussing other variants of FGLA, adding new experimental results) have been addressed in the updated version of the paper. One of the reviewers suggested running additional experiments on GLUE-style tasks (with a masked language model) to be really sure if the technique is convincing, and particularly try it with larger models (T5 was suggested). While adding those experiments would be a plus, I disagree that this is crucial - this paper is focusing on seq2seq tasks and is already considering several tasks: summarization, MT, and grammar correction. The results found by this paper are interesting and can foster future research extending this beyond these 3 tasks. Even if larger models can make the improvements smaller, there are many inconveniences in just increasing scale (memory consumption, energy consumption, etc.) It is my opinion that the community should value research that tries to understand the weaknesses of smaller models, rather than relying on large scale models to solve all problems.
Decision: Accept (Poster)