Stacking Diverse Architectures to Improve Machine Translation
Authors: Andrea Schioppa
Nal Kalchbrenner
Venue: Accepted by TMLR ()

Abstract: Repeated applications of the same neural block primarily based on self-attention characterize the current state-of-the-art in neural architectures for machine translation. In such architectures the decoder adopts a masked version of the same encoding block. Although simple this strategy doesn't encode the various inductive biases such as locality that arise from alternative architectures and that are central to the modelling of translation. We propose Lasagna, an encoder-decoder model that aims to combine the inductive benefits of different architectures by layering multiple instances of different blocks.  Lasagna’s encoder first grows the representation from local to mid-sized using convolutional blocks and only then applies a pair of final self-attention blocks. Lasagna’s decoder uses only convolutional blocks that attend to the encoder representation. On a large suit of machine translation tasks, we find that Lasagna not only matches or outperforms the Transformer baseline, but it does so more efficiently thanks to widespread use of the efficient convolutional blocks. These findings suggest that the widespread use of uniform architectures may be suboptimal in certain scenarios and exploiting the diversity of inductive architectural biases can  lead to substantial gains.


Published in Transactions on Machine Learning Research (02/2023)
Stacking Diverse Architectures to Improve Machine Trans- lation
Andrea Schioppa Google Research
rrischooppe@google.cco
Nal Kalchbrenner
nalk&kgoogle.com com
Google Research
Reviewed on OpenReview: htips: // openreview. net/ forum? id- XXXX
Abstract
Repeated applications of the same neural block primarily based on self-attention characterize the current saate-offthheart in neural architect for machine translation. In such architec- tures the decoder adopts masked version of the same encoding block. Although simple this strategy doesn's 't encode the various inductive biases such as locality that arise from alternative architectures and that are central to the modelling of translation. We propose Lasagna, an encoder-decodeer model that aims to combine the inductive benefits of different architectures by layering multiple instances different blocks. Lasagna's encoder first grows the representation from local mid-sized using convolutional blocks and only then applics pair of final self-attention blocks. Lasagna's decoder uses only convolutional blocks that attend to the encoder representation. On large suite of machine translation tasks, find that Lasagna not only matches outperforms the Transformer baseline, but it does so more efficienly thanks to widespread use of the efficient convolutional blocks. These findings suggest that the widespread use of uniform architect.res may be suboptimal in certain scenarios and exploiting the diversity of inductive architectural biases can lead to substantial gains.
1 Introduction
Transformers (Vaswani et al. 2017) have become the standard architecture for most natural language tasks and improvements attributed architectural modifications to the vanilla Transformer do not often generalize across tasks and implementations (Narang et al. 22021). The key component of Transformers is self-attention which is regarded as useful inductive bias that aggregates global context through pair-wisc interactions between the elements in the sequence. While recent work architectures (Tosstikhin et al. 2021 Wu
et al. 2019 Liu et al. 2021 Tay et al.] 2021) is questioning the necessity of self-attention, in general such architectures need to be combined in some way with self-attention to close the emsuing quality gap with the Transformer [Luu et al.] 2021 Tay et al.) 2021). Both the vanilla Transformer and these alternatives are made of repeated applications of the same layer block, such that all layers have the same structure and self-attention appcars as a sub-component each layer.
From computational standpoint, operations simpler than self-attention lead to better computational efficiency while from theoretical point of view wishes to better understand why self-attention is often necessary to achieve SOTA results in these language tasks. We propose however an additional motivation to study the effects of simpler operations on language lasks. Different layers can offer different, but complementary inductive biases that can prove useful. For example, convolutions provide strong bias for local contexts around each clement the sequence, which improves performance tasks such as speech recognition (Gulati) et al. 2020), text summarization (Aksenov al. (2200) and character-level machine translation (Kalchbrenner et al. 2016 From historical perspective, locality is one of the main motivations for n-gram language

---

Published in Transactions on Machine Learning Research (02/202))
Ericidur
Deccoder
Masived Gated Light Convolution
Pointwise Mull pizsttion
Sulf Attention
Light Con w7w
Eneooder
Light Convelution
QELU
GFLU
Light Convolution
1K1 Conv
(a)
(b)
Figure 1: (a) Lasagna architecture that stacks diverse architectures to capture multiple inductive biases. (b) Diagram of the Gated Light Convolution layer used in Lasagna.
models and of early neural probabilistic language models (Bengio et al. 2003). Additional evidence comes from modern language models that have access to global context but often default to primarily using the local one (Khandelwal et al. 2018).
We propose to combine complementary inductive biases into heterogeneous architecture, called Lasagna, to efficiently improve the performance of machine translation models. The architecture is heterogenous in that it does not repeat the same layer block multiple times like prior approaches, but stacks different layers onto each other capturing the respective inductive biases. We analyse the inductive biases of various layers basod on two dimensions: the degree of interaction between the elements in the sequence, on the one hand, and the size of the context, the other. Based this categorization Light Convolution. Light Convolutions (LConv), Gated Light Convolutions (GLConv) and Self-Attenting (SA) are the three main building blocks of Lasagna and they are stacked on top of each other in sequence. We conjecture that early layers might prefer to focus on a local context whereas later layers might prefer a global onc. This suggests stacking layers according to a local-to-global pattern starting with convolutions and ending with self-attention. The sequential stacking of the different. blocks Lasagna allows the model achieve greater efficiency than homogeneous approaches that channel different operations in parallel within the same block and repeat the block multiple times (Liu et al.) 2021 Tay al.) 2021 So al., 2019 The resulting Lasagna architecture is carefully designed combination of convolutional and self-attention layers. Despite prior work aiming to merge convolutions and self-attention either in domains other than machine translation (Cordonnier et al.) 2020 Yu et al.) 2018)) or with a different overall purpose (e.g. the local attention of (Yang et al., (2119) or the architecture search in [aan et al.) 2020 showing that this architectural combination yields marked improvement the performance--ficcinncy trade-off of NMT models is the open problem that Lasagna addresscs.
propose new layer block, the Gated
We design and apply Lasagna to machine translation and that it outperforms the Transformer both translation quality and inference speed. We achieve improved results in three different task settings that involve, respectively, autoregressive left-to-right generation, non-autoregressive parallel generation, and the intermediate setting that uses a parallel deep encoder and a left-to-right, but shallow decoder (Kasai et al., 2021). Our results show that Lasagna improves translation quality by up to +0.9 BLEU points depending on language pair and sotting, while maintaining an inference speed that is, respectively, up +30%, up +40% and up +60% higher than the baseline Transformer. Ablation studies give additional insight on the contributions of each part of Lasagna.
2

---

Published in Transactions on Machine Learning Research (02/2023)
21.7
Nr. Gated Light Conv. layers
27.8
layers
Gated Light Conv. layers
.28.9
.9.1
28.5
.2.77
lasagna
.2.99
.8.66
29.2
28.8
22.88
,2899
Lasagna
29.0
29.0
25.3 29.0
.29.2
.9.2
29.2
29.2
29.1
29.1 28.8
28.8
.29.0
28.8
29.1
29.0
29.2
29.0 22.8 28.9
28.4
.2.44
.29.1
.29.2
22.83
28.9
.2.0
29.0
29.0
Nr. of Light Conv. layers
2
5
Nr. of Light Conv. layers
(b)
(a)
Figure 2: BLEU scores on (EnDe) at model size Big as function of the number of Light Convolution and Gate Light Convolution layers. Encoder on the left and Decoder on the right. For the Encoder the complement to 7 gives the number of Self-Attention Layers, e.g. 2 for the dashed green line. The variance of scores is 0.1.
2 Related work
We summarize related work in different fields. In computer vision, the Mixer architecture (Tolstikhin al. 2021 proposes alternative simpler architecture to the SA used in Vision Transformers (Dosovitskiy et a.. 2021p. Similarly, the more recent gMLP architecture (Liu et al. 2021 outperforms the Mixer image classification and uses fewer paramcters. (Xiao et al.) 2021 demonstrates that replacing the standard patchily-stem of Vision Transformers with a standard stride of stacked convolutions improves training stability and can yield gains on classification accuracy. This idea of using convolutions to preprocess the inputs of the Transformer was also considered in Automated Speech Recognition (Mohamed et al. 2019), showing that the positional embeddings are no longer necessary. machine translation, [Kalchbrenner and (Gehring et al.) 2017) introduce fully convolutional architecture slightly prior the Transformer; discussed in Vaswani et al. (2017) however using standard convolutions makes the FLOP count of such models unfavorable compared to that of the Transformer. There have been various altempts at reducing or removing SA starting with Average Attention Networks (AANs) (Zhang et al. 2018), which increase inference speed, while incurring a small drop in translation performance, by replacing the SA the docoder with an averaging mechanism. This is in line with the findings of (Domhan 2018) that show that SA is less important in the decoder than in the encoder. Starting from the observation that separable convolutions have favorable FLOP count SA, (Wu et al., 2019)) introduces two models, Light and dynamic Convolutions, which replace the SA in the encoder and the decoder with layers based separable convolutions. While both models perforform AANs on inference speed, light convolutions can incur a significant drop in translation quality and in some cases dynamic convolutions are shown to outperform the Transformer. Note that these convolutions are different from standard CNNs, as there is parameter sharing across the heads; moreover, one can also look at SA as a separable convolution where the weight matrix not a model parameter, but is generated by the query-key pairs. the following we will refer to convolutions as separable ones. Approaches that channel SA with other operations within the same blocks include (Tay et al., (021)) that proposes a general family of models, called the Synthetizers, that use "synthetic" attention matrices. However (Tay et al. 2021) finds that for machine translation Synthetizers require channel-level combining with SA to close the gap with the Transformer. [Liu et al., 2021) also studies gMLPs on natural language tasks. They consider encoder-only models and do not study generative lasks. While on pre-training perplexily gMLPs match the Transformer, on downstream tasks combining gMLPs with a "tiny" Attenntto module the same layer is necessary close the gap with the Transformer. Enhancing the Transformer by running SA-layers
(2016)
parallel with convolutional ones has been explored in automated speech recognition (Gulati al. 2020), machine translation and language modeling (So et al. 2019) and computer vision
[Li et al. 2021
Note
3

---

Published in Transactions on Machine Learning Research (02/2023)
that (So et al. 2019: used Neural Architecture Search (NAS) to find the architecture; large model space is proposed and then pruned by optimizing the model quality on given task. For more recent approach to (NAS) that allows each layer to be different see (Fan et al. 2020;; however the results of such architecture scarches are rarely simple and can be hard to interpret and reuse. Note that we do not run convolutions and SA in parallel each block, but instead replace SA-layers with simpler and more local operations and stack the operations sequentially. The hybrid strategy of running convolutional layers instead of the MLP layers directly in the SA blocks (Lei al. 2018) proved not be competitive in our experiments (see Appendix [F) Our work is thus closer to the Sandwich Transformer (Press et al. 2020) that reorders the operations in the Transformer resulting in non homogeneous stacking strategy. The (Press et al. 2020) uses only SA as a sequence operation and does not improve over the Transformer baseline on machine translation. Finally, hybrid architeclures with RNN-decoders and Transformer Encoders were considered in (Chen et al. 2018); moreover (Chen et al. 2018) considered also hybrid Encoders combining RNNs and SA either at channel level or subsequent layers.
3 Model Description
While several operations have been proposed to replace SA, a more thorough categorization of the features of these operations is missing. For example, (Tay al.) 2021) introduces a framework that relies heavily on the notion of self-alignments and thus does not cover operations such gMLPs (Liu al. 2021 light and dynamic Convolutions (Wu et al., 2019). Similarly, (Domhan 2018) introduces language to describe models, layers and sequence operations, but treats each sequence operalion as black-box without trying to compare different operations in terms of interaction order, locality or computational complexity. On the other hand, LLu et al. (021) suggests that gMLPs have interaction order looking at their polynomial expansion. Wo start from this observation and look at sequence operation from two different angles:
1. When building the new representation of a time step, how many interactions appear in the calculation? For example SA requires combining together a query, key and a value, resulting in an interaction of order 3. LConv directly averages the time steps via a convolution, so the interaction order 1. We will make this idea more precise in the next subsection.
2. What is the size of the sequential context that can be used to build the new representation for a given time step? For example in LConv this range is related to the size of the kernel, whereas SA is naturally global over the whole sequence.
3.1 Interaction degree
The interaction degree of layer is the degree of the polynomial obtained by replacing its non-linearities with the identity function. Let consider some concrete examples starting with the Multi-Head Attention (Vaswani ct al. 2017). The input here is triplet (g, k u) of embedded sequences. We use the superscript h to denote the head, and the subscripts I and e to index, respectively, the location in the sequence and the specific channel. The operation is then defined as:
Attn(g. k. v)l.
Σ softmax
If we replace the softmax with the identity we obtain a linear combination of the monomials h h dh leading to the operation having a degree of order 3. Now let us consider L.Conv (wu al. 2019). The input is single embedded sequence z. Let Wh denote the convolutional kernel of width 2k + for the head h; then:
LConv(z)
Σ
wheh
*ttjcct
So we see that the operation L.Cov has order 1.
4

---

Published in Transactions on Machine Learning Research (02/2023)
Layer
Degree
Local
Attention
3
No
Mixer
1
No
Light Convolution
1
Yes
Dynamic Convolution
2
Yes
Random Synthetizer
1
No
Dense Synthetizer
2
No
gMLP
2
No
Table 1: Classification of sequence operations by degree and locality.
3.2 Locality
Besides the interaction degrec, the context each operation takes input. A global operation allows all the time steps to interact, while local one allows only the time steps within a window of size K to interact. Self-attention (SA), Mixer and Synthetizer are examples of global operations, while LConv and Dynamic Convolutions are examples of local ones. Besides capturing a local inductive bias, such operations can benefit from a reduction in the number of memory accesses during the computation, which is one of the bottlenecks of modern accelerators (Ma et al., 2018). We summarize our classification of existing layers in Table 1
analyze operations in terms of the second criterion, that is the size of
3.3 Gated Light Convolutions
The gMLP operation stands out as it achieves an interaction degree with a cheap point-wise product, and without using additional operations that act along the sequence dimension, as it the case, for example, in the Dense Synthetizer and the Dynamic Convolutions. The other ingredient of gMLPs is the use of gating through GELU non-lincarity (Hendrycks & Gimpel 2016). Note that in Table the only degree local operation is the Dynamic Convolution and there is no local analogue for gMLPs. We thus propose to obtain local version of gMLPs by proposing the Gated Light Convolution (GLConv):
GLConv(z, w)t, gelu(z(,)) -Coonngeelull
which is local and has interaction degree 2. Note that while standard gMLPs do not use heads, we partition the channel dimension into heads in the case of LConv. See Figure Lb for diagrammatic description of the layer. Note also that in this Section we use the convention of just looking at the sequence operation itself. On the other hand, a full-layer description, requires also to show how and w are obtained; therefore the full layer description is that Figure [b] Specifically, these are obtained applying an inner projection to the input of the layer, similarly how keys, values and queries are obtained in the case of SA. Similarly, the output of the layer also requires an outer projection as in the case of SA.
3.4 Lasagna Architecture
The goal of Lasagna is to stack diverse layers onto each other in order to optimize inductive biases, performance and efficiency. Lasagna's decoder is made up entirely of GL.Conv layers. The reasons for this are twofold. First, in both autorcgressive (AR) and non-autoregrossive (NAR) machine translation, the dccoder is called repeatedly and therefore its efficiency has an oversized effect on the model's overall efficiency. Lasagna's 's decoder aims to minimize of the relatively inefficient SA operation, while using convolutional operation instead. Previous work (Zhang et al. 2018 Domhan 2018) also shows that it is possible to simplify the operations in the decoder without hurting overall performance. As second reason, we found that among the various convolutional variants, GLConv had the highest degree of interaction of 2, while being still relatively cheap to compute with respect to alternatives like Dynamic Convolution.
If we restrict our search by making the assumption that blocks of the same layer follow each other, we thus conduct exact search by optimixing the lengths of the blocks of each layer type, 4.2 The
5

---

Published in Transactions on Machine Learning Research (02/2023)
motivation for using layers with lower interaction order at the beginning is discussed in 4.6 In the rest of this section we will describe the final architecture we found.
Lasagna's 's encoder is formed by stacking 2 initial LConv blocks, followed by 2 GLConv blocks and then by SA blocks. Besides the efficiency of the convolutional blocks, the encoder also captures a crucial local-ta-global inductive bias, whereby the initial layers build up representations for short sequences of tokens in the input sentence and the final SA layers create a fully global representation of the sentence. See Figure Ia for diagram. In initial experiments we found the local-to-global scheme to perform better than inverse global-to-local scheme. The Sicoder-Decodeer attention is still used in the standard Transformer architecture by letting cach decoder layer attend to the last encoder layer (Vaswani et al. 2017). We also experiment with two versions of Lasagna that have more layers in the encoder: Lasagna-9 that has 3 layers of each kind and Lasagna-12 12 that has 4 layers of each kind.
4 Experiments
4.1 General setup
To obtain a fair comparison between the performance of various architectures we use the same code base, Fairseq {Ott et al. 2019). for benchmarking all models and, after fine-tuning the training hyper-parameters on the bascline Transformer, we keep the hyperparameters fixed for all architoctures (Narang et al.) 2021). As recommended by Marie et al., 2021) we reproduce rather than copy results from published work, we report sacreBLEUU]] (Post. 2018) and do not use adhoc post-processing steps. For each model we select the best checkpoint based on the perplexity
validation data and use 3 training runs with different seeds to estimate BLEU scores. Except for English to Turkish (EnTr), we find the variance of different runs to be 0.1 BLEU points, while for (En'Ir) we estimate it to be 0.3. A complete hyper-paramete parameter setup can be found in the Appendix C For estimating inference speed we always decode 128 sentences in a single batch 1) to avoid the inference speed artefacts that might arise with very smaller batch sizes such as and 2) to consider a more realistic bulk-inference setting (Kasai et al., 2021). We train models at two scales, Base and Big, as defined in terms of the number of heads and channel dimensions in (Vaswani et al.) 2017). Obviously, the parameter count will vary compared to the Transformer when considering different architecture. We train and evaluate autoregressive Base models on V100 GPUs, whereas the use GPUs A 100 for models of size Big. Our code is available on GitHub at l _
4.2 Optimizing the stacking pattern
We first fix a 6-layer decoder consisting of GL.Covv and optimize the stacking scheme in the encoder. We consider the English to German (EnDe) task (see 4.3 for more details) at a model size corresponding to that of Transformer Big, matching the same number of heads and the embedding dimensions. To approximately match the parameter count of Transformer Big, Lasagna uses layers in the Encoder instead of 6, but still stays faster. Tn the search, we allocate the first x layers to T.Conv, the following y layers to GLConv and the remaining layers to SA; see grid search space in Figure 2(a) Having two fewer SA layers decreases the BLEU scores compared to the baseline Transformer. Most models that interleave some LConvs and some GLConvs perform similarly, with BLEU scores in the range [29.0, 29.3]. We choose the stacking scheme in Lasagna as it has the highest BLEU score (sce Fig. a)). Note also that the plausible assumption that one could just use GLConvs instead of LConv is contradicted by our experimental results as scores degrade more quickly on the line x = 0 than the line y ==0 as one moves away from the origin.
We then fix the layers in Lasagna's ' encoder and optimize layer configurations for the decoder. Results are reported in Figure b).. As we can match or outperform the Transformer baseline without using SA, we run fewer experiments that include SA and focus instead on the edges of the search space. The horizontal edge corresponds to replacing SA with LConv, the vertical edge to replacing SA with GLConv, while the diagonal edge corresponds to combining different numbers of LConv and GLConv blocks. As expected, in terms of inference speed we observe that is preferable not to include SA the decoder search space, see
signnauue in Appendix
6

---

Published in Transactions on Machine Learning Research (02/2023)
Appendix G for more details. However we do not find clear pattern in terms of BLEU scores, suggesting that the specific configuration of layers in the Decoder might matter less.
A different approach to optimizing the architecture might have used Neural Architecture Search (NAS), e.g. (So et al.
2019 Fan et al., 2020). However this would have presented some issues:
Search costs. (NAS) can incur high search costs. For example (So al. 2019: required 200 Google TPUs v2 for the search on the (EnDe) Machine Translation task; for the same task the scarch in (Fan
et al.
2020 required 75 GPUs days and an ad-hoc CPU-GPU colla storage.
Lack of compute-quality trade-off. (NAS) assumes that there is clear optimization objective, for example perplexity in (So et al. 2019 BLEU in (Fan et al. 2020). However, we are also interested in increasing model speed and using more simple sequence operations. This is more a multi-optimization problem, which requires finding a Pareto frontier of models, each of which is "optimal" after picking a speed/quality trade-off.
Architecture complexity. The outcome of (NAS) might be hard to interpret, especially in the case of diverse layers as in (Fan et al. 2020). We feel that is harder to improve models that are harder to interpret. To make the choice more understandable, we would need to add ad-hoc rules in the search. In our case, using some heuristics, we managed instead to reduce the search to a 2-dimensional space.
4.3 Autoregressive Machine Translation
For this task we consider four data benchmarks and two model sizes, where the baselines are, respectively, the Base Transformer and the Big Transformer. As standard benchmarks we consider 1M English to German (EnDe) and English to French (FnFr) as in (Vaswani et al., 2017). We prepare the data using the scripts in the Fairseq translation example To test more distant language pairs, that could benefit from more global inductive bias as captured by SA, we consider the WMT117
Chinese to English (ZhEn) benchmark, following the preprocessing in (Wu et al. 2019 Hassan et al. 2018 (about 20M pairs), and the WMT'17 English to Turkish (EnTr) benchmark following the preprocessing in (Zhang et al. 2018) (about 0.2M pairs). For both benchmarks the evaluation is performed on newslest17. Note that as the architecture search was conducted at model size Big on (EnDe), this should not be considered as a test case as we might. have overfitted on it. We therefore evaluate on additional language pairs and consider also the model size Base for (EnDe).
On (EnDe) at model size Base (Table 2 we find that our Lasagna architecture improves by 0.4 BLEU points over the Transformer baseline with 35% higher inference speed. By making the encoder deeper we can improve up to 0.7 BLEU points keeping the same higher inference speed. Note that using LConvs in the encoder and the decoder results in a significant drop of almost 1.5 BLEU points. Lasagna matches Dynamic convolutions on parameter count and inference speed with a gain of +0.2 BLEU points over Dynamic Convolutions. On this task and model size we also cousidered Mixer (Tolstikhin et al.j 2021) and the original gMLP with the "tiny" Self- Attentionn (Liu al.) 2021). We find that Mixer incurs a significant drop in translation quality, while gMLPs (using the "tiny" SA) can close the gap with the Transformer but incur big drop in inference speed. We then move to model size Big (Table 4 as said above this should not be considered as a test scenario, but we need still to compare the performance to the other baselines. We observe that models perform comparably on translation quality except for LConvs and Mixer. For Mixer the situation improves at this model size, with just a small drop in translation quality, but there is no significant improvement on the inference speed compared to the baseline. Note that in the case of Lasagna models we can slightly improve the BLEU score (up to +0.4) while increasing the inference speed by about 20%. Note also that model size Big we did not observe an increased decoding speed by using Dynamic Convolutions. Finally, in both settings we saw that inference speed correlates well with training speed and Lasagna reduces the training time by about 25%. While we evaluate models on BLEU, for (EnDe) at model size Big we also ran an evaluation on model- based metrics, using the framework COMET (Rei et al. 2020); we find that Lasagna and Transformer are on par, while (12) outperforms the Transformer, see Appendix A

---

Published in Transactions on Machine Learning Research (02/2023)
Model
BLEU
Params (M)
tokens/ /s (k)
L1sagna(12)
27.2 (+0.7)
81
3.91 (+35%)
Lasagna
26.9 (+0.4)
67
3.92 (+35%)
Lasagna(9)
26.8 (+0.3)
72
3.92 (+35%)
Dynamic Conv.
26.7 (+0.2)
67
3.69 (+28%)
Transformer
26.5 (+0.0)
66
2.88 (+0%)
gMLP as in (Liu al. 2021
26.4 (-0.1)
67
1.62 (-44%)
Mixer
25.7 (-0.8)
65
3.71
(+29%)
Light Conv.
25.1 (-1.4)
66
4.12
(+43%)
Table 2: (EnDe) Base Model. Our Lasagna models outperform the Transformer baseline Translation quality, while achieving the same inference speed as Dynamic Convolutions. The original gMLP (Liu et al. 2021) architecture with tiny attention is substantially slower than the Transformer. Mixer and Light Convolutions achieve lower BLEU scores than the baseline. For details see 4.3
For the baseline the Evolved Transformer (So et al. 2019 we were not able to re-implement it in our codebase. But the authors Table 3 (So et al. 2019) report a comparison on the same WMT En-De benchmark that we use after making the training conditions exactly the same for the Transformer and the gains
Evolved Transformer. We can thus compare the same way as in (So el al. 2019). With respect to this Transformer baseline, we achieve a +0.4 gain for Lasagna at model size Base vs a +0.5 gain for the Evolved Transformer; Lasagna also achieves +0.2 gain at model size Big vs a +0.2 gain for the Evolved Transformer. Note that the baselines for these gains are different as each model is compared to its respective Transformer baseline. Moreover, at model size Big we can report a comparison with NAS search (So et al. 2019 Fan al. 2020) using BLEU instead of SacreBLEU, At model size Big our Lasagna achieves 30.2 29.8 of the Evolved Transformer, and 30.1 of (Fan et al., 2020). We therefore match the score in (Fan el. al. 2020) without an expensive NAS search and simpler model architecture. We note however that, although presented with the greatest possible care, such comparisons have to be taken with a grain of salt as the bascline is quoted rather than reproduced under the same training conditions (Narang et al. 2021
the Transformer computing the BLEU score in
For (ZhEn) we observe a similar picture. model size Base (Table 3 we find that Lasagna and Lasagna- 12 achieve +0.4 and +0.8 over the bascline while maintaining higher decoding speed. At model size Big (Table 5 Lasagna can stiil outperform the baseline on BLEU score (+0.3) and inference speed (+10%). Here observe that using L.Covvs in the Decoder results in decrease in BLEU.
For (EnFr) and (EnTr) we report the complete experimental results in the Appendix B and summarize here the main findings. For (EnFr), at model size Base we can improve BLEU by +0.9 and inference speed by 30,, while at model size Big we can improve BLEU by +0.3 and inference speed by +28%. For (EnTr), we find that the Transformer achieves a test BLEU of 17.7 while Lasagna-9 achieves 18.2 at comparable inference speed.
We also used bootstrap resampling to estimate the probability that Lasagna scores higher than the Transformer baseline. Overall, see that these align with the increases in BLEU scores; details can be found in Appendix
4.4 Non-Autoregressive Machine Translation
For non-autoregressive Machine Translation we took baseline the Levenshtein Transformer (Gu et al. 2019 using the Fairseq implementatiorf] This model uses the decoder to iteratively refine the translations using a post-editing process learned through imitation learning. We were unable to reproduce the original result. of [Gu et. al.) 2019), where the non-auttoregressive model matches the auto-regressive baseline. This was the case both when training with our own distillation data, obtained from our best (EnDe) model at size Big, as well as with the distillation data released with Fairseq; we report results based on the latter. We find
http:///hhhhbmmpppt/erreeee http:///wwwt.ooooooooo/////////m transsaatnnnnnn

---

Published in Transactions on Machine Learning Research (02/2023)
Model
BLEU
Params (M)
tokens/s (k)
Lasagna(12)
23.3 (+0.8)
118
3.70 (+9%)
Lasagna
22.9 (+0.4)
104
4.01 (+19%)
Lasagna(9)
22.7 (+0.2)
109
3.77 (+12%)
Dynamic Conv.
22.6 (+0.1)
105
3.64 (+8%)
Transformer
22.5 (+0.0)
104
3.38 (+0%)
Light Conv.
22.5 (+0.0)
100
3.98 (+18%)
Table 3: (ZhEn) Base Model. Our Lasagna models outperform the Transformer and Dynamic Convolutions baselines both on Translation quality and inference speed. For details see 4.3
Model
BLEU
Params (M)
tokens/s (k)
Lasagna(12)
29.5 (+0.4)
277
3.77 (+18%)
Lasagna
29.3 (+0.2)
221
3.86 (+21%)
Lasagna(9)
29.1 (+0.0)
243
4.00 (+25%)
Transformer
29.1 (+0.0)
221
3.19 (+0%)
Dynamic Conv.
29.0 (-0.1)
224
3.30 (+3%)
Mixer
28.8 (-0.3)
206
3.14 (-1%)
Light Conv.
28.6 (-0.5)
206
3.22 (+1%)
Table 4: (EnDe) Big Model. Increasing the scale the differences on Translation quality as measured by BLEU become smaller. Even Mixer and Light Convolutions almost match the Transformer and Dynamic Convolution baselines. Our Lasagna models outperform the baselines in terms of inference speed and BLFU. For details see 4.3
Model
BLEU
Params (M)
Lokens/ /s (k)
Lasagna
24.7 (+0.3)
296
3.78 (+10%)
Lasagna 9)
24.6 (+0.2)
318
3.89 (+13%)
Lasagna(13) (12)
24.6 (+0.2)
353
3.65 (+6%)
Transformer
24.4 (+0.0)
296
3.45 (+0%)
Dynamic Conv.
24.2 (-0.2)
299
3.31 (-4%)
Lasagna(9) + LConv Decoder
24.0 (-0.4)
312
3.79 (+10%)
Light Conv.
24.0 (-0.4)
281
3.46 (+0%)
Table (ZhEn) Big Model. Our Lasagna models slightly outperform the Transformer baseline BLEU and both the Transformer and Dynamic Convolutions inference speed. Note how using Light Convolutions instead of Light gMLP Decoder results in -0.4 drop in BLEU. For details see 4.3

---

Published in Transactions on Machine Learning Research (02/2023)
Model
BLEU
Params (M)
tokens/s /s (k)
Lasagna(12)
25.3 (+0.9)
80
5.29 (+39%)
Lasagna (9) + LConv Decoder
24.9 (+0.5)
71
5.56 (+46%)
Lasagna(9)
24.7 (+0.3)
71
5.47 (+44%)
Lasagna
24.6 (+0.2)
66
5.26 (+38%)
Dynamic Conv.
24.5 (+0.1)
66
5.05 (+33%)
Transformer
24.4 (+-0.0)
66
3.81 (+0%)
Light Conv.
23.8 (-0.6)
62
5.81 (+52%%
Table 6: (EnDe) using the Levenshtein Transformer. Our Lasagna models improve Translation quality over the Levenshtein Transformer baseline decoding speed. For details see 4.4
Gu al. 2019 up to +0.9 BLEU points with +-40% increase in
Encoder
Decoder
BLEU
Params (M)
tokens/s /s (k)
Transformor(9 (6)
Transformer(())
24.4 (+0.0)
296
3.45 (+0%)
Transformer(12)
GLConv(1)
23.8 (-0.6)
286
10.71 (+210%)
(asagna(12)
GL.Covv (1)
23.6 (-0.8)
274
10.69 (+210%)
Lasagna 12)
Transformer (1)
23.5 (-0.9)
275
10.16 (+194%)
Transformer(122)
Transformer(1)
23.5 (-0.9)
288
8.65 (+150%)
Table 7: (ZhEn), Deep Encoder / Shallow Decoder as in (Kasai et al. 2021). Using a Light gMLP decoder increases decoding speed by 60%. Replacing a Deep Transformer Encoder with a Lasagna one increases decoding speed by 40% without affecting the BLEU score. Note however, how the original Deep Encoder Shallow Decoder using SA suffers loss of almost 1 BLEU point to the Transformer baseline. For details see
that the Lasagna model outperforms the Transformer also in this setting (Table 6 with improvements up to +0.9 BLEU points and about +400 increase in decoding speed. Note that in the case of the Levenshtein Transformer the decoder is called for fixed number of iterations on the whole sequence, so the complexity analysis is different from the auto-regressive case; see Kasai et al. 2021 for details.
4.5
Deep-shallow models
Recent work (Ksai al. 2021) questions whether non-autoregressive Machine Translation is actually faster than aulo-regressive one; they propose aulo-regressive models with Deep Encoder and Shallow Decoder faster alternative for larger batch sizes, corresponding the bulk-inferent case. Specifically, they propose Transformer Encoder with 12 layers and a Transformer Decoder with single layer. So far measured speed-upe ups for models where the encoder and the decoder have comparable depth, and the difference might become negligible when the decoder consists of a single layer. Moreover, a decoder using only local operations might take a considerable hit on Translation quality compared to one using SA. We thus test Lasagna also in this setting.
We observe (Table 7 that replacing the Transformer encoder with Lasagna-12 (with just one layer in the decoder) results in the same BLEU score but increased inference speed (+60%). Note also that the best. BLEU score not achieved by Lasagna, but by combining standard Transformer encoder with an CLConv-decoder i.e. a decoder consisting of a single layer employing a Gated Light Convolution; on the other hand, using a docoder with a single Transformer layer results in a decrease of -0.3.
4.6 Ablation studies
We evaluate the impact of various choices that we made in Table We use (EnDe) model size Base. We first invert the order of the layers in the encoder (Inverted Lasagna) to compare what happens if we decide to
10

---

Published in Transactions on Machine Learning Research (02/2023)
Encoder
Decoder
BLEU
Params (M)
tokens/ /s (k)
Lasagna
GLConv
26.9 (+0.4)
67
3.92 (+35%)
Lasagna
LConv
26.7 (+0.2)
66
3.87 (+34%)
Lasagna, GL.Conv - L.Conv
GLCCov
26.7 (+0.2)
66
4.04 (+40%)
Dynamic Conv.
Dynamic Conv.
26.7 (+0.2)
67
3.69 (+28%)
Transformer
LConv
26.5 (+0.0)
63
4.07 (+41%%
Translormer
GL.Conv
26.5 (+0.0)
65
3.95 (+37%)
Transformer
Transformer
26.5 (+0.0)
66
2.88 (+0%)
Inverted Lasagna
GLConv
26.3 (-0.2)
67
3.97 (+38%)
Lasagna, SA GLConv
GLConv
26.2 (-0.3)
66
3.71 (+29%)
L.Conv
I.Conv
25.1 (-1.4)
66
4.12 (+43%)
Table 8: Ablations on design choices for the Lasagna. We show that both the order of layers (from local to global) and the use of GL.Covs matter. We show that the Transformer Decoder can be replaced by one using GLConvs or LConvs without affecting the BLEU score. For details see 4.6
first work with global operations and then use local ones. This has big negative impact as the performance decreases by -0.6 BLEU points relative to Lasagua. Secondly consider what happens if we replace the SA layers in the encoder with GLConvs. In this case the performance also decreases by -0.7 points. However, note that in both cases the performance drop compared to the Transformer baseline small. This ablation suggests that the locality prior of LConv and GLConv complements the global attention prior of SA when LConv and GL.Conv are used before SA. Both inverting the order of layers Lasagna as well as removing SA from Lasagna hurt translation quality.
Note that in the (AR) setting the Encoder is called once but the Decoder is called multiple times. In this setting in which the number of layers between the Encoder and the Decoder is the same we found that the inference speed depends on the Decoder only. For example using the Lasagna Encoder or the Transformer Encoder with a decoder using GLConvs results in the same inference speed. Therefore, in this setting the Decoder is responsible for the inference speed gains and the Encoder for the slightly improved translation result (+0.4 BLEU points). We thus compared the time for the forward pass in the Lasagna Encoder and the Transformer Encoder for batch size 64 and found that the former achieves about a +30% speed increase. The situation is different in the case of the Deep and Shallow models in Table 7 where the Decoder consists of just one layer and the Encoder has 12 layers. In this setting we found the speed-up in the Tasagna Encoder also to have an effect, even when combined with a Transformer Decoder. Finally we warn the reader that the speed measurements might depend on the framework and the computational model. We used Fairseq (Ott et al. 2019 which uses PyTorch thus allowing the batch size to be readjusted during beam scarch as computations are only done on the hypotheses that need to be continued. The speed is therefore affected also by the nunber of tokens generated during the search process. On the other hand, frameworks computational model requiring fixed batch size, and therefore they do forward
like Tensorflow JaX passes also hypotheses that do not need to be continued.
We then look at the role of the GLConvs in the encoder and the decoder. If we replace them with LConvs there is a small impact of -0.2 points. Note however that for (ZhEn) at model size Big (Table 5 the impact was larger with a -0.4 drop compared to the Transformer baseline (Table 5 Combining a Transformer encoder with a decoder using LConvs or GLConvs does not improve the BLEU score over the baseline.
5 Conclusions
We have proposed a novel architecture for machine translation that improves over the Transformer in terms of translation performance and inference speed across a large number of benchmarks and model sizes. The architocture stacks different layers onto each other in order to capture multiple inductive biases and increase performance while increasing inference speed; as result, most of the self-attention layers that are standard in the Transformer replaced by lower degree and more local operations. We hope that our work spur
11

---

Published in Transactions on Machine Learning Research (02/2023)
more interest in this sort of heterogeneous architectures and result in models that more efficient to train and deploy than the Transformer.
Broader Impact Statement
The architecture proposed here can reduce the energy consumplion for training and deploying Neural Machine Translation Models, as training time reduced and inference speed is increased. Despite the advances in Machine Translation, such systems may still produce incorrect translations, and this can negatively affect their users. However this negative impact affects Neural Machine Translation in general, and is not specific to the methodology proposed here.
Acknowledgments
We wish to thank the anonymous reviewers for their insightful comments and suggestions that helped to improve the exposition.
References
Dmitrii Aksenov, Julian Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg, Leonhard Hennig, and Georg Rehu. Abstractive text summarization based on language model conditioning and locality modeling. In Proceedings of the 12th Language Resources and Evalvation Conference, pp. 6680 6689, Marseille, France, May 2020. European Language Resources Association. ISBN 199-10955556-34-4 URL sttps://aclanthol// www.g//.rg/000000 1rec-leeee.. 1.825
Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 2003.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 76-86, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v/ /1/1//1101P0.. URL http:: htttp:::/aclannhooogy.oooo org//18--188
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In International Conference on Learning Represenlations, 2020. URL https:
//openrevvew.net// http::/wwww...oorrrriiinnnnnnnn.
Tobias Domhan. How much attention do you need? a granular analysis of neural machine translation architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Lúnguistios (Volume 1: Long Papers), pp. 1799 1808, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P11 1167. URL /ttp:ttt////cllaatooloogg http://ccaaaaaoooooooororr///eee otg/p18- 1167
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.
Yang Fan, Fei Tian, Yingce Xia, Tao Qin, Xiang- Yang Li, and Tie- Liu. Searching better architectures for neural machine translation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 1574-1585, 2020. doi: 10.1109 TTT.TT..............00
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1243-1253 PMIRR 06-11 Aug 2017. URL httph:ttp///t httppp::///rccceddngneee/o mlr. press//ee//.0///eeeee//
higpr//inn.......eee///////////n hwwp:///w/gggrggg....o html.
Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein Transformer. NewrIPS, 2019.
12

---

Published in Transactions on Machine Learning Research (02/2023)
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolation-augmennee Transformer for Speech Recognition. InterSpeech, 2020.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Janczys-Dowmmne William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingee Xia, Dongdong "hhnng Zhirui Zhang, and Ming Zhou, Achieving Human Parity on Automatic Chinese to English News Translation. Xiv e-prinls, art. arXiv:1803.05500 March 2018.
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv e-prints, art. WrXiv:1606008411 June 2016.
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. Xi preprint 1rXio:1610.10000 2016.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation. ICLR, 2021.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguislies (Volumee 1: Long Pupers), pp. 284 2944 Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1 /P10//11PP1112/// URL httpps//acc.mtthooooo..o////////
Tao Lei, Yu Zhang, Sida 1. Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4470-4481, Brussels, Belgium, Dectobr-NNovembr 2018. Association for Computational Linguistics. URL https::/acclmtthooog..oooo
doi: 10. 10.18653/11/D133
1477
Yawci Li, Kai Zhang, Jiczhang Cao, Radu Timofte, and Luc Van Gool. LocalViT: Bringing Locality to Vision Transformers. Xixi e-prinls, art. arXiv:2104.05707, April 2021.
Hanxiao Liu, Zihang Dai, David R. So, and Quoe V. Le. Pay attention mlps. In NewrIPS, 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient architecture design. In Proceedings of the European conference computer vision (ECCV), pp- 116-131, 2018.
Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Proceedings of the 59th Annaul Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7297-7306, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 /202121222222 acc-lo..cloooo.5e0 URL htttat///a arg/2021.acl1-ooonn 566
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolutional context. for ASR ar e-prints, art arXiv:1904.11660, April 2019.
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel Do transformer modifications transfer across implementations and applications? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5758 5773, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 22t/://11122/e//// wewnlpmmainn4eee m65. URL htttps://aclaatto///
org/ emnlp-main. 465
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangicr, and Michael Auli fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
13

---

Published in Transactions on Machine Learning Research (02/2023)
Matt Post. A call for clarity in reporting BLEU scores. In WMT, 2018. URL https://www.aclwb) anthology/W18-633
Ofrr Press, Noah A. Smith, and Omer Levy. Improving transformer models by reordering their sublayers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2996-399990 3005, Online, July 2020. Association for Computational Linguisties. doi: 1118663//112022.acc--a https://aclanthollgg. wrg.llogg..rggggooo acl main. 270.
URL
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Unbabel's participation in the WMT20 metrics shared lask. In Proceedings of the Fifth Conference on Machine Thanslation, pp. 911 -200 Online, November 2020. Association for Computational Linguistics. URL httpe://aclaathoooggy wrg/2000 wmt- 1. 101
David So, Quoe Le, and Chen Liang. The evolved transformer. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5877-5886. PMLR, 09-15 Jun 2019. URL https
//proceedings.. m1r. www.pp//ss/////....... html
Yi Tay, Dara Bahri, Donald Metzler, Da- Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. ICML, 2021.
Nly Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. NewrIPS, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with light and dynamic convolutions. In ICLR, 2019.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollar, and Ross Girshick. Early con- volutions help transformers see better. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 30392-30400. Curran Associates, Inc., 2021. URL httpss:/pproceeddngg www.iig....urss.... ct/ccc/ /c/paperr/021//illl/ pdff
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. Convolutional self-attention networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4040 4045, Minneapolis, Minnesola, June 2019. Association for Computational Linguistics. doi: (0.18633/v1/N199
URL
httpr/hooooooooo//11-----1-1 org/N19-
Adams Wei Yu, David Dohan, Quoe Le, Thang Luong, Rui Zhao, and Kai Chen. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations, 2018. URL
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via. an average attention network. In Proceedings of the 56th Anrual Mesting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1789-1798, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653 vv //t////1/PP1//// 1168-1666. URL http:https:///e htttps://ww..cclveb.b. aclweb. wttpae/w..rrrggaaooooo/-/-------
A Appendix: Significance Testing and Model-based Evaluation.
We used BLEU as the main evaluation metric. In order to assess the significance of the BLEU score differences, we used bootstrap re-sampling to compute the Win Rate between Lasagna and the Transformer Bascline. Specifically, we repeatedly sample subsets of the test set and compute the probability that the BLEU score of Lasagna is higher than that of the Transformer. We feel that using the Win Rate makes the comparison
14

---

Published in Transactions on Machine Learning Research (02/2023)
Model
Language Pair
Model Size
Win Rate BLEU
Lasagna
EnDe
Base
85%
Lasagna(9)
EnDe
Base
69%
Lasagna(12)
FnDe
Base
95%
Lasagna
EnDe
Big
87%
Lasagna(9)
EnDe
Big
41%
Lasagna(12) (12)
EnDe
Big
70%
Lasagna
ZhEn
Base
94%
Lasagna(9)
ZhEn
Base
68%
Lasagna(12 (12)
ZhEn
Base
100%
Lasagna
ZhEn
Big
89%
Lasagna(9)
ZhEn
Big
93%
Lasagna(12)
ZhEn
Big
78%
Lasagna
EnFr
Base
77%
Lasagna(9)
EnFr
Base
88%
Lasagna(12
EnFr
Base
100%
Lasagna
EnFr
Big
72%
Lasagna(9)
EnFr
Big
97%
Lasagna(12)
EnFr
Big
97%
Lasagna
EnTr
Base 86%
Lasagna(9)
En'r
Base
72%
Lasagna(12)
En
Base
60%
Table 9: Win Rates for Lasagna over the Transformer for BLEU. Overall the Lasagna systems score higher, i.c. the Win Rate is > 50%.
Model
wmt20-comet-da
pval
wmt21-comet-nn
pval
Lasagna (12)
0.3731
0.002
0.0310
0.0005
Lasagna (9)
0.3622
0.2564
0.0308
0.2329
Lasagna
0.3585
0.56
0.0307
0.9
Transformer
0.3553
0.0307
Table 10: (EnDe) Big Model. Comparison of Lasagna models with the Transformer model-based metrics. We report the p-values of paired t-test with the Transformer scores.
between a system and the baseline more intuitive, as it gives an unbiased estimate of the probability that system scores higher than the baseline.
The results of this significance estimation are in Table Overall the Lasagna systems score higher than the Transformer counterpart.
For (EnDe) models size Big also evaluated model-based metries, using the the COMET implementa- tion Rei et al. 2020 Scores are produced by model that learns predict human evaluations. Results are in Table 10 for the metrics wmt20-comet-da and wmt21-comet-mom.
Appendix: Tables for English to French and Turkish.
We report our experimental results on (EnFr) model size Base in Table 11 and Big Table We report experimental results on (En'Tr) at model size Base in Table 12 Note that Tight. Convolutions perform
15

---

Published in Transactions on Machine Learning Research (02/2023)
Model
BLEU
Params (M)
tokens /s (k)
Lasagna (12)
38.3 (+0.9)
81
3.83 (+30%)
Lasagna(9 (9)
37.9 (+0.5)
72
3.92 (+33%)
Dynamic Conv.
37.9 (+0.5)
69
4.00 (+36%)
Lasagna
37.8 (+0.4)
67
3.76 (+28%)
Transformer
37.4 (+0.0)
67
2.94 (+0%)
Light Conv.
37.2 (-0.2)
63
4.30 (+-46%)
Table 11: (EnFr) Base Model. Our Lasagna models outperform the Transformer on BLEU and inference
speed. Light convolutions also good baseline, probably because of the closeness between language pairs.
Model
BLEU
Params (M)
Lokens/s (k)
Lasagna(9)
18.2 (+0.5)
66
3.13 (+4%)
Lasagua
18.0 (+0.3)
60
3.18 (+5%)
Lasagna(12)
17.8 (+0.1)
75
3.25 (+8%)
Transformer
17.7 (+0.0)
60
3.02 (+0%)
Dynamic Conv.
17.5 (-0.2)
60
3.19 (+6%)
Light Conv.
16.8 (-0.9)
60
3.31 (+10%)
Table 12: (EnTr) Base Model. Differences S 0.3 on BLEU are not significant. Lasagna(9) improves over the Transformer while Light Convolutions drop by almost BLEU point.
competitively on (EnFr) but experience a significant drop on (EnTr), perhaps because these are more distant language pairs and Turkish word order less rigid than English.
C Appendix: Training Hyper-parameten.
We preprocess the data with Fairseq Ou et al. 2019) using BPE vocabularies with splits and training syper-parametery reported in Table 14 Note that we sometimes simulate training on a larger number of GPUs by using gradient accumulation steps, which in Fairsoq are specified with the -update-freg parameter.
There are three dropouts: one for the MLPs, one for atiention and one for the convolutional weights. For (EnDe) use, respectively, 0.3, 0.1 and 0.1; for (EnFr) 0.1, 0.1, 0.1; for (ZhEn) 0.2, 0.2, 0.2; for (En'I )0.3, 0.1, 0.1.
For the learning rate warmup use linear schedule to the peak rate with 4k steps at model size Base and (EnFr) at model size Big, 10k at model size Big for (EnDe) and (ZhEn) and NAT.
Model
BLEU
Params (M)
tokens/s (k)
Lasagna(9)
42.1 (+0.3)
245
4.22
(+28%)
Lasagna 12)
42.1 (+0.3)
279
4.13 (+25%)
Lasagna
41.8 (+0.0)
222
4.15 (+26%)
Dynamic Conv.
41.8 (+0.0)
225
3.93
(+19%)
Transformer
41.8 (+0.0)
222
3.30
(+0%)
Light Conv.
41.6 (-0.2)
207
4.20
(+27%)
Table 13: (EnFr) Big Model. Our Lasagna models improve the Transformer for BLEU and inference speed.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Model size
Task
GPUs
Steps
Tok. in Batch
Grad Acc.
Lr schedule
Peak
Vocab splits
Base
(EnDe)
8 V100
120k
4k
inverse square root
5 x 10 4
32k (shared)
Base
(EnFr)
8 V100
120k
4k
1
inverse square root
5 10
40k (shared)
Base
(ZhEn)
8 V100
120k
4k
1
inverse square root
5 10-4
32k (separate)
Base
(EnTr)
V100
10k
4k
inverse square root
10-3
32k (shared)
Big
(EnDe)
16 A100
30k
3.6k
4
cosine decay
10-3
32k (shared)
Big
(ZhEn)
16 A100
40k
3.6k
cosine decay
10-3
32k (separate)
Big
(EnFr)
16 A100
120k
5k
2
iuverse square root
5 x 10-4
40k (shared)
NAT
(EnDe)
8 A100
300k
8k
1
inverse square root
5 x 10
32k (shared)
Table 14: Hyperparameters for Training. Tok, in Batch is the maximum number of tokens in a batch on a single accelerator, while grad accumulation the number of gradient accumulation steps reach desired mumber of tokens to update When the vocabulary splits are shared it means there is single shared vocabulary between source and target languages.
The embedding dimensions and the number of heads are the standard ones for model Base and Big. For Base we use 512 embedding dimensions and 8 heads, for Big 1024 embedding dimensions and 16 heads. The kernel sizes for Light and Dynamic Convolutions are like those in [Wu et al. 2019)
D Appendix: Inference Hyper-parameters
We use beam size For (EnDe), (ZhEn) and (EnFr) we finc-tuned the length penalty for the Trans- former the validation set and the same one across all models. We use 0.6 for (EnDe) and (EnFr) and 1.4 for (ZhEn). For (EnTr) we fine-tune the length penalty on the validation set for each model, we consider the range 0.4 to 1.8, in increments of 0.1. For sacreBLEU we use the signature case. wwwedtnrefs.......m... 11smooth. wwwsmoot.expttoo.... 13atv.1.5.1.
E
Appendix: Speed Gains and Batch Size
In Figure we plot the relative speed gain over the Transformer measured at different batch sizes. We take the Lokens/s of each model relative to the tokens/s of the Transformer. We that the relative gains can change with the batch size. However, the oscillations tend to be less than 10%. We report 128 to simulate bulk-inference case in which predictions are batched together.
F Appendix: Gating-augmentee Feed Forward Layers
We compare the approach using the Lasagna model with an alternative route that enhances SA, motivated by the hypothesis that one should complement rather than replace SA; see (Tay et al.] 2021 Lei et al., 2018). In Lei al.] 2018) the feed-forward layers were augmented with recurrent operations, but was found that increasing translation quality would result into slightly decreasing inference speed. We observe that the recurrent operation of SRUs relies on two main ingredients: 1) usage of two gating operations, 2) light recurrent operation based on sort of averaging mechanism (compare with (Zhang et al. 2018 ) However, an efficient implemenation of SRUs requires a somewhat-involved ad-hoc CUDA kernel and an adhoc initialization schemn. We thus investigate enhancing the wedd-forwrdd layers with the simpler GLConvs, which also uses gating and an averaging mochanism through the convolutions, and has also a straightforward implementation. In order not to slow down too much the decoder, we opt for the following architecture: a Transformer encoder with GL.Convs replacing the forward layers and the Lasagna decoder.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Tokens/s (Relative to Transformen
1.5
1.4
13
1.2
Transformer
1.1
Lasagna
Light Conv.
Dynamic Conv.
1.0
000
+30%
1
32
64
128
256
Batch Size
Figure 3: The speed gains over the Transformer do not depend much on the batch size.
Model
BLEU
Task
Params (M)
tokens/s (k)
Lasagna
24.7
(ZhEn)
296 (-12%)
3.78 (+42%)
galing-augmented
24.6
(ZhEn)
337
2.60
Lasagna
29.3
(EnDe)
221 (-17%)
3.86 (+69%)
gating-augmenter
29.2
(EnDe)
259
2.29
galing-augnented
42.0
(EnFr)
260
3.39
Lasagna
41.8
(EnFr)
222
(-17%)
4.15 (+22%)
Table 15: Our Lasagna model is more effective than augmenting the feed-forwards with Gated Light Convolutions. Less parameters are required to match the same BLEU score and inference speed improves significantly. For details see Appendix
18

---

Published in Transactions on Machine Learning Research (02/2023)
G Appendix: Optimizing layers
We report in Figure the results on speed for the layering search for the decoder. The Lasagna encoder already results in a speed gain over the Transformer, even when the decoder uses only SA. However, after removing few layers of SA most speed results are comparable, given a variance of +..1k tokens/s.
19

---

Published in Transactions on Machine Learning Research (02/2023)
.3.86
6
Nr. of Gated Light Conv. layers
.3.89
.3.92
5
.3.73
.3.97
4
.3.94
.4.03
3
.3.76
.4.10
2
.3.53
.3.97
1
.3.38
.3.43
,3.82
.3.84
.3.96
.3.74
.4.44
0
i
4
Nr. of Light Conv. layers
Figure The speed results corresponding to Figure 2
20


Reviews:
[{"review_id": "wzWbdkFcyY", "reviewer": "Reviewer_8B11", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper is written with clarity.\n- The empirical results are strong and detailed analysis are provided to demonstrate the utility of each design.\n\nWeakness:\n- It's unclear if the proposed architecture is specific to MT, or it can be used as a general architecture for pretraining (like Transformers).\n- It would be nice if the related work engages with the NAS literature.\n\nOverall, this is a solid paper and I'd recommend acceptance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a non-uniform variant of the transformer architecture, where the lower layers focus on learning local interactions using convolutional layers and the upper layers learn global interactions using self-attention layers. This hybrid architecture allows the model to have efficient training and inference. Experiments on machine translations demonstrate the advantage of the proposed architecture in accuracy and speed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be nice to add discussions regarding the two points in \"weakness\" above, but this is not critical to my recommendation.", "broader_impact_concerns": "None observed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DmwacJxLXI", "reviewer": "Reviewer_XryH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper is well-written. \n2. This paper introduces a well-designed encoder-decoder architecture for solving neural machine translation tasks.\n\n**Weaknessnes**\n1. Although some significant progress has been achieved, the novelty of this paper is still limited. Combining CNN and Transformer is not an interesting idea, and has been adopted in many works [1][2][3][4] (including manual design and automatic design). Therefore, it is not surprising that combining self-attention with cnn (lightConv or Gated LightConv) could achieve better results. Locality in machine translation or other generation tasks also has been studied in many works [5], which may also reduce the contribution of this paper.  \n2. Just as aforementioned, the design of Lasagna is completely hand-crafted. It lacks enough insights or motivation to prove why such a design is optimal. For example, considering the components used in this paper as a search space (LConv, gated LConv and Self-attention), can we use NAS to obtain a better architecture?\n3. The improvements are marginal, which is not satisfactory. For example, In EnDe base and Large setting. Lasagna achieves 26.9 (+0.4) and 29.3 (+0.2) over the Transformer baseline, which are not significant. \n\n[1] Searching Better Architectures for Neural Machine Translation\n\n[2] Convolutional self-attention networks\n\n[3] ON THE RELATIONSHIP BETWEEN SELF-ATTENTION AND CONVOLUTIONAL LAYERS\n\n[4] QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension\n[5] Convolution sequence to sequence learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper points out that the existing works for solving translation tasks mainly ignore the various inductive biases from different architectures. Therefore, to leverage the advances from different architectures, this paper proposes Lasagna, an encoder-decoder model, which combines convolutional networks and self-attention networks. Experimental results also demonstrate that the proposed Lasagna can outperforms the Transformer baseline. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The biggest problem of this problem is its novelty. Many works have investigate the locality problem in machine translation, and introduces different designs by combining transformer and cnn layers. Authors should provide convincing justifications to prove the necessity of the proposed architectures. \n2. I want to check one detail: why Lasagna(12) has more parameters than Lasagna, but achieves faster speed? And authors argue that using a deeper encoder can achieve better performance with a higher speed (Page 6). This is really counterfactual and can authors check it again?", "broader_impact_concerns": "There is no concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GKjGq76F4J", "reviewer": "Reviewer_3qo6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper covers multiple language pairs (both to/from English), tasks and model sizes.\n\nThe proposed approach may improve translation quality while maintaining model size and accelerating inference speed. It is also quite simple.\n\n**Weaknesses**\n\nThe architecture search (section 4.2 and figure 2) is conducted on the En-De test set. There is a risk of overfitting to the test data.\n\nThe description of gated light convolutions is unclear. How does the equation in section 3.3 relate to figure 1(b)?\n\nSome additional work is missing. In particular, similar work in computer vision and ASR should be cited (e.g. [1] and [2]). Convolutional NMT [3] should also be cited.\n\n[1] Xiao et al. Early Convolutions Help Transformers See Better. 2021. NeuIPS\n[2] Mohamed et al. Transformers with convolutional context for ASR. 2019. https://arxiv.org/pdf/1904.11660.pdf\n[3] Gehring et al. Convolutional Sequence to Sequence Learning. 2017. ICML", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "For neural machine translation, this paper replaces the transformer encoder by a stack of convolutional and self-attention layers. The convolutions aggregate local information, while the self-attention layers combine global information. In particular, the paper introduces gated light convolutions as a cheap, local and expressive operation. The decoder is also modified to use such convolutional layers.\n\nOver many machine translation datasets and tasks, the proposed architecture can achieve slightly better results than the Transformer while using approximately as many parameters and improving the latency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarify that the En-De task should not be considered as a test scenario because architecture search was conducted on it. Critical.\n\nAdd additional related work mentioned above. Critical.\n\nMore clearly explain the benefits of the proposed changes in the encoder and decoder separately. In Table 8, Transformer encoder + GLConv decoder is faster than Lasagna encoder + GLConv decoder. However, appendix G mentions \"The Lasagna encoder\nalready results in a speed gain over the Transformer, even when the decoder uses only SA.\". Critical.\n\nFix a few typos (p1.suit, p6 outpeforms, p9 BLUE). Not critical.\n", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores new architectures for machine translation. The main insights are convolution for local interactions and self-attention for global interactions. The review and discussion processes are good: reviewers have provided good comments and suggestions, and the authors have actively replied and addressed most of these problems. Two reviewers lean to accept and one reviewer leans to reject.

Some reviewers mentioned that it lacks insights and motivation to demonstrate the benefits of this kind of architectural design, improvements over baselines are not large enough, and also comparisons with some previous architecture-design work (e.g., Evolved Transformer) on machine translation are missing. Suggest the authors could add sufficient discussions to clarify the insights and motivation of the architecture design, and compare with more baselines if possible in the revised version. 
Decision: Accept (poster)