DeLighT: Deep and Light-weight Transformer
Authors: Sachin Mehta
Marjan Ghazvininejad
Srinivasan Iyer
Luke Zettlemoyer
Hannaneh Hajishirzi
Venue: ICLR 2021 Poster (2021)
Keywords: Transformers
Sequence Modeling
Machine Translation
Language Modeling
Representation learning
Efficient Networks

Abstract: We introduce a deep and light-weight transformer, DeLighT, that delivers similar or better performance than standard transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using the DeLighT transformation, a deep and light-weight transformation and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on benchmark machine translation and language modeling tasks show that DeLighT matches or improves the performance of baseline Transformers with 2 to 3 times fewer parameters on average. 

Published as a conference paper at ICLR 2021
DELIGHT: DEEP AND LIGHT- TRANSFORMER
Sachin Mehta', Marjan Ghazvininejad", Srinivasan Tyer? Luke Zettlemoyer and Hannanch Hajishirzi"
Universitt of Washington Facebooo Al Research Allen Institute for Al
ABSTRACT
We introduce deep and light- weight transformer, DeLighT, that delivers similar or better performance than standard transformer-bo based models with significantly fewer parameters. DeLigh! more efficiently allocates parameters both (1) within each Transformer block using the DeLi ghT transformation, a deep and light- weight transformation and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on benchmark machine translation and language modeling tasks show that DeLighT matches improves the performance of baseline Transformers with to times fewer parameters on average.
INTRODUCTION
Attention- -baied transformer networks (Vaswani et al. 2017) are widely used for sequence modeling tasks, including language modeling and machine translation. To improve performance, models often scaled to be either wider, by increasing the dimension of hidden layers, or deeper, by stacking more transformer blocks. For example, T5 (Raffel et al. 2019) uses a dimension of 65K and GPT-3 (Brown et al. 2020) uses 96 transformer blocks. However, such scaling increases the number of network parameters significantly (e.g., T5 and GPT-3 have 11 billion and 175 billion parameters. respectively), and complicates learning, i.e., these models either require very large training corpora (Raffel al. 2019 Devlin et al.) 2019 Brown et al.) 2020) or careful regularization (Hinton et al., 2012 Wan et al. 2013 Merity et al. 2a). In this paper, we introduce a new parameter-efficiet attention-based architecture that can be easily scaled to be both wide and deep.
Our Deep and Li ght-teight Transformer architecture, DeLighT, extends the transformer archi- tecture of Vaswani et al. (2017) and delivers similar or better performance with significantly fewer parameters and operations. At the heart of DeLight T is the DeLighT transformation that uses the group linear transformations (GLTs) of Mehta et al. (2018) with an expand-reduce strategy for varying the width and depth of the DeL.i T block efficienlly. Since GLTs are local by nature, the DeLi ghT transformation uses feature shuffling, which analogous to channel shuffling in convolutional networks (Zhang al. 2018), to share information between different groups. Such wide and deep representations facilitate replacing the multi-head attention and feed-forward layers in transformers with single headed attention and light-weight feed-forward layers, reducing total network parameters and operations. Importantly, unlike transformers. the DeLighT transforma- tion decouples the depth and width from the input size, allowing us allocate parameters more efficiently across blocks by using shallower and narrower DeLighT blocks near the input and deeper and wider DeLi T blocks near the output.
We demonstrate that DeLighT models achieve similar or better performance than transformer models with significantly fewer parameters and operations, on two common sequence modeling tasks, (i) machine translation and (ii) language modeling. On the low resource WMT'16 En-Ro machine translation dataset, DeLighT attains transformer performance using 2.8x x fewer parameters. On the high resource WMT 14 En-Fr dataset, DeLighT delivers better performance (+0.4 BLEU score) with 1.8x fewer parameters than baseline transformers. Similarly, on language modeling, DeLighT matches the performance of Transformer-XI. DDai et al. 2019)) with 1.5x fewer parameters

---

Published as a conference paper at ICLR 2021
on the WikiText- 103 dataset. Our source code is open-source and is available at: https//github.com sacmehta/delight
RELATED WORK
Several methods have been introduced to improve the transformer ar- chitecture. The first line of research addresses the challenge of computing self attention on long input sequences (Child et al., 2019; Kitaey et al., 2020; Beltagy et al. 2020). These methods can be combined with our architecture. The second line of research focuses on explaining multi-head attention (Raganato and Tiedemann 2018 Brunner et al.] 2020). They show that increasing the number of transformer heads can lead to redundant representations (Voita et al.] [2019]] Michel et al., 2019) and using fixed attention heads with predefined patterns (Raganato et al. (0020) or synthetic attention matrices (Tay et al., 2020) improves performance. The third line of research focuses on improving transformers by learning better representations (Wu et al. [019 2020; So et al., 2019). These works aim to improve the expressiveness of transformers using different transformations for example, using convolutions (Wu et al. 2019; Gehring et al.) 2017), gated linear units (Dauphin et al., 2017), or multi-branch feature extractors (So al.] 2019: Wu et al. 2020). Our work falls into this category. Unlike previous works, we show that it is possible to efficiently allocate parameters both at the block-level using the DeLighT transformation and across blocks using block-wiss -wiee scaling.
Improving transformers:
Model scaling: Model scaling is a standard method to improve the performance of sequence models
(Vaswani et al. 2017 Raftel et al. 2019;
Lan et al. 2020; Devlin et al., 2019] Shoeybi et al. 2019)
Tan and Le| 2019 Brown al.) 2020).
Model dimensions are increased in width- wise scaling
(Vaswani et al. 2017] Devlin et al.
(2119)
while more blocks (e.g.. Transformer blocks) are stacked
in depth-wise scaling (Shoeybi al.
2019 Brown et al.) 2020: Wang et al. 2019). In both cases (and
their combination), parameters inside each block of the network are the same, which may lead to a
sub-optimal solution. To further improve the performance of sequence models, this paper introduces block-wise scaling that allows for variably-sized blocks and efficient allocation of parameters in the network. Our results show that (1) shallower and narrower beLighT blocks near the input and deeper and wider D blocks near the output deliver the best performance, and (2) models with block wise scaling coupled with model scaling achieve better performance compared to model scaling alone. We note that convolutional neural networks (CNNs) also learn shallower and narrower representations near the input and deeper and wider representations near the output. Unlike CNNs (e.g.. ResNet of He et al. (016)) that perform fixed number of operations at each convolutional layer, the proposed block-wist wise scaling uses a variable number of operations in each layer and block.
Improving sequence models: There is also significant recent work other related methods for improving sequence models, including (1) improving accuracy using better token-level representations for example, using BPE (Sennrich et al. (2016), adaptive inputs (Baevski and Auli 2019) and outputs (Grave et al. 2017a), and DeFiNE (Mehta et al. 2020). and (2) improving efficiency - for example, using compression (Chen al.
2018: Sun al. 2020), pruning (Han et al. 2016 Voita al. 2019b (b),),, 2015 Sanh et al. 2019). The closest to our work is the DeFINE transformation, which also learns representations using an expand-reduce strategy. The key difference between the DeFINE transformation (Figure Icp and the DeLighT transformation (Figure Id) is that the DeL gh transformation more efficiently allocates parameters within expansion and reduction layers. Unlike DeFINE, which uses fewer groups in group linear transformations to learn wider representations, DeLighT transformation uses more groups to learn wider representations with fewer parameters. The DeLighT transformation achieves comparable performance to the DeFINE transformation but with significantly fewer parameters.
and distillation (Hinton et al.
DELIGHT: DEEP AND LIGHT-WEIGHT TRANSFORMER
A standard transformer block (Figure Lab comprises of multi-head attention that uses a query-key- value decomposition to model relationships between sequence tokens, and a feed forward network (FFN) to learn wider representations. Multi-head attention obtains query Q, key K, and value V by applying three projections to the input, each consisting of h linear layers (or heads) that map the d,, dimensiooni input into a d, dimensioonl space, where d, = d h is the head dimension. The FFN consists of two linear layers, where the first expands the dimensions from dm to dy and the
2

---

Published as a conference paper at ICLR 2021
Lightwiiight FPN Deiligh taroleenniio Singlobbee Altaaiin
des
dus Key
Japux dimessional)
No. of wapers (deya))
N (Fig
Expunioo
Malti lead Anntttoanncationn
Ancation
d
Alization
Query
Vilue
Rediction
ope:
Cuncas Ayy
O(dm 7k"
Alsenion
Ourpot (d, dimensional)
Ameeiion O(d,n²)
din
(c) DeFINE transformation
Food Forward Nework (FFN)
Add
de
Input (uuedimeniiona))
Redoction Expancion
No. of liyor (clapet (dkyel))
FFN params:
dy
8d
ITN params:
di
Depch
Ada
Depth
Output (l,ddmenionaa))
(a) Transformer block
(b) DeLi block
(d) DeLighT transformation
Figure 1: (a, b) Block-wise comparison between the standard transformer block of Vaswani al. (2017 and the Deliigh ghT block. In the DeL.i gh transformation, the number of operations in computing attention are reduced by half while the number of parameters (and operations) in the FFN are reduced by 16x. Transformations with learnable parameters (Linear and DeLighT) are shown color. The shape of linear transformations indicate their operation (expansion, reduction, etc.). (c, d) compares the DeFINE transformation (Mehta et al. (2020) with the DeLight DiTTghT transformation. Compared to the DeFINE transformation, the DeLighT transformation uses group linear transformations (GLTs) with more groups to learn wider representations with fewer parameters. Different colors are used to show groups in GLTss. For simplicity, feature shuffling not shown in (d).
second reduces the dimensions from d to dm- The depth of a transformer block is 4, consisting of (1) three parallel branches for queries, keys, and values, (2) a fusion layer that combines the output of multiple heads, and (3) two sequential linear layers in the FFN. In general, transformer- networks sequentially stacks transformer blocks to increase network capacity and depth.
This paper extends the transformer architecture and introduces deep and fight-weight transformer, DeLighT. Our model uses a deep and light-weight expand-reduce transformation, DeLighT trans- formation (Section 3.I). that enables learning wider representations efficiently. It also enables replacing multi-heas attention and feed forward network (FFN) layers with single-heat attention and light-weight FFN (Section 3.2). DeLi ghT transformation decouples attention dimensions from the depth and width, allowing us to learn representations efficiently using block-wise scaling instead of uniform stacking of transformer blocks (Section 3.3
3.1 DELIGHT TRANSFORMATION
DeLi ighT transformation maps dm dimensional input vector into high dimensional space (ex- pansion) and then reduces it down to a d,, dimensional output vector (reduction) using N layers of the group transformations of Mehta al. (2018), as shown in Figure ld During these expansion and reduction phases, DeLighT transformation uses group linear transtormations (GLTs) because they learn local representations by deriving the output from a specific part of the input and are more efficient than linear transformations. To learn global representations, the DeLighT transforma- tion shares information between different groups in the group linear transformation using feature shuffling, analogous to channel shuffling in convolutional networks (Zhang et al. 2018).
standard approach to increase the expressivity and capacity of transformers is to increase the input dimensions, dm. However, increasing dm linearly also increases the number of operations in multi- head attention (nn"dd,) where n is the sequence length) in a standard transformer block (Figure Tah. In contrast, increase the expressivity and capacity of the DeLighT block, we increase the depth and width of its intermediate DeLighT transformations using expansion and reduction phases. This enables to smaller dimensions for computing attention, requiring fewer operations.
Formally, the DeLighT transformation is controlled by five configuration parameters: (1) number of GLT layers N, (2) width multiplier Wi. (3) input dimension d (4) output dimension d., and
3

---

Published as a conference paper at ICLR 2021
Tibar
GLT [Groupa
JOross 4
Figure 2: Example illustrating the expansion phase in the DeLight transformation that uses GLTs, feature shuffting, and input mixer connection, learn deeper and wider representations efficiently. For illustrative purposes, we have used the same input and output dimensions.
(5) maximum groups Imer in a GLT. In the expansion phase, the DeLighT transformation projects the d,m -inddmensiooal input to a high-dimensional space, dmaz Wmdmm linearly using [ layers. In the reduction phase, the DeLighT transformation projects the mine--dimensions vector GLT layers. Mathematically, define the
do fo-dimensional space using the remaining N output Y at each GLT layer as:
F F W b' b',g))
1=1
F H (X, Y Y
W b' b',g", Otherwise
(1)
bbb o are the learnable weights and biases of group linear transformation with g' groups at the /-th layer. Briefly, the function takes the input X (or H (X, Y!-1)) and splits into g' non-overlapping groups such that X = {X1,... .x' }- The function Ff then linearly transforms each X; with weights W and bias by produce output Y! = : b/- The outputs of each group Y! are then concatenated to produce the output Y The function H first shuffles the output of each group in Y and then combines it with the input X using the input mixer connection of Mehta et al. (2020) to avoid vanishing gradient problems. Figure 2 visualizes the expansion phase in the DeLigh wig transformation with group linear transformation. feature shuffling, and the input mixer connection.
where W W W } and b' = (b.....,.u.
The number of groups at the 1-th GLT in DeLighT transformation computed as:
g' d'=
min Jmaz))
1 1 [N/2]
n
Otherwise
In experiments, Omar
d that each group has least 32 input elements.
3.2 DELIGHT BLOCK
Figure [b] shows how we integrate DeLighT transformation into the transformer block to improve its efficiency. The dm -idmensiioa inputs are first fed to the DeLighT transformation produce d, do-dimensional outputs, where do < dm- These d -dimmnsioonll outputs are then fed into a single head attention, followed by a light-weight FFN to model their relationships.
DeLighT layer and single head attention: each of dimensionality dm. These n, dn dimensioonl inputs are first fed the DeLight ghT transforma- tion to produce dimeenssio outputs, where d, < dm These n, d, dimenssioa outputs are then projected simultaneously using three linear layers to produce d, f.-dimensional queries Q, keys K, and values V. We then model contextual relationships between these tokens using scaled dot-product attention (Eq. 3). To enable the use of residual connections (He et al., (016)) the d, dy-dimensional outputs of this attention operation are linearly projected into dm -dimensional space.
Let us assume we have sequence of n input tokens,
QK?
Attention K, Q, V) = softmax
VTT
We hypothesize that the ability of DeLighT to learn wider representations allows us to replace multi-head attention with single-head attention. The computational costs for computing attention in
4

---

Published as a conference paper at ICLR 2021
360-
Uniform Iopux
Block-wise Itpux
No. of paraneters
791
Uni form
No. of operations
(in niltion!)
150
Uni form
Block-wtse
140
=V
Muni
230
B blocks
892
:28-
(see Eqg
593
401
N
N8-1
Nouz
689
80
81
82
83
84
85
86
87
20 B6
Ouipur
Ouiput
Decoder blocks
Decoder
blocks
(a) Uniform vs. block-wise
(b) Distribution of parameters and operations within each block
Figure 3: Block-wise scaling efficiently allocates parameters and operations across blocks, leading to shallower and narrower DeLighT blocks near the input and deeper and wider DeLighI blocks near the output. In (b), DeLighT networks with both uniform
=8) and block-wise (Nmin. =4, Nmaz =8) scaling have about 16.7 M parameters and perform 3.5 B operations (computed for sequence length of n 30), however, the De'ligh network with block-wise scaling delivered 2 points better perplexity.
the standard transformer and the DeLight ghT block are (dmn"² and Odgn²) respectively, where d, < dm Therefore, the DeLighT block reduces the cost for computing attention by factor of dm/d, /d.- our experiments, we used d, dm 2,, thus requiring 2x fewer multiplication-additoo operations as compared to the transformer architecture.
Leght-weight FFN: Similar to FFNs in transformers, this block also consists of two linear layers. Since the DeLighT block has already incorporated wider representations using the DeLig ghT transformation, allows us to invert the functionality of FFN layers in the transformer. The first layer reduces the dimensionality of the input from dm to /r./r while the second layer expands the dimensionality from dm/r to dmm. where is the reduction factor (see Figure lb Our weghtt FFN reduces the number of parameters and operations in the FFN by a factor of rd//dm. /dn- In the standard transformer, the FFN dimensions are expanded by factor of 4 4 In our experiments, used r 4. Thus, the light- FFN reduces the number of parameters in the FFN by 16x.
The DeLighT block stacks (1) DeLighT transformation with N GLTs, (2) three parallel linear layers for key, query, and value, (3) a projection layer, and (4) two linear layers of a light-weight FFN. Thus, the depth of DeLighT block is N + 4. Compared to the standard transformer block (depth is 4), DeLighT block is deeper.
Block depth:
3.3 BLOCK- -ILOCK-WISE SCALING
Standard methods for improving the performance of sequence models include increasing the model dimensions (width scaling), stacking more blocks (depth scaling), or both. However, such scaling is not very effective on small datasets. For example, when Transformer- Base (d,, = 512) network is replaced with Transformer-Large (dm = 1024) the WMT' En-Ro corpus, the number of parameters increases by approximately 4x while the performance does not change appreciably (BLEU: 34.28 vs. 34.35). We hypothesize that this happens because scaling model width and depth allocates parameters uniformly across blocks, which may lead to learning redundant parameters. To create deep and wide networks, extend model scaling the block level (see Figure 3)
The DeLi ghT block learns deep and wide representations using the DeLighT transformation, whose depth and width are controlled by configuration parameters: the number of GLT layers N and the width multiplier Wm. respectively (Figure 3a). These configuration parameters allow independently of the input dm and output d, dimensions. Such calibration is not possible with the standard transformer block because their expressiveness and capacity are a function of the input (input dimension = number of heads x head dimension). Here, we introduce block-wise scaling that creates a network with variably-sized DeL. g blocks, allocating shallower and narrower DeLi DhLighT blocks near the input and deeper and wider heLighT blocks near the output.
Scaling the DeLighT block:
increase the number of learnable parameters inside the DeLighT block
To do so, we introduce two network wide configuration parameters: minimum Nmin and maximum N number of GLTs in a DeLi LiT transformation. For the b-th DeLighT block, we compute
Transformer- base uses d,,, 551 and 2242048 while Transformer-large uses d,, =1024 and 40=40
5

---

Published as a conference paper at ICLR 2021
the number of GLTs No and the width multiplier w in DeLighT transformation using linear scaling (Eq. 4). With this scaling, each DeLighT block has a different depth and width (Figure Bay.
(Nmax Nmsa)
(Nmax _mmmn
No Nuain
w = wm
0 5 B 1 (4)
1-1
Nmi ( -)
Here, B denotes the number of DeLighT blocks in the network. We add superscript b to number of GLT layers N and width multiplier Wm to indicate that these parameters are for the b-th block.
Network depth: The depth of transformer block is fixed, i.e., 4. Therefore, previous works (Raffel al.] 2019 Brown al., 2020; Wang et al. 2019) have associated the depth of transformer-batt based networks with the number of transtormer blocks. In DeLighT, we present different perspective to learn deeper representations, wherein each block variably-sized. To compute the network depth, we use the standard definition across different domains, including computer vision (e.g., ResNet Hf (et al. 2016) and theoretical machine learning (Telgarsky) 2016). These works measures network depth as the number of sequential learnable layers (e.g., convolution, linear, or group linear). Similarly, the depth of D i ghT and transformer networks with B blocks is 26- (No + 4) and 4B, respectively.
4 EXPERIMENTAL RESULTS
We evaluate the performance of DeLi ghT on two standard sequence modeling tasks: (1) machine translation (Section 4.1 and (2) language modeling (Section 4.2)
4.1 MACHINE TRANSLATION
Datasets and evaluation: We benchmark DeLighT models on four datasets: (1) IWSLT' 14 Ferman-Englise (De-En), (2) WMT 16 English-Romaniae (En-Ro), (3) WMT 14 English-Germat (WMT' 14 En-De), and (4) WMT' 14 English-Frende (WMT' 14 En-Fr). For the IWSLT' 14 De-Ein En dataset, we replicate the setup of Wu et al. (2019) and Edunov et al. (2018), which uses 160K/7K/KK sentence pairs for training. validation, and testing with a joint BPE vocabulary of about 10K tokens, respectively. For the WMT' 14 English-German (En-De) dataset, we follow the setup of Vaswani et al. (2017). The dataset has 3.9M/39K/3K sentence pairs for training, validation, and testing respectively with a joint BPE vocabulary size of 44K 2 For the WMT 14 English- French (En-Fr) dataset, we replicate the setup of Gehring et al. (2017). which uses 36M/27K/3K sentence pairs for training. validation, and testing respectively with joint BPE vocabulary size of 44K. The performance is evaluated in terms of BLEU (Papineni et al.) 2002) (higher is better) on the test set. We follow Wu et al. (2019) for beam search related hyper-parametert
Architecture: We follow the symmetric encoder-decoder architecture Vaswani et al. (2017) with sinusoidal positional encodings. Both the encoder and the decoder have B DeLighh blocks. Decoder blocks are identical to the encoder blocks (Figure lb), except that they have an additional source-target single-head attention unit before the light- FFN. In the cource-target single-heat head attention unit, keys and values are projections over the encoder output (full details in Appendix [A). In our experiments, we use Wm. 2, Nmmn 4, and Nmaz 8 for WMT 16 En-Ro, 14 En-De, and WMT' 14 En-Fr; resulting in 222 layer deep De i ghT networks. For IWSLT'14 14 De-En, we used Wm = Nynin = 3, and Nmaz for IWSLT' 14 De-En; resulting in 289 layer deep network. For simplicity, we set B = NMaz. We use a learnable look-up table that maps every token in the vocabulary to 18-dimensional vector. We implement our models using Fairseq (Ott et al. 2019) and use their provided scripts for data pre-processing, training, and evaluation.
Training: For IWSLT' 14 De-En models, we follow the setup of Wu et al. (2019) and train all our models for 50K iterations with batch size of 4K tokens on a single NVIDIA GTX 1080 GPU. For WMT' 16 En-Ro, we follow the training setup of Ghazviinnneja et al. (2019) and train models for 100K iterations on 16 NVIDIA Tesla V V100 GPUs with an effective batch size of 64K tokens. For WMT' 14 En-De and WMT' 14 En-Fr, we follow the training set-up of Wu et al. (2019) and train our models on 16 V100 GPUs for 30K and 50K iterations, respectively. We use Adam (Kingma and Ba 2015) to minimize cross entropy loss with a label smoothing value of 0.1 during training. For a fair comparison, we trained baseline transformer models using the same training set-up.
We use training and validation data that is compatible with the Tensor2Tensor library Vaswani et al. in order have fair comparisons with recent works (e.g., Evolved Transformer).
2018

---

Published as a conference paper at ICLR 2021
IWSET De-En
I6 En-Ro
Model
Params
Ratio
BLEU
BLEU
Purums
Rutio
BLEU
A BLEU
Transforer Vaswaani et 2017
34.4
62 M
34.3F
Transfomer (Our impl.)
42.
1.0x
34.3
62
1.0x
34.3
DeLight
14 1
0.3%
33.8
-0.5
22
0.35
34,3
0.0
DeLight
30
0.7%
35.3
+1.0
53 M
0.85x
34.7
+0.4
(a) Results small corpora
WMT 14 En-De
14 En-Fr
Model
# Params
Ratio
BLEU
A BLEU
# Params
Ratio
BLEU
A BLEU
Transformer (Naswani et al. 2017
62 M
27.3
62.M
38.1
Transformer (Our impl.)
10x
1.0%
39.2
DeLigh7
M
0.55x
27.6
-0.1
M
0.55K
39.6
+0.4
DeLigh7
54 M
0.80%
28.0
+0.3
54
0.80x
40.5
+1.3
(b) Results on large corpora
Table 1: Comparison with baseline transformers on machine translation corpora. DeLighT models require significantly fewer parameters achieve similar performance. Here, and indicate the best reported transformer basclines from Wu ct al. 2019) and Ghazvininejad al. (2019) respectively.
Depth
Params
# MACs
BLEU
(MeT 14 En-De)
Transformer
67 M
11.1 B
39.2
1.8x parsmeters
Deliigh
222
37 M
5.6 B
39.6
DeLighT
222
M
8.1 40.5
Table DeLighT networks are deep, light- weight and efficient as compared to transformers. BLEU score is reported on the WMT' 14 En-Fr dataset. To compute network depth. number of sequential layers in the network (Sec- (.3)..3). We used 20 source and 20 target tokens for computing multiplication-addee addittionn operations (MACs). See Appendex for details.
BLEU
22
++.44 BLEU
BelighT Trans
Evolved Trams
26
46 66
Paraneters (in nillion)
count the
Figure 4: Comparison of DeLighT with Transformers and Evolved Transformers at two different settings, on the WMT' 14 En- De-De corpus: (1) the number of parame- ters is the same and (2) the performance is the same.
4.1.1 RESULTS
Comparison with baseline transformers: Table compares the performance of DeLighT with the baseline transformers of Vaswani et al. (2017) different corpora. DeLighT delivers better performance with fewer parameters than transformers. across different corpora. Specifically, on low-resource (WMT' 16 En-Ro) and high resource (WMT" 14 En-De & WMT 14 En-Fr) corpora, DeLighT delivers similar or better performance with 2.8x and 1.8x fewer parameters, respectively. increased, DeLighT outperforms transformers. For example, on WMT' 14 En-Fr dataset, DeLighT is 3.7x deeper than transformers and improves its BLEU score by 1.3 points yet with 13 million fewer parameters and billion fewer operations (see Table
When the number of parameters
Particularly interesting are the performance comparisons of DeLi ihT with the baseline transformers of Vaswani al. (2017) and its neural search variant, i.c., Evolved Transformer of of al. (2019) at two different parametric settings on WMT' 14 En-De corpora in Figure 4 For small models (< 10 M parameters), De ghT models delivers better performance and for attaining the same performance as these models, DeLighT models requires fewer parameters.
Comparison with state-of-the-art methods: Most ofate-of-the-ar methods have evaluated the performance on WMT' 14 En-De while some have also evaluated IWSLTT' 14 De-En. Table 3 compares the performance of DeLi ghT with state-of-the-ar methods on these two corpora. DeLighT delivers similar or better performance than existing methods. It is important to note that existing methods have improved baseline transformers with different design choices - for example, the asymmetric encoder-decoder structure (Wang al. 2019) and neural architecture search (So) et al. 2019). We believe that DeLighT, in the future, would also benefit from such design choices.
Scaling up DeLighT models: Figure SggureSssoow shows the performance of DeLighT models improves with increase network parameters; suggesting their ability to learn representations across different corpora, including low-resource.
7

---

Published as a conference paper at ICLR 2021
Model
Params
BLEU
Model
Params
BLEU
Transformers Vaswani et al.
2017
42 M
34.3
Transformnr (Nrwami et 2017
M
27.3
Variational (Zee
(Wangel
Dynamic comvolution: Viewan aa 2017
43
35.2
Evolved Transtormer St 2019
46
27.7
Lite Transfommer' [W mr. 20A1
23.6
Like Transfommrr
Wo ol 2020
26.5
DeLight (Ours)
30
35.3
BellighT (Ours)
27.6
(a) IWSIT De-Fin
(b) WMT En-De
Table 3: Comparison with -fate-of-the-art methods on machine translation corpora. Delli gh'l delivers similar or better performance than state-of-the-art models with fewer parameters. Here, indicates that the network uses neural architecture search (NAS) and indicates that full network parameters are not reported.
36.
34.3
27.2
39.
33.6
33.6
25.0
30.0
32.4
32.9
31.2
32.2
23.3
25.9
28
31.58
10 20
66
22.0
40
38.5g
50
Paraneters (in nillion)
Parazaters (in million)
Parameter
(in million)
Parameters
(in million)
(a) IWSLT' 14 De-En
(h) WMT 16 En-Ro
(c) WMT' 14 En-De
(d) WMT" 14 En-Fr
Figure 5: Scaling up DeLighT models. The performance of DeLighT improves with increase in the number network parameters, across different corpora, including low low-resource (WMT En-Ro).
4.2 LANGUAGE MODELING
Datasets and evaluation: We evaluate on the WikiText-103 dataset (Merity al. 2017) that has 103M/217K/245K tokens for training, validation, and testing. It has a word-level vocabulary of about 260K tokens. Following recent works (Baevski and Auli 2019 Dai al. 2019). we report performance in terms of perplexity (lower is better) on the test set.
Architecture: We use the transformer-based decoder architecture of Baevski and Auli (2019) with B DeLighT blocks. We use =2= Mmni==4, and N =12 We scale d,, using values (384, 512, 784, 1024] for increasing network parameters. For simplicity, we B Nmax. Follow- ing standard practice, we use adaptive input (Baevski and Auli] [2019] look-up table and adaptive output (Grave et al., 7017a) as the classification layer with one head (head dimension is 128) and two tails (tail dimensions are 64 and 32). We also share weights between the input and the output layers.
Training: We follow the training setup of Baevski and Auli 2019), except that we train our models on NVIDIA Tesla V100 GPUs for 100K iterations with a context length of 512 and an effective batch size of 64K tokens. We use Adam during training and use a context length of 480 during test.
Results: Table 4b compares the performance of DeLighT with previous methods on WikiText- 103. Table 4a plots the variation of perplexity with number of parameters for DeLighT and Transformer- XL (Di ct al. 2019) 2019) - which outperforms other transformer- based implementations (c.g.. (c.g..(Baevss) Baaesski and 2ul12019). Both tables show that DeLighT delivers better performance than state-of-the-art methods (including Transformer-X XL. and does this using a smaller context length and significantly fewer parameters, suggesting that the DeLighT transformation helps learn strong contextual relationships.
5 ANALYSIS AND DISCUSSIONS ON COMPUTATIONAL EFFICIENCY
Training time and memory consumption: Table compares the training time and memory con- sumption of DeLight with baseline transformers. For an apples-to-apples comparisons, we imple- mented the Transformer unit without NVIDIA's dedicated CUDA kernel, and trained both transformer and DeLi DeLighT full-precision networks for 30K iterations on NVIDIA V100 GPUs. The trans- former and DeLighT models took about 37 and 23 hours for training and consumed about 12.5 GB and 14.5 GB of GPU memory, respectively (R1 vs. R2). When we enabled the dedicated CUDA kernel provided by APEX library for multi-head attention in Transformers, the training time of the
hhan:://gthuuhcom/////DAAA
8

---

Published as a conference paper at ICLR 2021
32-
DuLighT [ours]
Transformer-A
Method
Network
Depth
Contest Length
Params (In million)
Perplesity (Test)
30-
28
LSTM Noural Cachs (Grae ei sl. 20176
QRNN (Merity et al 20150
151 M
33.00
40.80
26
Tranfomer-Xi X1. Dai
2019
640
151 M
24.03
Transfommer-ii. (hhr might
6440
151
24.34
Transfomer-Xi. (Our impl.)
64
480
151 M
24.91
Paraneters (in million)
Belighh (Oun))
(a) Del.i ghT vs. Transformer-Xi
(b) Comparison with existing methods
Table 4: Results on the WikiText-106 103 dataset. Compared to Transformer-XL, DeLight delivers similar or better performance (lower perplexity) with fewer parameters. For Transformer-Xe XL, reproduce results using official source code. evaluating Transformer-o XL with context length of 480, we set the mem hyper-paramen to 480 the official evaluation scripts.
Training time
Memory (in GB)
Row
Model
(in million)
(WMT'14 En-Frr
Transformer (unoptimizsd)
67
hours
12.5
R2
DeLighTT (unoptimized)
54
40.5
23 hours
14.5 GB
Transformer (w/ Apex optimized)
67
39.2
16 hours
11.9 GB
DeLighT (w/ optimized grouping)
40.5
hours
11.5
Table 5: Comparison with baseline transformers in terms of training speed and memory consumption. In R4, we implemented CUDA kernels grouping and ungrouping functions only (see Appendix D.. We expect DeLighT to be more efficient with a single and dedicated CUDA kemee for grouping, transformation, feature shuffling, and ungrouping. Memory consumption is measured on a single NVIDIA GP100 GPU (16 GB memory) with maximum of 4096 tokens batch and without any gradient accumulation.
Model
Dropout
BLEU
Transformer (62 M)
0.10
27.3
Transformer (62 M)
0.30
27.7
DeLighT M)
0.05
Table 6: DeLighT requires less regularization as compared to baseline transformers (Dataset: WMT" En-De).
transformer model reduced from 37 16 hours while did observe any significant change in memory consumption. Motivated by this observation, we implemented dedicated CUDA kernels for grouping and ungrouping functions in GLTs (see Appendix E With these changes, training time and GPU memory consumption of DeLighT reduced by about 4 hours and 3 GB, respectively. We em- phasize that grouping, linear transformation, feature shuffling, and ungrouping, can be implemented efficiently using single CUDA kernel. In future, expect dedicated CUDA kernel for these operations would further reduce the memory consumption as well as training/inference time.
Regularization: T shows that DeLi DeT.ighT delivers similar performance to baseline transformers, but with fewer parameters and less regularization. This suggests that learning representations with better transformation functions alleviates the need for dropout.
6 CONCLUSION
This paper introduces a deep and light-weight transformer architecture, DeLighT, that efficiently allocates parameters both within the DeLighT block and across DeLighT blocks. Compared to -fate-of-the-art transformer models, DeLighT models are (1) deep and light- and (2) deliver similar or better performance. In the future, plan to apply DeLighT to other tasks, including language model pre-training, question answering, and language generation.
Acknowledgements: This research was supported by ONR N00014-18- 18-1-22--- DARPA N66001- 19-2-403, NSF (ISS-161612) 12, (IS1552855) and Allen Distinguished Investigator Award. Authors would also like to thank members of the UW-NLP and the H2Lab at The University of Washington for their valuable feedback and comments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint N.Kv::1900.10833 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint ArXiv:2005. 14165, 2020.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Pupers), 2019.
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXivv 2012.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference machine learning, pages 1058-1066, 2013.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018a. URL https://openneeiee..ee// forum Syd=SyyPPOTTZ
Sachin Mehta, Rik Kedzio-Kedziorski, Mohammad Rastegari, and Hannaneh Hajishirzi. Pyramidal recurrent unit for language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Pruceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer xl: Attentive language models heyond fixed-length context. In Association for Computational Linguistics, 2019.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever: Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya, Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
Iz. Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: arXiv:2004.05150 2020.
The long-document transformer.
Alessandro Raganato and Jörg Tiedemann. An analysis of encoder representations in transformer- machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, November 2018.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers. In International Conference on Learning Representations, 2020. URL
http://oppeneeveee.eetffrrumu :idBJ 166EFDB
Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up cuttom-pppoo evolution of representations in the transformer: study with machine translation and language modeling objectives. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, 2019a.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really hetter than one? In Advances in Neural Information Processing Systems, pages 14014-14024, 2019.
10

---

Published as a conference paper at ICLR 2021
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-atiention patterns in transformer- based machine translation. arXiv preprint arXiv:2002.10266 2020.
Yi Tay. Dara Bahri, Donald Metzler. Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743, 2020.
Felix Wu, Angela Fan, Alexci Bacvski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representatinss 2019.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-shon range attention. In International Conference on Learning Representations, 2020.
David So, Quoc Le. and Chen Liang. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, pages 5877-5886, 2019.
Jonas Gchring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequenee learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1243-1252. JMLR. org, 2017.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning- Hoarminn-Voluuee Volume 70, pages 933-941. JMLR. org, 2017.
Zhenzhong Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, Alhert: A lite bert for sule-supe learning of language representations. In International Conference on Learning Representations, 2020.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Ing Training molti-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.0880 080553 2019.
Mingxing Tan and Quoe V. Le. Efficientaet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), August 2016.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. International Conference on Learning Representations, 2019.
Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. Efficient softmax approximation for GPUs. In International Conference on Machine Learning 2017a.
Sachin Mehta, Rik Koncel-Kedziorske Mohammad Rastegari, and Hannaneh Hajishirzi. DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling. In International Conference on Learning Representations, 2020.
Patrick Chen, Si Si. Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low-rank ap- proximation for neural language model shrinking. In Advances in Neural Information Processing Systems. 2018.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou, Mobilebert: a compact task-agnostic bert for resource-limite limited devices. In Association for Computational Linguistics (ACL), 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference for Representation Learning, 2016.
Elena Voita. David Talbot, Fedor Moiseev, Rico Sennrich. and Ivan Titov. Analyzing multi head aelf-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 201 9b.
11

---

Published as a conference paper at ICLR 2021
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Victor Sanh, Lysandre Debut, Julien Chaumond. and Thomas Wolf. Distilbert, distilled version of bert: smaller. faster, cheaper and lighter. 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing NewrlPS, 2019.
Matus Telgarsky. Benefits of depth in neural networks. COLT, 2016.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Classical structured prediction losses for sequence sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner. Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/ 0111180007166 2018. URL http:///////axi.oorraa///// (883
07416
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguisties, 2002.
pages 1-318.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: fast, extensible toolkit for soquenee modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeutlemoyrr Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 61233123, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. Latent alignment and variational attention. In Advances in Neural Information Processing Systems, pages 9712-9724, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous
cache. In International Conference Learning Reprexentations, 201
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv: 180.:::88000044444 2018b.
12

---

and Published as a conference paper at ICLR 2021
A DELIGHT ARCHITECTURES FOR LANGUAGE MODELING AND MACHINE TRANSLATION
DeLighT architectures for language modeling and machine translation are shown Figure For language modeling, we follow the architecture in Baevski and Auli (019) while for machine translation, we follow the architecture in Vaswani et al. (2017).
Language modeling; Figure 6a shows the architecture for language modeling. The architecture stacks B DeLighT blocks, the configuration of each block is determined using block wise scaling. Each block has three sub-layers. The first layer a DeLight transformation that learns representations in high-dimensional space. The second layer a single-head attention that encodes contextual relationships. The third layer is a position-wise light- feed-forward network. Similar to Vaswani et al. (2017), we employ residual connections (He et al. 2016). Similar to previous works (Baevski and Auli 2019) Dai et al. 2019), we tied adaptive input (Baevski and Auli 2019) and adaptive softmax (Grave al. 201 map tokens
vectors tokens, respectively.
Machine translation: Figure 6b shows the architecture for machine translation. The encoder stacks DeLighT blocks, the configuration of each block is determined using block-wise scaling. Similar to lan- guage modeling, each encoder block has three sub-layers. The first layer is DeLighT transformation that learns representations high-dimensional space. The second layer is single-head attention that encodes textual relationships. The third layer position-wis wise light-weight feed norwafo network. Similar to Vaswani et al. (2117). we employ a residual connections (He et al. 2016). We use learnable look-up table to map tokens
lagits
Logits
Lincar
Adaptive Softmax
Add Nort
Add & Norm
Lightowniht FFN
Inpur and outpur weights are tied
Light weight FIN
Add
Imput and output weights Ded
Read Alicztius
Add Norm
B
FFN
Masked Single-
BX
Add & Norn
Add & Norm
head Attention
Amsetion
Positional Eacoding
Poctiinaa Emsiing
Posisoaal Encading
Embedding
Layer
Embedding
Embedding
Adaptive Inputs
Layer
Look-up
Layer
Ipputs (hiited right)
Ingus
Oupputs (shiiled right
Language Modeling
(b) Machine translation
Figure 6: Sequence modeling DeLighT. Here, green color hexagon represents DeLi ghT transfor-
mation.
13

---

Published as a conference paper at ICLR 2021
vectors. Similar the encoder, the decoder also stacks B blocks. Decoder blocks identical encoder blocks, except that they have additional source-target single-head attention unit before the light-weight FFN. Keys and values in source-tane couree-target single-head attention unit are projections over the encoder output. We use standard learnable look-up table to map tokens to vectors and lincar classification layer map vectors to tokens.
B GROUP LINEAR TRANSFORMATION WITH INPUT-MIXER CONNECTION
Group linear transformation (GLT) splits a dimeession input into non-overlapping groups such that X = Concat (X1, g)), where X; is du -imennsiona vector. X, 's are then simultancously transformed
using linear transforms W; E R dm dh produce the final d.-dimensional output Y Y y))
produce outputs Y; W.W. Y:'s are then concatenated to
Figure 7a shows example GLT in the expansion phase DeLighT transformation. For illustrative purposes, we have used the same dimensions in this example. Recall that as we go deeper in the expansion phase, the number of groups increases. In this examplc, the first layer has one group, the second layer has two groups and the third layer has four groups. GLTs learns group-specific representations and local. To allow GLT learn global representations, Figure 7b] Furthermore, training deep neural networks by merely stacking linear or group linear (with or without feature shuffle) is challenging because of vanishing gradient problem. Residual connections introduced by He al. (2016) mitigates this problem and helps train deep neural networks. However, such connections cannot be employed when imput and output dimensions
feature shuflle. An example of GLT with feature shuffle shown in
not the same (c.g.. during the expansion and reduction phases DeLighT transformation). To stabilize the training learn deeper representations, we use imput- mixer connection Mehta et al. (2020). Figure 7c shows an example GLT with feature shuffle and input mixer connection.
Groups = 1
Grodke
Inpittrian
Groups =2
Groups
Shiffa
Shuffle
Groups 4
Groups
Croaze
(c) GLT with feature shuffle and input mixture connection
GLT
(b) GLT with feature shuffle
Figure 7: This figure visualizes different variants of group linear transformations that are used in the DeLighT transformation.
C MULTIPLICATION
OPERATIONS IN DELIGHT
The Del.i ThT block is built using linear transformations, GLTTs, and scaled dot-product attention. Total number of mulliplication-addiiiooo operations (MACs) in network is accumulation of these individual operations.
n denotes the number of source tokens, denotes number of target tokens, dm denotes input dimension, d, denotes the output dimension, and g denotes the number of groups in GLT. The procedure for counting MACs for each of these operations is described below.
Group linear transformation (GLT): GLT F has g learnable matrices W, R
7. There-
fore, GLT learns
indda
parameters and performs ands
MACs to transform dm dimensional input to
14

---

Published as a conference paper at ICLR 2021
de-dimensional ouiput. Following standard practice, e.g., ResNet of He He al. (2016). addition and multiplication one operation instead of two because these operations can fused recent hardwares.
Importantly, when g = I, the GLT is the same as linear transformation.
Self-attention in DeLighT: The scaled dot-produce self-attention in DeLighT is defined as:
QK
Attention K, Q. V) = softmax
(S)
VaL
where Q &R K V Bn denotes query, key, and value, respectively.
The attention operation involves two dot-products. The first dot product between Q and K while the second dot product is between the ouiput of first dot product and V. Both dot products require d,n MACs. Therefore, total number of MACs computing scaled dot-product self-ateeniio are 2don
In case of source-taie target attention (as in machine translation), K's and V's 's are from the souree (eneoder) and Q's are incrementally decoded (one token at time). Therefore, the number MACs required decode m
target tokens given source tokens are 2kndo
D ABLATIONS ON THE WIKITEXT- 103 DATASET
stuuiis the impact of DeLig g block parameters on the WikiText 103 dataset, namely (1) minimum number of GLT's Nmin. maximum number GLTs Naaa. (3) width multiplier Wm. and (4) model dimension dm (see Figure Tbp. Figure Figure and Figure TogureT[sshows the impact the Del. transformation, feature shuffling, and the light-weight FFN. Table[sshows Del.igh" block while Figure 12 shows the effect of scaling De i gh networks. We choose the WikiText- 103 dataset for ablations because it has very large vocabulary compared to other datasets (267K vs. 30-40K), allowing to test the ability under large vocabulary sizes. The performance is reported in terms of perplexity (lower better) on validation set. In our ablation studies, we used the same settings for training as in Section 4.2 except that we train only for 50K iterations.
Del.i effect of position DeliighT transformation the
Ligi block: Overall, Table 7 shows that scaling depth and width using DeLi DeL transformation and block-wise scaling improves performance. We make following observations:
a) Block-wise scaling (R4, R5) delivers better performance compared to uniform scaling (RI-R3). For instance, DeLigh ghT with Nmi = 4 and Nmax = 8 (R4) is 1.25 shallower than De bili with Nman = and Nmaz = 8 (R2), but delivers better performance with similar number of parameters and operations. Scaling tmm improves performance (R2 vs. R3), however, the improvement is significantly lower than for the model with block-wise scaling (R3 vs. RS). This suggests that non-uniform distribution parameters across blocks allows the network to learn better representations.
b) Different ratios between Near and Nuci yields different results. We observe significant performance improvements when the ratio is greater than or equal to two. For example, when we scale Nmaz from 2 to 3 (R6 vs. R8), the perplexity improves by ~55 points with only moderate increase in network parameters. On the other hand, when the Navus. is close to (R6 vs. R7), performance does not change appreciably. This
Nmin likely because the allocation parameters across blocks is close to uniform (Eq. 4.. This consistent with our previous observation.
Learning shallower and narrower representations near the input and deeper and wider representations near the output achieves better performance. For example, when we scaled Nmax from 8 to 12 for Nmon = = (R6, R8), T delivered better performance with similar number of parameters compared model with Nn.. 6 (R7, R9). This likely because ratio of Nmaz and Nmán is higher when Nman which helps allocate parameters per block more effectively.
d) Deeper and wider representations near the input and shallower and narrower representations near the output hurts performance (313 vs. R16).
e) Scaling width using wm. and dm improves performance (R10-R15), however, their impact is different. For example, when we scale wm. and dm by two, the rate of increase in number of parameters and operations is more rapid with dm compared Wm. DeLighT's ability learn wider representations in different ways may be useful in selecting application specific models.
Impact of DeLighT transformation: We replace DeLighT transformation in the DeLighT block (Figure
1b with (1) the DeFINE transformation and (2) a stack of linear layers. Figure shows that Del.i ghT transformation delivers similar performance with significantly fewer parameters compared to the DeFINE unit
15

---

Published as a conference paper at ICLR 2021
Row
N Nonin
N Nmaz
Wm
dm
Depth
Parameters
MACs
Perplexity
Uniform vs.
block-wise scaling
RI
4
4
256
43
14.1 M
2.96 B
56.19
R2
8
8
256
115
16.6 M
3.49 B
48.58
R3
8
8
256
115
22.1 M
4.64 B
45.10
92
16.7
3.51
46,30
R5
4
2
256
158
21.0
41.18
Varying depth (Nmin and Noar (Eq. 4
R6
4
8
256
92
16.7 M
3.51 B
46.30
R7
6
8
256
102
16.5 M
3.46
46,68
R8
12
2
256
158
21.0
4.41
41.18
R9
12
172
20.0
4.20
42.26
Varying DeLighT transformation's width 2mm (Eq.
R10
4
12
2
256
158
21.0 M
4.41 B
41.18
R11
4
12
3
256
158
23.8 M
4.99 B
39.92
R12
4
12
4
256
158
27.1 M
5.69 B
39.10
Varying model width dmm
R13
4
12
2
256
158
21.0 M
4.41 B
41.18
R14
4
12
2
384
158
29.9 M
6,28 B
35.14
R15
4
12
2
512
158
43.8 M
9.20 B
30.81
Deeper and wider near the Input
R16
12
158
21.0
4.41
43.10
Table 7: Ablations on different aspects of the DeLighT block. including uniform vs. block-wise scaling, depth scaling, and width scaling. Rows partially highlighted in color have the same configuration (repeated for illustrating results). Our experimental sctup is similar to Section except that train our models for 50K iterations. Multiplication and addition operations (MACs) are computed for 20 time steps.
and linear layers. In these experiments, the settings the R13-RIS (Table 7 except, Nmaz because models with a stack of linear layers learn many parameters.
Feature shuffling: Figure 2 shows feature shuffling improves the performance of DeLighT by perplexity points. Here, we use the same settings as in R13-R15 (Table 7
Light- FFN: Figure 10 shows the impact of varying the reduction factor r in the light-weight FFN. We use the same settings as in R13 (Table 7 We did not observe any significant drop in performance until 4, we
drop in performance (perplexity increases by ~~2 points). In such cases, the inner dimensions of the light-weight FFN are very small and hurt performance. Notably, the lightt FFN with r = 2 delivered the same performance as r = 2-2, 2, but with 1.28x fewer network parameters, At r = 2 2-2, the lighht FFN is the same the FFN in Vaswani et al. (2017). This suggests that the ability of
4. Beyond
DeFINE
w/ shuffle
Perplexity
DeLighT
Perplexts eeyiy
40
w/o shuffle
Linear
40
35
35
1.3 x
20
40
60
20
40 Parameters (in million)
25
30
35
Parameters (in million)
Figure 8: Impact of different transformations. DeLighT transformations are more parametric efficient than DcFINE and linear transformations. Lower perplexity value means better performance.
Figure 9: Impact of feature shuffling. Feature shuffling allows us to learn representations from global information and improves performance. Lower perplexity value means better performance.
16

---

Published as a conference paper at ICLR 2021
44
Parameters (in mi million)
20.8 8
43
Perplexity
42
26.9
21
41
22.2
21.0
40
2-2
2
2
2
23
Reduction factor (r)
Figure 10: Impact reduction factor r light-weight FFN. The ability of Deli gh'l' transformation to learn representations high-dimensional spaces efficiently allows us to reduce the computational burden on the FFN. Lower perplexity value means better performance.
80
Nmin=4,
M4a
Mean=4
Nmm Nmax=8,
Mean=8
Uniform Input
Block-wise
Perplexity
70
Nmm==4, Nmax=8,
Mean=5. 6
Input
Nmin
Nmax=12,
Mean=7.6
60
N
No Nmin
B blocks
50
(see Eq.
40
N
NB-1
N Nmax
128
256
384
Output
Output
Model dimension (dm)
Figure 11: Uniform vs. block-wise scaling. (a) contrasts the uniform and block wise scaling methods. (b) compares the results of DeLighT with uniform and block-wiss scaling methods on the WikiText- 103 dataset. DeLighT networks with block-wise scaling delivers better performance across different settings. Lower perplexity value better performance.
DeLighT transformation to learn representations high-dimensional spaces efficiently allows us to reduce the computational burden on the FFN.
We also tested removing the geihh FFN and while it reduced parameters by ~0.5-1 M, performance dropped by about 2-3 perplexity points across different parametric settings.
Uniform block-wis wise scaling: Figure II compares the performance of DeLighT with uniform and block- wise scaling. For given model dimension d,n, Delli gh' models with block-wise scaling delivers better performance.
Position of DeLighT transformation: We studied three configurations for the Del.i gh' transforma- tion on WikiText- 103 validation set (Table [8: (1) DeLighT transformation followed by single-headed attention and light-weight FFN, (2) single-headed attention followed by DeLighT transformation. and (3) hingle-headed attention followed by De DeLighT transformation and light-weight FFN, For similar number of parameters, found that (2) and (3) drops the performance of (1) significantly across different parametric
settings. This suggests that deeper and wider representations helps learn better contextual representations; allowing us to replace heade- attention with single- attention.
Scaling up DeLighT: Figure 12 shows the results of DeLighT models obtained after varying configuration parameters of DeLighT transformations (Nmin=(4. 6). NMaz==(8, 12). Wm=12. 3, 4). and /256. 384, 512)). We can see that scaling one configuration parameter (e.g., dm) while keeping other configuration parameters constant (e.g., Nmins Nmaz, and mm consistently improves performance.
17

---

Published as a conference paper at ICLR 2021
Configuration
Parameters
Perplexity
DeLighT transformation Single- head attention Light- FFN
31 M
34,20
Single-head attention + DeLighT transformation
30 M
39,02
Single-head attention DeLighT transformation Light- FFN
31 M
39.43
DeLig ghT transformation Single-head attention weight FFN
M
23.16
Single-head attention DeLig ghT transformation
M
28.33
Single-head attention DeLi ghT transformation Light weight FFN
M
27.94
Table 8: Effect of the position of DeLighT transformation. Lower value of perplexity means better perfor- mance.
70
13
14
70
13
14
Wm
14
Wm 2
60
Wm
60
Wm 3
Perplexity
Wm
Perplexity
Wm 4
50
17
1820
50
16 1828
40
232629
22:25 28
313643
303541 41
128
256
384
512
128
256
384
512
Model dimension (dm)
Model dimension (dm)
(a) Nmin=4, 88
(b) Nmin=6, Nmaz= =8
70
70
Wm 2
Wm 2
15
15
60
16
Wm
60
1617
Wm 3
Perplexity
Wm
Perplexity
Wm 4
50
50
2124
2023
40
40
303541
283238
445263
414858
30
128
384
512
128
256
384
512
Model dimension (dm)
Model dimension (dm)
(c) Nmin=d, N1
(d) Nmin =6, Nmm==12
Figure 12: Scaling up DeLighT. Scaling one configuration parameter (e.g.. d...) while keeping other configu- ration parameters constant (e.g., Nmin, Nmaz. and w.) consistently improves performance. The numbers on top of cach bar represents network parameters (in million). Lower value of perplexity means better performance.
This work investigates relationships between Nm Nmaz. Wo.. and dm.. manually. We believe that more principled approach, such as compound scaling Tan and Le (2019), that establishes relationships between these parameters would produce more efficient accurate models.
E SOURCE CODE FOR GROUP LINEAR TRANSFORMATION
source code for implementing group linear transformation (GLT) PyTorch is shown Listing The source code for efficiently implementing the grouping function GLT is shown in Listing 2 Since the ungrouping kernll is similar to grouping kernel, we have not shown it here.
The reshape and transpose operations in naive PyTorch implementation for grouping and ungrouping replaced with dedicated CUDA kemels,, resulting in reduced memory footprint and faster training.
18

---

Published as a conference paper at ICLR 2021
Listing 1: "Naive implementation of GLT in Pytorch"
import torch
def glt_function (x, n_groups, weights, bias-None):
'/'
:param x: Input tensor of size [B x N), where is batch size and N
is input dimension
:param n_groups: number of groups in GIT
sparam weights: glt weights [g N/g x M/g]
:param blas: GLT bias (optional) of size Ig x x M/g]
:return: output tensor of size [B M]
bsz xx)
## GROUPING FUNCTION: Converts [2 x NJ tensor to N/g # [B x NJ --> IB x x N/gl
x.centiguous view (bsz, n. groups, -1)
[B g x N/g] --> [g B N/gl
x x.transpose (0, 1) $ transpose so that group is first
## TRANSFORMATION FUNCTION: Transforms from N/g-dimensional space to M/g-dimensional space ##
[g x x N/g] Ig x N/g x M/g] --> Ig x x M/g]
x = torch.bam (x, weights) & multiply with Weights
add bias
if bias is not None: torch, add (x, bias)
# REGROUPING FUNCTION: Converts [g x B x M/g] tensor to [8 x M] 44 [g x B x M/gl --> [B g M/gl x.trranspse (0, 1) # transpose that batch is first [ x g x M/gl --> [B M] = x.contiguous () view (bsz, -1)
return
Listing 2: "Grouping kernel in CUDA"
/* Grouping Kernel: Transforms input from x NI to 1g x N/gl w/ temp Letessyennee scalar_t>
g_obal__________ void grouping Ltt_eeeeeeeee____ forward scalar t* input, const int groups, const int total_elements, const int input_features, const int group_features, const int batch_size, scalar_t* cutput) const int index IMDL (blockIdxx x, blockDim.x) threadIdx. if (index >= total_ elemens)) return;
const b_idx index group_features; const int g_f_idx = (index group_features); int in_offset, out_offset;
#pragma unroll
for (int g=0; g groups; g++){
on idx input_ + (g
g_f_idx;
out offset = ((g batch size + b. idx) group group_features + idx;
output [out_offset] [nnot(in_fffset]
19


Reviews:
[{"review_id": "u_a_lCwSbon", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a variant of Transformer where low-dimension matrix multiplications and single-head attention are used. Stacked group-linear-transformation (GLT) are applied on input of each layer to perform dimension growth and then reduction. The paper is well-written and easy to follow. Experiments demonstrate the propose architecture matches or improves the performance of baseline Transformers with fewer parameters.\n\nAlthough the proposed architecture has fewer MACs, it would be interesting to know the real decoding time. Deep models could reduce model size or improve performance, but layers have to be executed consecutively and may slow down decoding in practice especially when using deep decoders. I’m also curious about the performance when keeping decoders unchanged.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "leDAD3GdnTv", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new transformer model called DeLighT. DeLighT differs from the original transformer in the following ways:\n- a DeLighT transformation is performed before the self-attention \n- the self-attention is performed in a vector space with fewer dimensions and with only 1 head\n- the MLP’s hidden layer is 16 times smaller \n- The DeLighT transformation, as far as I understand, is a sequence of block-sparse linear layers and permutations. The number of intermediate layers in DeLightT transformations is higher in the upper layers of the whole model. \n\nThe key claims of the paper are that DeLight performs as well as or better than Transformer, while being deep, having less parameters and involving fewer floating point operations. With regard to depth, I don’t think that depth can be the goal in and of itself, it is the means for achieving good performance. I think the paper puts too much emphasis on DeLight depth. I also find it rather controversial that the depth in the paper counts all linear layers, and not just the number of nonlinearities between the input and the output.\n\nDeLighT does seem to have a consistent edge over the original Transformer model in terms of both performance and memory footprint. Compared to TransformerXL and EvolvedTransformer, DeLighT performs on par, but has less parameters. The set of experiments performed to assert DeLighT strengths is quite extensive.  The experimental side of paper appears technically sound. \n\nIt is important to build more efficient and compact models, but I find the paper’s analysis of models computational demands rather incomplete. If computational efficiency is the main focus of the paper (and it appears to be), it would be great to see some analysis of how much time and memory is required for model training and inference. Is the model size the dominating factor here? \n\nOverall, the architecture ends up being quite complicated. The supplementary material contains an extensive set of ablations justifying many of the complexities, but I find that a few straight-forward remain unanswered:\n- how does DeLighT compare to a Transformer with one head\n- how does DeLighT compare to a Transformer with a narrow MLP\n- what about both above modifications? \nI think it is important for the paper to justify the use of the DeLighT layer. \n\nWith regard to writing, the paper is generally quite clear and accurately done. A few places raised my eyebrows: \n- “Since the DeLighT block learns wider representations of the input using the DeLighT transformation, it enables us to replace multi-head attention with single-head attention” - I think the causation here is not obvious and needs to be proven. Or the wording should be changed to illustrate that this is just your intuition.\n- “This is likely because scaling model width and depth allocates parameters uniformly across blocks, which may lead to learning redundant parameters.” - same, I think it is important to state that this is a hypothesis\n- I think the “input mixer” layer and how it widens the state should be discussed in the main text. Otherwise the role of w_m is completely unclear. \n\nThe paper’s novelty appears quite limited. DeLightT layers seem similar to the DeFINE layer. The difference is not sufficiently explained neither in the main text nor in the appendix. Overall it seems that the paper takes existing components and does a lot of tweaking to yield a better model with good performance. \n\nIt is hard to make a call in the case of this paper. \nPros:\n- the final moment model does deliver some improvement\n- the execution is quite accurate\nCons:\n- the paper feels very incremental \n- the papers leaves an impression that its key goals could be achieved in a simpler way\n- compute time and runtime memory footprint are not discussed. \n\nUPD: the score was updated after the rebuttal stage.", "extra_scores": {}, "extra_text": {}}, {"review_id": "giVV6wyDWmU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The original Transformer includes many design heuristics, e.g., the arrangement of layers and how layer widths are tied ($d_{ffn} = 4 \\cdot d_{model}$, $d_k=d_q=d_v$, etc). While much effort has been spent on up-scaling, it is also important to question these basic design heuristics.\n\nThis work does just that and presents a more parameter-efficient Transformer variant by combining existing techniques (GLT, Mehta et al, 2018; Channel Shuffling, Zhang et al. 2018) and exploring some known trade-offs (feedforward net ratio, nheads vs dhead) under the new architecture. The resulting model, DeLighT, while more complicated to describe, performs on par with the original Transformer with far fewer trainable parameters. The ablation studies seem thorough and most design choices are more or less accounted for.\n\nMy main concern is the complete absence of any discussions on time-efficiency, or latency. The only reference I could find is in the conclusion: “we expect a dedicated CUDA kernel for DeLighT units to be much more efficient, both in terms of speed as well as memory during forward and backward passes.” This implies that the current implementation might not be efficient speed-wise. However, even after excluding the benefits from CUDA kernels, this new architecture appears to be less parallelizable as it is significantly deeper than other Transformer variants. This is crucial as neural network accelerators are becoming increasingly available, even in edge devices. Before fully recommending this paper for acceptance, I’d like to see more discussions on how the proposed changes affect latency, a latency benchmark against other relevant Transformer variants, especially with parallelism, and some concrete statements regarding latency in the abstract and the conclusion. I would still consider this work valuable even if the result on latency is not strongly positive, but the work seems incomplete without any discussions of it.\n\nMinor issues:\n1)     In table 4, Delight has a context length of 480. Why not 640 for a more direct comparison?\n2)     There’s little detail on how the hyperparameters are selected for the ablation study. Are they swept to account for the change in optimal hyperparameters with architectural change?", "extra_scores": {}, "extra_text": {}}, {"review_id": "-FzMy9xHS4g", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper replaces the standard MLP block with a novel building block called DeLight. DeLight is a deeper MLP of partially group-wise linear layers, which leads to parameter and potential compute savings. The authors show that their approach outperforms other transformer-like architectures with more parameters on machine translation and language modeling.\n\nThe results of the paper look encouraging at first glance but I have a couple of major concerns with this paper as it is right now: \n\n1) The paper's motivation is that parameter rich, wide and shallow models are problematic to train due their requirement of large training data or strong regularization. However, the authors use the same training setup for both delight and standard transformers on the same datasets, and results for standard transformers are just fine, though lacking a bit behind delight. In particular, the authors do not show that using delight significantly alleviates any of the mentioned problems. They do not show that it needs less data, nor that it is \"easier\" to regularize.\n\n2) The paper seems to suggest that using less parameters is better per-se, which I strongly disagree with. Especially with huge amounts of data the main driving factors for performance are compute + number of parameters. Making networks wider (instead of deep) has the advantage of massive parallel computation, which is always faster than deeper networks. That's one of the main reasons why RNNs are not used anymore in language. Furthermore, deeper models will have other issues such as storing many more activations which can result in memory bottlenecks.\n\n3) The paper misses to discuss the most important factor which is train efficiency. In other words, how much faster can delight lead to better results than standard transformers? Note, that \"larger models [can] train faster\" [1], which is especially the case given large enough datasets which is definitely the case for language modeling. Unfortunately, this study optimizes for and normalizes by number of parameters. However, it should normalize by on-device training time or at least the idealized training time in flops or MACS. Right now there are some MACS comparisons but they are just for a single forward pass, not total training, which is a bit meaningless. I would like to see a proper comparison of performance to training time/flops. The best way to do this study would be to control for total training time/flops.\n\nFinally, without showing a clear advantage in terms of training speed it is hard to judge the significance of this architectural block. I would also like to see how the differences progress when increasing the total compute equally for the vanilla transformer and delight. Will the gap between those widen or narrow the larger the models become? From my experience, the larger the models and datasets it is mostly the total compute that decides the performance, so I would actually expect the gap to become narrower.\n\nNow I realize that the operations required by DeLight are not well supported by current accelarators (and they probably won't be in the near future), because memory shuffling is very expensive on these devices. And that's why most of the significance of this work would rely on custom kernels or even hardware. The results presented by the paper, though good, are not convincing enough for me to suggest that this customization is necessary.\n\n[1] https://arxiv.org/abs/2002.11794\n\n== UPDATE after rebuttal ==\n\nI am willing to update my recommendation to weak accept after the rebuttal. Some of my main concerns remain, but added experiments, comparisons and discussions alleviate some of those. I still believe that there is too much focus on fairly useless metrics such as number of parameters which might lead to future work that will follow this trend, but the results are overall strong enough to warrant acceptance.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents some innovations to transformers allowing some significant reductions in parameter count. While some reviewers were concerned that the proposed innovations seem incremental and may not stand the test of time, all reviewers recommended acceptance after engaging in a rich and interactive author discussion. Given the clear importance of making transformers more efficient I think this paper will be of interest to the community and is worthy of acceptance at ICLR. 
Decision: Accept (Poster)