Wide Attention is the Way Forward for Transformers
Authors: Jason Ross Brown
Yiren Zhao
Ilia Shumailov
Robert D. Mullins
Venue: Submitted to ICLR 2023 (2023)
Keywords: transformer
attention
wide
deep
accuracy
latency
interpretability
xformer
size

Abstract: The Transformer is an extremely powerful and prominent deep learning architecture. In this work, we challenge the commonly held belief in deep learning that going deeper is better, and show an alternative design approach that is building wider attention Transformers. We demonstrate that wide single layer Transformer models can compete with or outperform deeper ones in a variety of Natural Language Processing (NLP) tasks when both are trained from scratch. The impact of changing the model aspect ratio on Transformers is then studied systematically. This ratio balances the number of layers and the number of attention heads per layer while keeping the total number of attention heads and all other hyperparameters constant. On average, across 4 NLP tasks and 10 attention types, single layer wide models perform 0.3% better than their deep counterparts. We show an in-depth evaluation and demonstrate how wide models require a far smaller memory footprint and can run faster on commodity hardware, in addition, these wider models are also more interpretable. For example, a single layer Transformer on the IMDb byte level text classification has 3.1x faster inference latency on a CPU than its equally accurate deeper counterpart, and is half the size. Our results suggest that the critical direction for building better Transformers for NLP is their width, and that their depth is less relevant.

Under review as a conference paper at ICLR 2023
WIDE ATTENTION Is THE WAY FORWARD FOR TRANSFORMERS?
Anonymous authors Paper under double-blind review
ABSTRACT
The Transformer is an extremely powerful and prominent deep learning architec- ture. In this work, we challenge the commonly held belief in deep learning that going deeper better, and show an alternative approach that is building wider at- tention Transformers. We demonstrate that wide single layer Transformer models can typically equal or sometimes outperform deeper ones in a variety of Natural Language Processing (NLP) tasks when both are trained from scratch. The impact of changing the model aspect ratio on Transformers studied systematically. This ratio balances the number of layers and the number of attention heads per layer, while keeping the total number of attention heads and all other hyperparameters constant. On average, across 4 NLP tasks and 10 attention types, single layer wide models perform 0.3% better than their deep counterparts. We show an in-depth evaluation and demonstrate how wide models require far smaller memory foot- print and can run faster on commodity hardware, in addition, these wider models are also more interpretable. For example, single layer Transformer the IMDb byte level text classification has 3.1x faster inference latency on CPU than its equally accurate deeper counterpart, and half the size. We therefore put forward wider and shallower models as viable and desirable alternative for small models on NLP tasks, and as an important area of research for domains beyond this.
INTRODUCTION
Since Vaswani al. 2017), Transformer- architectures have become widespread due to their advantages over previous architectures such as recurrent neural networks (RNNs) and sometimes convolutional neural networks (CNNs). Many new X-formers have also been proposed that improve on the original Transformer by overcoming its limitation on sequence length by providing more scalable attention mechanism (Choromanski al. 2020 Wang et al.) 2020b Beltagy al., 2020). However, little research has been done on the relevance of the size of the attention compu- tation in each layer, the number of attention layers, and how these parameters relate the resulting Transformer's 's characteristics.
The primary source of parameters in each encoder or decoder layer, and the linear layers which convert from the sequence feature dimension (often equal the initial embedding dimension) to the attention feature dimension, and back again after attention is applied. Each attention head typically has an equal number of attention features. Consider an input sequence E S where is the sequence length and E is the embedding dimension. Here, multi-head attention with H heads is used and each head operates on the learned projection with dimension of A. After the attention mechanism there is a FFN with single hidden dimension of size M. These layers are then stacked L times illustrated the left of Figure [ Often in a typical Transformer E. = AH. The total number of parameters Transformer encoder is given by:
Transformer network is the Feed-forward Network (FFN)
Encoder Parameters L(3EAH + AHE EM ME)
22LE + M)
In this paper, we investigate the effects of changing L and H while keeping their product, the total number of heads, the same. We start with typical values for I. and H and then down single

---

Under review as a conference paper at ICLR 2023
Ourgut
Ouppu
Layer & Softsax
Lincar Layer Softmax
1. Layers $
Network
8
Concateration Layer
Concationation Linear Layer
Heads
Attention
Atsention
Attention
Atteniion
Q
Linear Layers
QCV Lincar syers
Linear Layers
Linese Layers
B
Input& Positenal Eibedding
Enput Posttiona Embedding
Taput
Inpur
Figure 1: A comparison of deep Transformer based classifier (left, with L layers and H heads for each layer) vs an equivalent wide one (right, with single layer and L x H heads). Layer norms and residual connections have been omitted from the diagram for clarity, for details on the full Transformer architecture see Vaswani et al. (2017).
layer. A diagram illustrating our design space and the differences between our widest and deepest models is given in Figure [ We refer the ratio of layers heads as the model aspect ratio. This naturally leads to an intriguing question: What is the best model aspect ratio for the growing number of X-former models? We consider impacts of model aspect ratios accuracy, run-time performance, model size, and interpretability.
Based on the question above, we investigate the influence of various model aspect ratios on 9 X- former models, each with their own attention mechanism, in addition to the original Transformer. Prior work on Transformer architectures has mainly focused on designing more efficient attention styles (Wang et al.) 2020b Choromanski et al., (020) using Network Architecture Search (NAS) to discover an optimal combination of operators (So et al. 2019). By changing the model aspect ratio, we consider a more coarse-grained design space. This design space is not commonly explored in the NAS algorithms for Transformers, and we evaluate some interesting model architectures such single layer model with many parallel heads.
For each model aspect ratio we run our experiments with each X-former across a number of text classification tasks with various input sequence lengths ranging from 500 to 4000. We empirically observe that wider and shallower models can typically equal or sometimes beat the accuracy of deeper models. This observation challenges the common design paradigm of trying build deeper Neural Networks. We show several other major advantages of a shallower and wider model. First, more latency friendly commodity hardware. Second, wider models are smaller in terms of the number of parameters. And, third, outputs of wider models are more interpretable.
To summarise, we make the following contributions in this paper:
We demonstrate that wider and shallower models can typically equal or sometimes beat the accuracy of deeper models when there is no pretraining of weights or embeddings. Across
2

---

Under review as a conference paper at ICLR 2023
all 4 tasks, average accuracy for the vanilla Transformer increases by 0.4% between normal deep models and our single layer wide models.
We show that our results are consistent across variety of different attention mechanisms and input sequence lengths, and thus there is a general design equivalence in increasing the depth of a Transformer model vs increasing the width. Averaged across all non-vanilla attention types and tasks, accuracy increases by 0.3% from deepest to widest.
We show that widening the models by fixing the attention computation size results in less overall parameters and faster inference. We show that wider models are average 1.4x smaller and have 3.1x X faster inference latency on a CPU and 1.9x on a GPU, compared to deep models.
We demonstrate how single layer networks can have more interpretable predictions by in- specting the attention weights of each head in single layer.
2 RELATED WORK
2.1 WIDER NETWORKS
Zagoruyko Komodakis (206)) and Wu et al. (2019) respectively show and investigate how widen- ing and making shallower ResNet CNNs (He et al., 2016) can improve their performance. Trans- formers also use residual connections, so this helps motivate our investigation.
Xue et al. (2022b) find that wider layers in Transformers (in both attention and sequence features) can improve performance on vision and natural language tasks. However, they use a mixture-of- experts layer instead of the typical feed-forward network at the output of the Transformer encoder. They do this because a much larger dimension of sequence features means the number of parameters in the feed-forward layer would increase greatly. As we are only increasing the attention feature dimension we do not face this issue.
Xue et al. (2022a) investigate masked autoencoder training and how it can help reduce the problem of over smoothing in training deep Transformer networks embeddings of tokens converging to be similar at deeper layers. They explore the optimal configuration of the Transformer when using masked autoencoding to pretrain it, and then fine-tune it on specific tasks. They find that for vision Transformers, it is better to pretrain with a deep autoencoder rather than a wide one. As we are not using pretrainining and pretrained embedings, their results are not directly comparable to our own.
2.2
ARCHITECTURE OPTIMIZATION FOR TRANSFORMERS
lot of research in the Transformer field focused on finding a more efficient attention mechanism (Kahharppollos al.) 2020 Wang et al. 2020b Zaheer et al., 2020 Liu al. 2018 Beltagy al., Choromanski et al.) 2020 Kitaev et al. 2020 Tay et al. 2020a Child et al.j 2019 Tay et al, 2021 as the original dot- product attention has quadratic complexities with respect input sequence length. Long Range Arena (LRA, Tay et al. (2020b)) compares and contrasts these methods, noting that in absence of pretrained embeddings and model parameters, the best attention mechanism is task dependent. Thus there is no clear best attention mechanism. Because this we test different attention mechanisms on a variety of tasks with no pretrained embeddings or model parameters. This makes clear that our findings are largely independent of attention mechanism and task, and contextualise going wide as general design decision.
2020
The application of Neural Architecture Search (NAS) to Transformer models has also been explored. Both Neural Architecture Transformer (NAT) Guo et al.) (2019) and Hardware-aware Transformers
Wang et al. (2020a) use NAS to search for models that are more hardware efficient. [So et al. (2019) use evolutionary search optimize components and connections inside an encoder or decoder layer.
Liu et al. (2022) apply RankNAS (Hu et al., 2021) to cosFormer (Qin et al.) 2022) and standard Transformers (Vaswani et al. 2017). They perform search of the hyperparameters on a cosFormer network and compare these to the same method applied to the standard Transformer. Tsai et al. (2000) searches the hyperparameter space of a BERT (Devlin et al., 2018) model architecture het- erogeneously to find an optimal efficient network. These previous experiments have not varied or searched over different numbers of layers in the Transformer encoder, which is a key factor in what investigate.
3

---

Under review as a conference paper at ICLR 2023
3
EXPERIMENT SETUP
3.1 MODEL ASPECT RATIO
For our Transformer models we fix the number embedding features, sequence features, attention features, and the hidden layer dimension in the FFNs for each task. We vary the number of layers, L. and the number of heads per layer, H, whilst keeping L x constant. Starting with typical values for I. & H, we then move down a single layer with one two intermediate model aspect ratios, observing how test accuracy changes for trained models. See Figure for an illustration of deepest and widest models.
In all of our tasks we do not use pretrained embeddings or pretrained model parameters as this allows us make a fairer comparisons. Computing pretrained embeddings and weights that are optimised for each combination of attention and model aspect ratio would be computationally prohibitive, and using ones typically used for deep networks would introduce bias.
3.2 DATASTS AND MODELS
Table 1: The different tasks and datasets used.
Task Name
Classification
Dataset
Input Type
Input Length
IMDb Token Level
Binary
IMDb Reviews
Review text tokens
500
IMDb Byte Level
Binary
IMDb Reviews
Review text bytes
1000
Listops Document Matching
10-way
LRA Listops
Listop bytes
2000
Binary
ACL Anthology
Document bytes
4000
Primarily we investigate using 4 different text classification tasks, a vision based task is investigated in Section 5.5 The first two are sentiment analysis (binary classification) on the IMDb dataset. One uses input embeddings at the token level with an input sequence length of 500, and the other uses input embeddings at the byte level and an input sequence length of 1k. This second task is taken from LRA (Tay et al. 2020b), as are the final two. The third task is Listops 10-way classification with a sequence length of 2k. This task involves reasoning about sequences of hierarchical operations to determine a result, and the input is given at the byte level. The final task used is byte level document matching, a binary classification task with sequence length of 4k. This uses the ACL anthology network for related article matching (Radev et al. 2013). We summarise each task in Table [ further details them can be found in Tay al. (2020b).
For the text classification and Listops task we try four different model aspect ratios. In terms of number of layers and heads per layer these are: layers, heads; layers, 16 heads; 2 layers, 24 heads; and finally layer, 48 heads. As the matching task has an input sequence length of 4k, the models used are smaller offset the computation size involved. Thus the combinations we use are: 4 layers, 4 heads; 2 layers, 8 heads; layer, 16 heads.
In order investigate whether the type of the attention mechanism influences the effects of widening the attention layer, we test on 10 different types of Transformer attention. including the original dot-produee attention (Vaswani et al. 2017). The others are: Bighird (Zaheer al., 2020). Linear Transformer (Katharopoulos et al. (020). Linformer (Wang et al.] (020b)) Local attention (Liu cet al. 2018), Longformer (Beltagy et al. 2020), Performer (Choromanski et al. 2020). Sinkhorn (Tay et al. 2020a), Sparse Transformer CChid et al.) 2019). and Synthesizer (Tay al., [021 The implementations and
choices for each attention type are the same as used in LRA. Unlike LRA, do not test with Reformer (Kitaev et al.) 2020) due to it requiring the sequence features and attention features to have the same dimension. Training and other Transformer hyperparameters used for each task are given in Appendix
4

---

Under review as a conference paper at ICLR 2023
Table 2: Test accuracy on all tasks for deepest and widest models.
Attention
IMDb Token Wide
IMDh Byte Wide
Listops
Doc Matching Wide
Average
Type
Deep
Deep
Deep
Wide
Deep
Deep
Wide
BigBird
86.0
85.3
62.7
62.4
36.7
37.3
63.9
64.0
62.3
62.3
Linear
86.7
88.0
64.5
64.7
37.1
37.4
64.2
63.9
63.1
63.5
Linformer
84.2
84.1
56.3
52.6
29.9
37.0
64.5
63.1
58.7
59.2
Local
74.2
73.9
55.5
57.0
36.8
37.7
58.0
58.1
56.1
56.7
Longformer
86.0
84.1
61.6
57.5
36.8
37.7
61.2
58.1
61.4
59.4
Performer
87.0
87.0
64.4
64.8
36.2
36.6
65.0
66.3
63.1
63.7
Sinkhorn
86.1
86.5
62.3
61.6
19.4
21.7
64.0
65.7
57.9
58.9
Sparse
85.7
85.7
61.2
62.9
37.0
36.9
63.2
63.6
61.8
62.3
Synthesizer
86.7
86.5
61.4
61.1
36.5
37.3
71.1
72.3
63.9
64.3
Transformer
85.8
85.5
62.6
62.4
35.7
37.2
64.0
64.4
62.0
62.4
Average
84.8
84.7
61.2
60.8
34.2
35.7
63.9
64.0
61.0
61.3
RESULTS
4.1
OVERALL PERFORMANCE
Each combination of task, model aspect ratio, and attention, is ran three times with mean and stan- dard deviations recorded. We first summarise the differences in accuracy for our deepest and widest models in Table 2 For Deep models on the IMDb and Listops classification tasks, we mean Transformer with 6 layers each with 8 attention heads. On the document matching task this is re- duced to 4 layers each with 4 attention heads due to the 4k input sequence length greatly increasing the computational burden. These are the deepest models we consider in our setup. These model as- pect ratios are also the setup used in Tay et al. (2020b). For Wide models, we consider a single layer network that has its number of heads matching the total number in the Deep models. For instance, the original Deep model has a model aspect ratio of 6-8, its wider counterpart is 1-48. These are the widest models in our setup.
Table shows that Wide models achieved +0.3% accuracy compared to Deep models, averaged across 10 different attention types and 5 different datasets. On most tasks the performance of Wide and Deep was similar, with Listops being the exception (+1.5%). This result holds for both the 'vanilla' Transformer with dot product attention, and in general for many other types of attention. Sinkhorn sees the largest average gain by going wide (+1.0%) and of all the attention mechanisms only Longformer gets worse (-2.0%).
4.2 PERFORMANCE BREAKDOWNS FOR EACH TASK
Performance breakdowns for each task and model aspect ratio are given in Tables 3 and with standard deviations and averages.
For IMDb token level text classification, the results (Table 3 left) show that model performance invariant to model aspect ratio, with all averages across attention mechanisms being within 0.2%. Comparing the widest and deepest models for each attention mechanism, only Longformer has a >1% accuracy increase when deep (+1.9%). Conversely, only Linear Transformer has a >> % accuracy increase when wide (+1.3%).
At the byte level (Table 3 right) there's similar picture, with all averages within 0.5%. There however
notable deviations from the pattern of aspect ratio invariance. Two models per- form better by > 1% in their deepest configuration than widest: Linformer (+3.7%) and Longformer (+4.1%). Likewise models perform better by > 1% when widest: Local (+1.5%) and Sparse (+1.7%)
For standard deviations relevant entries in Tables and

---

Under review as a conference paper at ICLR 2023
Table 3: Test accuracy on IMDb token level and IMDb byte level for different model aspect ratios.
Attention
IMDb token level (layers-heads)
IMDb byte level (layers-heads)
3-16
2-24
1-48
6-8
3-16
2-24
1-48
BigBirl
86.0 0.1
85.7
0.1
85.8
1 0.1
85.3
1 0.1
62.7
1 0.4
63.5 1 0.1
63.8
0.4
62.4 0.2
Lineur
86.7 0.7
86.9
1 0.3
87.3
1 0.4
88.0
64.5
1 0.2
64.5 0.2
1.1.
Linformer
0.2
82.4+
0.1
81.1
84.1
56.3
3.6
52.8
5..6+0.1
Local
0.3
74.1 0.1
74.2
1: 0.2
73.9
1 0.1
55.5
0.9
0.2
3..0
Longformer
0.0
85.8 1 0.2
85.7 0.1
84.1
1 0.1
61.6.0.11
61.811
1.81033 0.2
6.3
1.0
Performer
8..0+
87.3+ 0.3
8..0
87.0
0.0
6..4+0.1
64.7
64.7-0.1
6.18
Sinkhorn
8.31
86.4 0.2
8.14
86.5
62.3
0.0
6.1
61.9 0.2
61.6
0.1
Sparse
8.2
85.6 0.4
85.8
0.1
85.7
61.2
0.6
-2.1-0.2
1.2
Synthesizer
0.1 0.2
87.1
0.2
86.5
61.4
0.2
0.0
-1.1-01
6.11
Transformer
85.8+ 0.8
85.8 + 0.1
85.9 + 0.1
85.5 0.1
62.6 + 0.4
63.4 + 0.2
03.3
0.2
+
Average
84.8 0.1
84.7 0.1
84.6 0.1
84.7
61.2 + 0.4
612
61.3 0.1 0.1
60.8 0.1
Table Test accuracy on Listops and document matching for different model aspect ratios.
Attention
Listops (layers-heads)
Document matching (layers-heads)
6-8
3-16
2-24
1-48
4-4
2-8
1-16
BigBird
36.7 0.2
37.0 + 0.7
37.1 + 0.2
37.3 1 0.3
63.9
64.7
0.7
64.0 1.0
Linear
37.1 0.4
37.3 0.4
0.5
6.5
63.8
0.1
63.9 0.5
Linformer
36.9 8.9
0.1
0.1
6.1
63.1 0.4
Local
36.8
0.1
0.3
0.2
0.2
58.1 0.2
Longformer
0.2
0.3
0.2
37.
0.2
0.4
0.6
Performer
0.1
34.1
6.4
3..6
6.5
1.2
+..4
Sinkhorn
2.14
18.0+
1.59
21.7
6..0
6.5
0.6
Sparse
3..0+
3..1+0.1
37.6
36.9
6.2
0.6
63.6
Synthesizer
0.3
0.2
3.89
37.
7..1
7.00
0.6
Transformer
1.2
0.3
0.17
0.2
6..0
64.6
64.4
Average
3..2+
+.3
0.9
3..7+0.4
63.9+
64.0 + 0.2
64.0 0.2
In the Listops task (Table|4 left) we see a strong tendency for wider models to better. For this task individual models in training would often reach 17-20% accuracy and then possibly make a jump to 36-38% accuracy as training went on. Illustrated by the Linformer and Sinkhorn entries, wider models were more likely to make this jump. The data also shows that even if all the runs of an attention and model aspect ratio combination make the jump, the wider variants still have 0.3- 1.5% higher accuracy. Overall there's average increase in accuracy of 1.5% from deepest widest models with many models seeing a 1% increase accuracy.
On the matching task (Table right), there's small difference in performance between wide and deep networks with the three averages being within 0.1% 1% of each other. Two models perform better by >1% when deepest than widest: Linformer (+1.4%) and Longformer (+3.1%). Three models perform better by >1% 1% when widest than deepest: Performer (+1.3%), Sinkhorn (+1.7%), and Synthesizer (+..2)%.
The results in Tables 3 and demonstrate some general trends. First, for some attention types, the original 6-8 model aspect ratio never shows optimal performance, egg. the original Transformer model. Second, the nature of the task seems to affect the impact of going wider. Listops saw significant increases compared to the others, perhaps this due to hierarchical reasoning benefiting from more attention heads. Overall, we would like to highlight the following observations from our results:
Wide Transformer networks typically offer equal or sometimes greater accurucy on range of classification lasks with different sequence lengths.
Increases in accuracy are task dependant. Listops had an average accuracy increase of 1.5% when going wide, whereas all the others had changes of < 0.5%.
The attention mechanism has some effect whether wider deeper is better with Long- former and Sinkhom being more sensitive to the model aspect ratio. We provide possible explanations for these outliers in Section 5.4

---

Under review as a conference paper at ICLR 2023
5 DISCUSSION
5.1 PERFORMANCE MODEL SIZE
As seen in Section wide Transformer networks typically offer equal greater accuracy range of classification tasks with different sequence lengths. The impact of wider and shallower networks accuracy is slightly influenced by the attention type, with some having more significant changes in accuracy than others. There was no significant difference between convergence time during training for wide and deep models. Each attention mechanism on each task achieved its best validation accuracy after roughly the same number of training steps for all model aspect ratios.
Whilst the total number parameters involved in the attention layers remains constant amongst different aspect ratios, the overall number of parameters decreases. This is due to the feed-forward network (FFN) part of the Transformer layer remaining unaltered as we change the models width. The models with fewer layers have fewer FFN blocks, and so fewer parameters.
The deepest IMDb byte level classification and Listops models are typically 230MiB, with the widest models typically being 1 10MiBB, only 48% of the size. On token level classification and document matching the sizes are closer. Averaged across tasks and attention mechanisms, the widest models are 71% the size of the deepest models. A full table of model sizes for the deepest and widest models across all tasks and attention types is given in Table in Appendix
5.2 LATENCY
As the number of layers in model decrease, does the number of dependencies in the computation graph. Because this a forward pass through the model will have lower latency, though overall throughput may remain unchanged. For systems which require very low latency, such real time processing in autonomous driving (Talpes et al. (2220), this feature could make a wide single layer model more desirable than an equally accurate deep one.
We measure inference latency for the deepest and widest models for each task and attention type on CPU with single input, and on GPU with a batch input. Experimental details are given in Appendix C along with raw numbers in Tables | and I0] We find that on average the widest models are 3.1x faster on CPU and 1.9x faster on GPU than the deepest models. The speed-up is consistent across all tasks and attention types.
5.3
INTERPRETABILITY
7
16
8
17
9-
18
10
19
11
20
This
novie
terrible whilst
lover
the main
the plot was kny
silly
Other movies
this director
have
terrible however
loved this one because
of the
character
The plor
was
litte
silly
(a) Prediction: strongly negative
(b) Prediction: weakly positive
Figure 2: The attention weights across selection of heads with predicted classification for the single layer IMDb token level text classification Transformer model on two unseen and similar examples.
Interpretabillity is increasingly important and a very active area of machine learning research (Gilpin al. 2018: Linardatos et al., 2020), especially when it comes to fairness. By having more easily inspect-able models, we see the reasons for given classification. For example, was a decision based on the mention of a protected characteristic, such as race or gender?
In Transformer-based architecture, the attention heads in layer can be inspected during inference to see what connections between input features that head found important. For deep networks, many layers means it can often become unclear what the final output was actually based on (Rigotti al. 2021]. For single layer wide network. interpretability is far easier as only one layer needs to be inspected. Thus what was considered important for the final output much clearer.

---

Under review as a conference paper at ICLR 2023
the attention weights across some of the heads of the widest token level text classification Transformer model, as well as the predicted class for two different example inputs that have been designed to be similar. Figure 2 Iiclluds subset of the total number of heads, for all 48 heads see Figure 3 in Appendix D
In Figure 2
We see the review the left has been confidently assigned negative review, and from the attention weights we can see that this due to the model recognising the relevance of the word "terrible". The review on the right has been less confidently assigned as positive. From the attention weights we can see it has recognised words such as "terrible", but the largest weight is on "The". This explains why the model might be confident its positive prediction because hasn't realised the importance of the word "loved".
5.4 THEORETICAL EXPLANATIONS OF OUTLIERS
Most attention mechanisms usually have up to a 0.5% increase in accuracy when going wider with two notable exceptions. Longformer (Beltagy et al., 2020) typically performs significantly better when deep, and Sinkhorn (Tay et al. 2020a) typically performs significantly better when wide.
For Longformer we only use sliding window attention, with a width of 512. This means, particu- larly for the longer tasks, each input feature can only have attention computed between it and its neighbours. For deeper models, features can propagate and so this limitation is reduced. However, single layer models suffers performance penalty.
Sinkhorn works similarly to local attention, where the input sequence is divided into blocks. Unlike local attention, which computes attention within these blocks, Sinkhorn sorts them and computes attention between the original block and the newly sorted block. This sorting mechanism is learn- able per head., thus each head can learn different sorting strategy. For lots of heads this maximises the overall chance of important long range connections within the input sequence being attended to.
5.5 VISION TRANSFORMER
There is growing interest in applying Transformers to computer vision (Wang et al., 2021 2022
Liu et al., 2021; Dosovitskiy al. 202I). We tested the Pyramid Vision Transformer model (PVT- V2-B1) Wang et al. 2022) and its wider variants on the CIFARIO dataset (Krizhevsky et al.) (014)). The PVT-V2-B1 model has four stages with each stage containing two attention layers. We then replace the two attention layers at each stage with single wide attention layer. The detailed ar- chitecture of the PVT-V2-B1 model and its wider variants are shown in Table [ The first wider variant (Wide) matches the total number of heads to the original PVT-V2-B1 model, whereas the later variant (Wide- V2) contains a larger embedding size and more heads. This a closer match to the original model in terms of the model size.
Table 5: Performance of the original Pyramid Vision Transformer (PVT) and its wider alternatives on the CIFAR10 dataset. The original model (PVTV2-B1) has 4 stages, each stage contains two attention layers. ((1, 1), (2, 2), (5, 5), (8, 8)) descrihes the original model, for instance, the first block contains two layers with single head each, represented (1, I).
Name
Configuration
Accuracy
Parameters
Bascline
(L,, (5,5), (8,8))
95.59 0.99
13.5M
Wide
(2)), (4), (10), (16))
94.54 + 0.31
7.7M
Wide-V V2
((4), (8), (20). (32))
94.94 0.20
12.6M
Table 5 illustrates that the wider variants do not outperform the original PVT-V2-BI model. Intu- itively, spatial features play an important role in vision tasks. The average pooling layer that comes before each attention layer is a crucial component the PVT model. With only a single wide layer, this pooling layer is not capturing much spatial information as before. The usage of single wide attention layer in vision Transformers is constrained by the fact that the majority of these vision Traanformers still use pooling or convolution layers before the attention.
8

---

Under review as a conference paper at ICLR 2023
Table 6: Test accuracy for each task for wide and deep mixed attention models alongside the averages of all homogeneous models, and the best performing homogeneous model for that task.
Deepest
Widest
Task
Hom. Avg
Hom. Best
Mixed
Hom. Avg
Hom. Best
Mixed
IMDb Token Level
84.8
87.0
87.3
84.7
88.0
86.8
IMDb Byte Level
61.3
64.5
60.1
60.7
64.8
63.0
Listops
34.2
37.1
37.1
35.7
37.7
38.0
Document Matching
63.9
71.1
64.0
72.3
66.9
5.6 MIXED ATTENTION
With wider attention layers, the advantages of mixing attention methods becomes more viable. We test a uniform mixture of attentions both wide and deep variants to see how these models per- form. For each layer we use equal mix of the following attention mechanisms: BigBird, Linear Transformer, Linformer, Local, Longformer, Performer, Sparse Transformer, and Synthesizer.
We omit Sinkhorn since it does not the [CLS] token for classification like the other attention methods. We also omit vanilla attention so that we have a total of 8 mechanisms, and our overall attention operation is efficient (sub-quadratic time and space complexities).
We initialise a separate multiheaded attention block for each mechanism and average all of their outputs when going back to the sequence features. The number of attention heads in each block scaled such that the total number equals that of the homogeneous model. For example the layers, heads configuration, each of our attention blocks has single head. In the layer 48 heads configuration, each has 6 heads.
Results given in T Deep mixed attention matching is not tested as there not enough heads to include every attention mechanism. We also include in this table repeats of the averages and bests for each task across all attention mechanisms. From the table we can see that even with mixed attention, the widest single layer models typically perform better (IMDb byte level, Listops) or only marginally worse (IMDb token level) than the deep Transformer models. Whilst beating the averages, all the best mixed models except Listops are outperformed by one of the homogeneous attention models in its widest configuration. The widest mixed model on Listops however points towards mixed attention having possible advantages over homogeneous attention for certain tasks.
CONCLUSION
When fixing the size of the attention computation, in general wider Tranformers sometimes out- perform, or else match, their deeper counterparts on NLP based tasks. Averaged across all tasks and attention mechanisms, Wide models achieved +0.3% accuracy compared Deep models. On most tasks the performance of Wide and Deep was similar, with Listops being the exception (+1.5%). On average going wide works well for almost all attention mechanisms. Sinkhorn benefits the most from being wide, whereas Longformer loses performance.
Furthermore, wider networks are much smaller, only 71% the size of our deepest models on aver- age. They also have additional advantages in interpretability and inference latency: 3.1 x faster on CPU and 1.9x faster on GPU for IMDb byte level text classification. image based tasks how- deeper Tranformers are still superior due to pooling layers being able to capture more spatial information.
Whilst our study considers the effects of attention width across many attention mechanisms and various tasks, remains to be seen whether these results hold for much larger models on much larger datasets, for example masked langauage modelling. We therefore put forward wider and shallower models as a viable and desirable alternative for small models on NLP tasks, and as an important area of research for domains beyond this.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
Section 3.1 along with Figure show how we are altering the standard Transformer architecture. The specific tasks trained our Transformers are given in Section 3.2 with details on the hyper- parameters used for training in Appendix A
The main body of our code can be found at GIT-LINK.
REFERENCES
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint FrXXi:2004.05/555 2020.
Rewon Child, Scott Gray, Alec Radford. and Πya Sutskever. Generating long sequences with sparse transformers. arXiv preprint urXiv:1904. 10509, 2019.
Krzysziof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18XXiv:8800.4440 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. 2021. URL https: //openzeview.c net/forumt 1d=YicbFdli
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang, Nat: Neural architecture transformer for accurate and compact architectures. Advances Neural In- formation Processing Systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Chi Hu, Chenglong Wang. Xiangnan Ma. Xia Meng, Yinqiao Li, Tong Xiao, Jingbo Zhu, and Changliang Li. Ranknas: Efficient neural architecture search by pairwise ranking. arXiv preprint arXiv:2109.07383, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, tukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint 0rXiv:2001.04455) 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 10 dataset. online: http://www. toronto. edu/kriz/cifar: huml, 55(5), 2014.
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretabilityy methods. Entropy, 23(1):18, 2020.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint 1rXiv.1801.10198 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF International Conference Computer Vision pp. 10012-10022, 2021.
10

---

Under review as a conference paper at ICLR 2023
Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixuan Sun, Jiacheng Xu, and Yiran Zhong. Neural architecture search on efficient transformers and beyond. arXiv preprint arXiv:2207.33955, 2022.
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Ling- peng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08791. 2022.
Dragomir Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus. Language Resources and Evaluation, 47(4) 2013.
Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind. and Paolo Scotton. Attention -oanuedoo---seddddeee interpretability with concept transformers. In International Conference Learn- ing Representations, 2021.
David So, Quoe Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pp- 5877-5886. PMLR, 2019.
Emil Talpes, Debjit Das Sarma, Ganesh Venkataramanan, Peter Bannon, Bill McGee, Benjamin Floering, Ankit Jalote, Christopher Hsiong, Sahil Arora, Atchyuth Gorti, et al. Compute solution for tesla's full self-driving computer. IEEE Micro, 40(2):25-35, 2020..
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pp.
PMLR, 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint arXiv:2011 0tt::201110400000 0.0006, 2020b.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Re- thinking self-attention for transformer models. In International conference on machine learning, pp. 10183-10192. PMLR, 2021.
Henry Tsai, Jayden Ooi, Chun-Sung Ferng Hyung Won Chung, and Jason Riesa. Finding fast transformers: One-shot neural architecture search by component composition. arXiv preprint arXiv:2008. 06800 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005.14188 2020a.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 20Xx:20006.07768, 2020b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415 424, 2022.
Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90:119-133, 2019.
Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Xin Jiang, and Yang You. Deeper vs wider: A revisit of transformer configuration. arXiv preprint arXiv:2205. 10505, 2022a.
11

---

Under review as a conference paper at ICLR 2023
Fuzhao Xue, Ziji Shi, Futao Wei, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. In Proceedings of the AAAI Conference on Artificial Intellignce, volume 36, pp. 8779- 2022b.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems,
2020.
12

---

Under review as a conference paper at ICLR 2023
A
EXPERIMENTAL TASK DETAILS
Table 7: Task specific hyperparameters, see introduction for definitions.
Task
Training Steps
Warmup
Seq. Length
E
M
IMDb Token Level
30k
8k
1
512
64
2048
IMDb Byte Level
30k
8k
1k
512
64
2048
Listops
10k
1k
2k
512
64
2048
Document Matching
10k
8k
4k
128
32
512
In all tasks we tested on, used an Adam optimiser with linear warmup and square root decay. We used a base learning rate of 0.05, 31 0.9, and [3 0.98 and a batch size of 32 for every task. Attention kernel and MI.P weights were initialised via uniform xavier. Biases, input embeddings, and positional embeddings were initialised via Gaussian distributions with variances of 10-", 1, and 0.02 respectively. All attention mechanisms, except Sinkhorn, used the [CLS] token for classifica- tion. Sinkhorn used mean classifier pooling. Task specific hyperparameters are given in table 7 In all Transformer models the feed-forward network (FFN) has a single hidden dimension. Before test- ing take the model which had the best validation accuracy during training, the long train times are to ensure convergence.
B
MODEL SIZE
Table 8: Model sizes for all tasks and attentions for deepest and widest models (MiB).
Attention
IMDb Token
IMDb Byte
Listops
Doc Matching
Average
Type
Deep
Wide
Deep
Wide
Deep
Wide
Deep
Wide
Deep
Wide
BigBird
779
659
230
110
229
109
13
8.0
313
222
Linear
779
659
230
110
229
109
13
8.0
313
222
Linformer
780
659
232
110
231
109
16
8.7
315
222
Local
779
659
230
110
229
109
13
8.0
313
222
Longformer
833
713
284
164
283
163
15
11
354
263
Performer
779
659
230
110
229
109
13
8.0
313
222
Sinkhorn
767
647
218
98
217
97
13
8.0
304
213
Sparse
779
659
230
110
229
109
13
8.0
313
222
Synthesizer
761
641
230
109
300
179
60
55
338
246
Transformer
779
659
230
110
229
109
13
8.0
313
222
Average 782
661
234
114
241
120
18
13.1
319
227
Table & shows the size of the deepest and widest models for every task and attention combination in mebibytes.
C LATENCY
To test latency input random full length input into the model 100 times and take the average of the time taken. The latency times for single input on CPU are given in Table 9 the hardware used 2x AMD EPYC 7763 64-Core Processor 1.8GHz. GPU times given in Table IO for this input batch of 32 inputs. We use a single Nvidia A 1000-XM-80GB GPU.
D
FULL ATTENTION SCORES
Figure shows the attention scores for every head for example in Section 5.3
13

---

Under review as a conference paper at ICLR 2023
Table 9: Latency for all tasks and attentions for deepest and widest models on CPU (ms).
Attention
IMDb Token
IMDb Byte
Listops
Doc Matching Deep
Average
Type
Deep
Wide
Deep
Wide
Deep
Wide
Wide
Deep
Wide
BigBird
116
46
106
24
103
30
125
33
113
33
Linear
54
33
11
33
11
26
10
37
16
Linformer
60
34
39
12
39
12
34
12
43
18
Local
59
34
11
27
38
17
Longformer
70
37
62
17
149
37
939
263
305
89
Performer
53
32
11
55
35
28
9
42
22
Sinkhorn
83
39
59
16
59
17
58
21
65
23
Sparse
56
40
38
14
47
27
88
66
57
37
Synthesizer
257
237
866
11
130
20
45
23
324
73
Transformer
60
34
40
13
41
16
56
36
49
25
Average
87
57
131
14
69
22
140
48
107
35
Table 10: Latency for all tasks and attentions for deepest and widest models GPU (ms). Aver-
ages across attention ignore Synthesizer due its extreme results.
Attention
IMDb Token
IMDb Byte
Listops
Doc Matching Wide
Average
Type
Deep
Wide
Deep
Wide
Deep
Wide
Deep
Deep
Wide
BigBird
154
63
154
59
210
128
232
115
188
91
Linear
52
63
37
76
52
22
61
28
Linformer
58
21
73
38
81
32
58
23
68
29
Local
58
26
68
41
86
40
63
34
69
35
Longformer
115
57
83
48
340
189
1202
540
435
209
Performer
47
17
60
35
95
51
49
19
63
31
Sinkhorn
92
33
112
50
121
49
106
52
108
46
Sparse
92
55
76
46
229
175
351
346
187
156
Synthesizer
25/64
34
5927
6001
325
199
338
293
7939
1632
Transformer
88
50
75
47
208
170
289
289
165
139
Average
76
34
40
145
87
240
144
144
76
E SMALL MODEL ABLATION STUDY
In order to verify the validity of going wide and going shallow, we run 'small' vanilla attention Transformer models on all 4 tasks. These models use the same number of heads per layer as the deepest models but have only layer. Therefore 8 heads on both IMDb tasks and Listops, and heads on the document matching task. These results are given in table TT along with repeats of the averages from the deepest and widest Transformer models. As performs worse than at least the widest one, though surprisingly beating the deep model on the Listops task. On the IMDb and document matching tasks it consistently performs 1.5%-3% worse than the others, as expected.
see the small model always
Table 1: Test accuracy for each task for the small vanilla Transformer model compared to its fully wide and fully deep counterparts.
Task
Deepest
Widest
Small
IMDb Token Level
85.8
85.5
84.0
IMDb Byte Level
62.6
62.4
59.6
Listops
35.7
37.2
36.9
Document Matching
64.0
64.4
62.4
14

---

Under review as a conference paper at ICLR 2023
3
8
10
10
12
12
13
13
14
14
15
15
16
17
17
19
19
20
21
21
22
22
23
24
24
26
26
28
28
29
29
30
31
31
33
33
34
34
35
36
38
37
38
38
40
40
41
41
42
42
43
43
44
44
45
45
47
47
This
movie
temriee
whiltt
loved
the main
the plot Aät
Other
movies
from
director
been terrible
however
loved this because
the main character
The plot was attle sily
(a) Prediction: strongly negative
(b) Prediction: wcakly positive
Figure 3: The attention weights across heads with predicted classification for the single layer
IMDb token level text classification Transformer model on two unseen and similar examples.
15


Reviews:
[{"review_id": "wHNFL6eN37L", "reviewer": "Reviewer_Kk5h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes modifying the standard transformer recipe by using more attention heads instead of using layers. Their model has less parameters overall since they keep the feed forward network constant. Despite that, their model demonstrates better performance on a wide range of classification tasks compared to deeper counterparts.", "summary_of_the_review": "The paper presents an intriguing idea that is worth exploring further, but ultimately I do not feel the that it is novel enough without the further exploration.", "strength_and_weaknesses": "The paper convincingly lays outs the advantages of their approach. Better classification performance, better latency, and interpretability. The tasks they used are standard benchmarks. Mixed attention is also interesting.\n\nWeaknesses:\n- They have restricted themselves to classification tasks. What about language modeling or translation for example?\n- They haven't convincingly explained away why models like PaLM would choose to scale to 118 layers and GPT-3 has 96 layers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lMk8lg6Kky", "reviewer": "Reviewer_MwhQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work empirically studies decreasing the number of layers in transformers while increasing the number of attention heads. On four classification problems and across many attention variants, this work found that a transformer with a single layer but more attention heads (such that the total number of attention heads remains constant) has on par or better accuracy than multi-layer transformers. This paper also argues that the attention matrices in a single-layer transformer are easier to interpret. Besides, results on image classification show that deep transformers work better there.", "summary_of_the_review": "I'm not recommending the acceptance of this paper because to me this work only shows that single-layer transformers work well for simple classification problems. I would be convinced the other way if the authors can show similar trends on more complex tasks such as sequence generation.", "strength_and_weaknesses": "Strengths:\n1. Surprising results that a single-layer transformer works as good as multi-layer transformers on sequence classification problems including listops which was constructed to test reasoning and presumably requires deep models.\n\nWeaknesses:\n1. I think the results do not support the title that wide attention is the way forward for transformers, since experiments are run on classification type problems, and it's not clear whether a single-layer transformer works on other types of problems such as text generation.\n2. While having fewer parameters is considered a plus, it is also possible that single-layer transformers overfit less due to having fewer parameters. What if you use really large datasets and more complex tasks such that model capacity becomes crucial? To convince readers that it's not the case, an experiment where the number of parameters of deep transformers is reduced to the same as single-layer transformers would be nice.\n3. In my opinion a fundamental limitation of single-layer models is that there is a single feedforward layer, which might lack the capacity to learn more complex tasks. A counter argument might be that we can increase the hidden dimension of the feedforward layer to increase capacity, but that comes back to general neural architecture search, and a really convincing experiment would be to run these experiments on more complex types of problems such as sequence generation.\n4. Regarding the interpretability argument, I'm not fully convinced it's more interpretable looking at Figure 3. It seems that we can pick one or two attention heads out of 48 heads that are interpretable, but I think we can do something similar in deep transformers.\n\nTypos:\n1. page 6: eg. the original ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C7c2VVmGu_", "reviewer": "Reviewer_xaMf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates deep vs. wide (in terms of the number of attention heads) models with a variety of recent transformer variants. It argues that, compared to their deep and thin counterparts, shallow and wide architectures (1) achieve the same or better performance, (2) has fewer parameters, and (3) run faster. The experiments on four text classification datasets provide evidence supporting the claim, while those on image classification yield negative results.\n\n\n", "summary_of_the_review": "The paper has serious flaws in its claims and is not ready for publication. ", "strength_and_weaknesses": "Strength:\n- Straightforward idea.\n- Writing is reasonably clear.\n- It’s great that the paper considers CPUs in addition to GPUs in efficiency comparisons.\n\nWeaknesses:\n- The paper overclaims: it opens with a general claim that going wide is better than going deep on a “variety of NLP tasks,” while the experiment is on three toy text encoding datasets. \n- The paper fails to acknowledge existing findings. The wider vs. deeper debate has been there for perhaps two decades, and recent results suggest that going deeper leads to better generalization. The paper argues for the opposite without discussing any existing work.\n\nFurther details and comments:\n- The wider vs. deeper debate has been there for a while. To convince the community that going wider is better, the paper needs to do a much better job. I suggest starting by covering a diverse set of tasks, e.g., language modeling, text generation, QA; and then considering larger-scale LM/MLM pretraining. Having a more comprehensive evaluation of the transformer model is more relevant than covering all the X-formers on toy datasets.\n- Given the existing work, my prior is that it is challenging (if at all possible) to match deep transformers’ performance with wide and shallow models on most tasks. If this is the case, I’d encourage the authors to explore the efficiency direction: can wider models achieve a better tradeoff between accuracy and efficiency? \n- I don’t get the 5.3 arguments on interpretability. Is there anything you can do with a wide transformer that cannot be done with a deep one?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1SgUOYkKiR", "reviewer": "Reviewer_RFvg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper argues that for transformers going deeper isn’t always better, and they show this by proving that wide single layer transformers sometimes outperform deeper transformers in 4 NLP classification tasks, with no pre-training.\n\nThe paper reports results on an interesting dimension: for different numbers of attention heads per layer, keeping the total number of heads and other hyper-parameters constant. They run 9 different transformer variations (e.g. BigBird, Longformer, ...), and use different input lengths between 500 and 4000.\n\nThe result is that one-layer transformers seem to perform the best on avg by about 0.3%. One-layer transformers are also smaller in terms of number of parameters 1.4x and faster to run (on CPU 3.1x times faster, on GPU 1,9x).\n\nThe authors also claim that one-layer transformers are also more interpretable showing some examples, but this is a difficult claim to make objectively.\n", "summary_of_the_review": "The lack of results for pretraining really weakens the paper, since virtually all current interesting results in NLP are based on massive pretraining.\n\nGiven that the authors claim improvements in speed and footprint, they could try making this contribution stronger by comparing against prior work optimizing transformers for efficiency.", "strength_and_weaknesses": "Strengths:\n  * The contains a pretty thorough and principled comparison between deep and wide transformers.\n  * The paper does suggest that shallow transformers might be useful as distilled production models, for example if run on CPU. However they don't have experiments in this direction.\n\nWeakness:\n  * The study only considers the case of no pre-training. This makes their exploration much easier and less expensive, but their conclusions might not hold at the scales that are currently widely used. In fact the paper cites Xue et al (2022), saying that they come to the conclusion that deep transformers are better than wide in pre-training, although this is for images.\n  * The datasets used are only for classification. It would be very nice to see some generation or translation results as well.\n  * The interpretability claim is anecdotal and it is not compared against strong baselines for model interpretability in deep models.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject