# A Historical Survey of Advances in Transformer Architectures
## 3 Survey Results
### Early Transformer Implementations
#### 3.1.2 Further Progression  
In early 2020, [PERSON] [28] proposed an improvement to the transformer model, which involved variants of Gated Linear Units (GLUs) [59] being applied to the feedforward sublayers of the transformer model. These variations were implemented using different linear and non-linear activation functions in place of sigmoids, and the authors report an improvement in performance over the generally used ReLU activation function when evaluating on the SQuAD, GLUE, and SuperGlue [60] tasks. It was in April of 2020 that a key architecture in the form of the Lite Transformer was introduced by [PERSON] et al. [29]. The reasoning behind the introduction of this architecture was that the authors argued that transformers require an enormous amount of computation in order to achieve high performance and, therefore, they would not be suitable for mobile applications that are constrained by hardware and battery resources. Therefore, they proposed the Lite Transformer specifically to be deployed to perform NLP on mobile devices. They introduce Long-Short Range Attention (LSRA), where one group of heads specialize in local context modeling using convolution while the other specializes in long-distance relationship modeling using attention. They report that this approach shows improvement over the vanilla transformer in three established language tasks, namely, machine translation, abstractive summarization, and language modeling. The Lite Transformer block can be seen in Figure 6.  
Using this approach, the proposed model reduces the computation of the transformer base model by 2.5\(\times\) with only a 0.3 BLEU score degradation. Furthermore, the authors report implementing pruning and quantization processes to compress the model size by 18.2\(\times\). [PERSON] et al. [30] propose a ground-breaking object detecting transformer named DETR that views object detection as a direct set prediction problem. The main components of the model are a set-based loss that forces predictions via a bipartite matching and a transformer encoder and decoder. The overall architecture of the model is illustrated in the following Figure 7.  
The CNN is used to extract a compact feature representation of the input image by generating a low-resolution activation map. The transformer's encoder and decoder follow the model architecture of [PERSON] et al. [1]. The decoder output encodings are decoded into box coordinates and class labels by the feedforward network. The object detection set prediction loss produces a bipartite matching between the predicted and the ground truth objects and then optimizes the object-specific losses. This model is on par with state-of-the-art Faster R-CNN baseline on the famous COCO object detection dataset. The Faster R-CNN was a model proposed by [PERSON] et al. which used a Region Proposal Network to generated region proposals which were then used by a Fast R-CNN for detection [61]. Around mid-2020, [PERSON] et al. [31] proposed a work which improved on the state-of-the-art NLP transformer model by proposing their improved GPT-3 model. The authors scale-up the model by training it with 175 billion parameters which results in a model which can perform a variety of tasks without requiring task-specific gradient updates or fine-tuning, unlike the previous generations of the model. The other variation from the architecture of GPT-2 is that of the use of alternating dense and locally banded sparse attention patterns in the layers of the transformer. The model is able to perform well and even achieve SOTA results on famous NLP dataset tasks with few-shot demonstrations which are specified purely via text interactions with the model. Figure 6: The Lite Transformer block. Figure 7: The DETR’s architecture. [PERSON] et al. [32] introduced the Vision Transformer (ViT) in late 2020, which caused a shift in the research field.