# A Historical Survey of Advances in Transformer Architectures
## 3 Survey Results  
### Early Transformer Implementations  
#### 3.1.1 Introductory Works  
Since the introduction of the aforementioned transformer model in 2017, a vast array of works have aimed to build upon its novel architecture in order to optimize its performance for a variety of domains. Indeed, the work proposing the transformer model has been cited more than 90,500 times as of 29 September 2023, according to Google Scholar [21]. Amongthe thousands of consequential works, a few emerge as notable models which have consequently contributed to pushing the overall state-of-the-art techniques and have established themselves as standards in their fields. Figure 3 displays a timeline of these notable works arranged chronologically and coded according to the domain of implementation. In order to benchmark these works, a number of datasets have been utilized by the various works. A few of the commonly used datasets are BookCorpus [48], WMT 2014 [49], Wikipedia [50], C4 [22], ImageNet [51], and COCO [52]. An early work building upon the transformer model was that of [PERSON] et al. [19], which simply involved extending the self-attention mechanism of transformers to efficiently consider representations of the relative positions or distances between sequence elements. This is done by modeling the input as a labeled, fully connected graph with the edges between input elements \(x_{i}\) and \(x_{j}\) represented by vectors \(a_{ij}^{V},a_{ij}^{K}\in R^{d_{e}}\). A modification is then made to the transformer equation wherein edge information is then propagated to the sublayer output as seen in Equation (6). \[z_{i}=\sum_{j=1}^{n}\alpha_{ij}\Big{(}x_{j}W^{V}+a_{ij}^{V}\Big{)} \tag{6}\]  
Using these improved embeddings, the authors were able to report improvements in both the EN-DE and EN-FR tasks over the vanilla transformer architecture. Another early and majorly consequential work was that of [PERSON] et al. [23] who proposed the famous Generative Pre-Training (GPT) model. The base model used for the work was the transformer architecture as it allowed the authors to capture long-range linguistic structures. The idea proposed by the authors was one where the model can perform more optimally for small amounts of labeled text data when it is generatively trained in an unsupervised manner on a large unlabeled text corpus consisting of diverse samples and then discriminatively fine-tuned on the specific task at hand. They do this by utilizing a multi-layer transformer-decoder [53] architecture which applies a multi  
Figure 3: The timeline of the state-of-the-art transformer models [1, 17, 19, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]. headed self-attention operation over the input context tokens followed by position-wise feedforward layers to produce an output distribution over the target tokens. These trained weights can then be used with an auxiliary objective for classification tasks. The architecture used by the model can be seen in Figure 4.  
A similar approach to that of GPT was seen by the Bidirectional Encoder Representations from Transformers (BERT) model proposed by [PERSON] et al. (2017), where unlabeled data is used to pre-train the transformer model in an unsupervised fashion before the model is fine-tuned using representative samples from the problem at hand. The major improvement proposed by the authors is the use of bidirectional encoder representations unlike previous solutions, which involved unidirectional models being used in the learning process such as GPT using a left-to-right architecture where each token in the self-attention layer was only able to attend to previous tokens. The BERT model achieves bidirectional learning by using a masked language model (MLM) pre-training objective which the authors adapted from the Cloze task ([PERSON] et al., 2018). This model randomly masks some of the tokens from the input with the objective of predicting the original vocabulary ID of the masked work based on the context.