# A Historical Survey of Advances in Transformer Architectures
## 4 Discussion  
### Historical Insight  
Table 1 summarizes the historical works discussed in the previous section. The works are color-coded in the timeline, wherein the works targeted towards text and NLP tasks are color-coded in blue and the works targeted at image-related tasks are color-coded in orange.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Name of Paper** & **Author** & **Date** & **Proposed** & **Datasets** & **Models Benchmarked** & **Results** & **No. of** **Citations** \\ \hline \multirow{4}{*}{Attention Is All You Need [1]} & \multirow{4}{*}{Vaswani} & \multirow{4}{*}{Jun 2017} & \multirow{4}{*}{Transformer} & \multirow{4}{*}{BoysLets, Deep-Att + PolNtx} & ByteNeU, DeepAtt + PolNtx, PosUnk, GNMT + RL, ConvS2S, Moe, and 41.8 BLEU for EN-DE & \multirow{4}{*}{90,568} \\  & & & & 2014 English-to-French & Ensemble, GNMT + RL & Ensemble, ConvS2S \\  & & & & & 2014 English-to-French & Ensemble & GNMT + RL \\  &  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  
The table above summarizes key information from the history of the studies discussed in the previous section. In addition to the name of the study, the author and the date, the table also outlines the approach presented as well as the datasets evaluated upon, the models benchmarked against, and the obtained results. Finally, the number of citations attained by the paper as of the writing of this paper are also listed in order to emphasize the importance of some of the presented studies.  
In general, it can be seen that a number of works have chosen to add or modify layers of the base transformer models, which has overall been seen to achieve good performance. Indeed, such an approach is seen in works such as those of [PERSON] et al. [19], [PERSON] et al. [25], [PERSON] et al. [27], and [PERSON] [28].  
Another common approach for NLP tasks which has been shown to work really well is to increase the size of the model to a very large number of parameters and to pre-train it in an unsupervised fashion on a large corpus of data. This has been seen in numerous state-of-the-art models such as the GPT [23], BERT [17], GPT-2 [90], RoBERTa [26], T5 [22], and the GPT-3 [31] models, in the work of [PERSON] et al. [37], and in the OPT model [39].  
Yet another form of approaches involves the addition or modification of the loss functions associated with the transformer model. Such an approach was seen in the case of the work performed by [PERSON] et al. [30].  
When it comes to images, the general procedure followed by the previous studies was to split images into patches and apply position embeddings on these patches, much like what is done for texts. This was indeed the process followed by the Vision Transformer (ViT) [32]. Other vision models implemented varied decoders such as the work proposed by [PERSON] et al. [33]. Similarly, studies such as that by [PERSON] et al. [34] make use of multiple pairs of heads and tails corresponding to different low-level vision tasks. The ViT-G model proposed by [PERSON] et al. [38] followed a procedure where the class token was removed and the non-linear projection before the final layer was removed.