# A Historical Survey of Advances in Transformer Architectures
## 4 Discussion
### Application-Based Implementations
#### 4.2.3 Miscellaneous Applications  
Through Table 4 below it is well illustrated that the contributions towards transformer models are not just limited to the domain of NLP and images, but they have also been  
\begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline  
recently used in audio and time series domains. Here, too, it is difficult to do a concrete quantitative analysis as the specific application domains of the works summarized above are all different. An interesting work to note is that of [PERSON] et al. [77], which merges NLP analysis within the audio domain and is quite successful in outperforming the results of the traditional LSTM model that is usually used for such an application, with a best score of 52.1 for the BLUE-1 dataset. [PERSON] et al. [76] achieve a WER score of 10.9 on the eval92 subset of the Wall Street Journal dataset, and [PERSON] et al. [78] achieve their best results on the Speech commands v2 dataset with an accuracy of 98.11% without adding additional audio data while training. The second half of the table demonstrates different areas in the domain of time series using transformers. The domains illustrated are those of the Time Series Classification by [PERSON] et al. [79], who were able to beat the state-of-the-art results on 7 out of 13 competitive datasets, those of the Time Series forecasting proposed by [PERSON] et al. [80], who achieved SOTA results in all 6 datasets, and the Time Series Anomaly Detection proposed by [PERSON] et al. [81], who also beat the SOTA results in their domain on 7 out of 10 competitive datasets.  
\begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**Name of Paper** & **Author** & **Date** & **Proposed Model** & **Datasets** & **Benchmarked Against** & **Results** \\ \hline Speech-Transformer: & & & & & & \\ A No-Recurrence & & & & & & \\ Sequence-To-Sequence Model For Speech Recognition [76] & [PERSON] et al. & 2018 & Audio Transformer & Wall Street Journal dataset & CTC, seq2 seq, seq2 seq + deep convolutional, seq2 seq + Unigram LS & WER 10.9 on eval92 \\ \hline A Transformer-based Audio-Captioning Model with Keyword Estimation [77] & [PERSON] et al. & 2020 & Audio-Conditioning Transformer (TRACKE) & [PERSON] dataset & Baseline LSTM, Transformer from same challenge & Beats in BLUE-1 with BLUE-2-30.9, BLUE-3-18.8, BLUE-4-10.8, CIDF-25.8, METEOR-14.9, ROGUE-1-34.5, SPIDE-9.7 \\ \hline AST: Audio Transformer [78] & [PERSON] et al. & 2021 & Audio Transformer & Converted pre-trained ViT to AST, used DeiT weights & Audio ResNet, trained GAN, used DeiT weights & AudioSet-AST (Ensemble-M) - Balanced PANN, PSLA single, PSLA Ensemble-S, PSLA Ensemble-S, PSLA Ensemble-M & AnAP-0.485 \\ \hline Gated Transformer [78] & [PERSON] et al. & 2021 & Time series classification Transformer & MLP, FCN, ResNet, Encoder, MCNN, tickersPurch, Time-CNN, TWIESN & Best SOTA results in 7/13 datasets, with best scores of 100\% for CMUsubject1, Time-CNN, TWIESN & Best SOTA results in 7/13 datasets, with best scores of 100\% for CMUsubject1, Time-CNN, TWIESN & Best score is AUC of 0.9994 and F1 of 0.9694 for the UCR dataset \\ \hline \hline \end{tabular}
\end{table}
Table 4: A summary of the transformer-related works in the audio and time series domains.