# Vision Transformer in Industrial Visual Inspection
## 2 Related Work
### Transformer Models in NLP  
Transformers are sequence to sequence models that were proposed for the first time by [PERSON] et al. in 2017 [7] in the context of natural language processing (NLP). They demonstrated exemplary performance in a broad range of language tasks, such as text classification, machine translation, and question answering [11], improving on previous records set by recurrent neural networks (RNN). This became possible through the combination of several, new or already existing, concepts. Word embeddings enable them to capture the semantic relations between words. Multi-head self-attention refines these raw embeddings in the context of specific inputs, as well as parallelizes the computation, by working on all input tokens at once, in contrast to the sequential process of RNNs. Lastly, they also utilize residual connections that increase the availability of information from shallower layers across the whole network and improve model convergence capability and speed by increasing gradient flow during backpropagation. The parallelization also makes them extremely scalable, which resulted in model size growth in popular models from 213 million parameters of the original transformer model to 340 million parameters of BERT large [8], 1.5 billion parameters of GPT-2 [15], and 175 billion parameters of GPT-3 [9]. This scaling resulted in an ever-growing performance and ability to generalize, but was not achieved without any drawbacks.  
Despite their great capabilities, transformers face similar and, in some cases, amplified versions of the problems that RNNs had as well. First, like with most state-of-the-art deep learning models, large datasets are required for training. Second, they show slower convergence than, e.g., RNN or CNN models, because their structure does not inject them with prior knowledge their developers deem useful for the task at hand (inductive bias). This results in a larger solution space that is searched during training and, therefore, requires more time to converge. Third, extremely powerful hardware, regarding memory and floating-point operations per second (FLOPS), is required to train them due to the quadratic scaling of self-attention with the input length.  
There have been several efforts to overcome these problems, mainly focused on reducing the complexity of the self-attention computation. [PERSON] et al. [16] introduced locality sensitive hashing attention to reduce the complexity of self-attention to \(O(n\log(n))\) by approximation. Additionally, they proposed reversible layers, which enable the storage of only the final activations for backpropagation, and, therefore, reduce the memory footprint at cost of additional calculations. [PERSON] et al. [17] improve on this with their Linformer, bringing the complexity of the approximation down to \(O(n)\) in time and memory space. This is possible because the result of the softmax operation in self-attention, the context mapping matrix \(\mathbf{P}\), is of low rank (Equation (1)). Therefore, the self-attention result can still be well approximated with a lower dimensionality linear projection of the key and value matrix (\(\mathbf{Q}\), \(\mathbf{V}\)).  
\[\text{Self-Attention}=\underbrace{\text{softmax}\bigg{(}\frac{QW^{Q}KW^{K}}{ \sqrt{d_{k}}}\bigg{)}}_{\mathbf{P}}VW^{V} \tag{1}\]