# Vision Transformer in Industrial Visual Inspection
## 2 Related Work
### Transformer Models in CV  
The impressive results of these models in NLP has drawn the attention of CV researchers to adapt them to their domain. To be able to feed image data into transformers it has to be transformed into sequences. Two different lines of pre-processing have been established depending on the role of the transformer model in the overall architecture: for backbone networks such as the vision transformer (ViT) [18] or the shifted windows transformer (Swin) [19] the input image is divided into patches. These are flattened and then each of them serves as one input sequence/token, like in NLP. Detection heads such as the detection transformer (DETR) [20] and the dynamic head transformer (DyHead) [21] process the flattened features extracted by an upstream CNN. In the following, we will present how DETR, deformable DETR (DDETR) [22], as well as Swin are structured and how they adapt ideas from NLP to reduce the complexity of the self-attention computation. The transformer model in DETR is structurally almost identical to the original model introduced by [PERSON] et al. The main differences are: it utilizes features extracted by a CNN backbone as input as visualized in Figure 1. These feature maps are encoded into a sequence with reduced overall dimension using a \(1\times 1\) convolutional layer. Static two-dimensional positional encodings are added to this sequence, as well as the input to all other encoder and decoder layers instead of only the first one. The decoder's input is not a forward masked version of the encoder input, instead so-called object queries are used. The model features two output branches, one for class- and one for bounding box prediction. Additionally, the outputs do not require any non-maximum suppression (NMS) post-processing, because the predictions are optimized to find the best bi-partite matching with the ground truth employing the Hungarian matching algorithm [23]. Although, the performance of this model is comparable to F-RCNN's with the same backbone, it has two problems: slow convergence speed, which was reported to be 48 days on one Nvidia V100 GPU [24], when trained from scratch; and high computational cost due to the use of full (self-)attention. DDETR improves on DETR by proposing deformable (self-)attention. It is based on the observation that the attention matrices are very sparse usually and the results are dominated by a small number of keys for each query. Therefore, the number of keys that is considered by a query is limited to \(k\), to reduce the number of required operations. The top center of Figure 2 shows the additional linear layer that is established to learn which \(k\) keys, around the reference point \(p_{q}\), associated with the given query features \(z_{q}\), to use for the attention computation. This concept, inspired by deformable convolutions [25], reduces the training time to about 14 V100 days, while also improving the performance [22]. In addition to deformable (self-)attention, [PERSON] et al. propose three additional improvements: extension of deformable attention to multiple feature map scales, iterative bounding box refinement, where each decoder layers further refines the predictions of the previous one and a two-stage version of the model, with an additional encoder part of DDETR acting as a proposal generator for a full DDETR head. Figure 1: DETR architecture [20]. Figure 2: Illustration of the deformable (self-)attention module [22]. Opposed to DETR and DDETR the Swin transformer operates on embedded, flattened patches of the input image and can function as the backbone of detection architectures like faster R-CNN (F-RCNN), Retina Net, or Yolo. To reduce the number of operations to compute self-attention, it is not evaluated over all patches, but only for non-overlapping windows containing \(M\times M\) patches (Figure 3). The window positions are not static, but shifted every layer to enable cross-window information flow, which partly preserves the global modeling power of self-attention.