# Vision Transformer in Industrial Visual Inspection
## 2 Related Work
### Recent Visual Inspection Examples Utilizing Deep Learning  
Since CNN's have made their entrance to CV with AlexNets first place in the 2012 Image Net Large Scale Visual Recognition Challenge (ILSVRC) [28], they have advanced the state-of-the art in closely related areas, such as automated visual inspection, by superseding traditional feature-based methods.  
[PERSON] et al. [29] proposed a CNN-based method for the inspection of 23 categories of train-bogey-parts, such as wheels, brakes, springs, bearings, and screws. They utilize a F-RCNN [30] architecture with ResNet-101 [31] backbone enhanced by a FPN [32] and an  
Figure 4: Swin tiny transformer architecture [19].  
Figure 3: Visualization of the concept of patches and windows in the Swin transformer.  
additional RNN submodel. This submodel is establishes a new form of gated recurrent unit (GRU) [33] to learn the strong structural correlation in technical systems. The model is evaluated on a dataset of 1130 images, containing 23 annotation classes in three sizes, where it achieves a mAP of 87.18%. In addition to this, the model is assessed on the benchmark datasets Pascal VOC 2007 and 2012 [34] and attains mAPs of 79.83% and 75.24%, respectively.  
[PERSON] et al. [35] developed a method for automatic defect detection on asphalt pavements utilizing a Yolo V3 [36] architecture. They trained their model to locate and classify four damage types: longitudinal, transverse, and alligator cracks, as well as potholes with a dataset of approximately 5000 color images and achieved an _F1-Score_ of 60%.  
Very recently, one of the first VI papers applying a transformer model to crack segmentation in stone and concrete surfaces has been published by [PERSON] et al. [13]. Inspired by SegNet, their architecture follows an encoder-decoder structure [37] with the convolutional layers in encoder and decoder replaced by self-attention blocks. They also proposed a scaling attention block to execute the processing in the feature aggregation path. They achieved new performance records on the public datasets Crack Tree 260 [38], CrackLS 315 [39], and Stone 331 [40], improving the previous results by 0.6%p to 2.1%p. [PERSON] et al. [14] tackled the detection of major, safety critical, components of railway tracks with a transformer model. Their models employs a ResNet-50 or Darknet-53 backbone and transformer detection head, with two parallel feed forward networks, one for class and one bounding box prediction, to generate the output. They achieve a mAP of 61.9% with their best model.  
Model Selection Based on Benchmarks and Recent Applications  
We selected CNN and transformer models based on their benchmark results on the COCO dataset on the one hand and their occurrence in recent VI research papers on the other hand [29, 35, 41, 42, 43]. The models are DETR, DDETR and Swin, Yolo V3, F-RCNN, and Retina Net. All models except Yolo V3 and Retina Net utilized a ResNet-50 backbone. While Yolo used its original DarkNet-53 backbone, Retina Net was employed with three different ones: ResNet-50 for comparison with the other models; ResNext-101 (Retina Next) as a representative of larger more recent CNN architectures with a similar number of parameters and performance to the transformer models [44]; Swin in its \"small\" version to have a detection head for the only transformer backbone in this study. To complete the list we would have liked to evaluate the model by [PERSON] et al. as well, but unfortunately it was not accessible.