# Vision Transformer in Industrial Visual Inspection
## 5 Conclusions and Outlook  
In this paper, we evaluated the capability of state of the art vision transformer models for industrial VI in three representative low data use cases. These use cases are from the context of damage assessment of freight cars after train decomposition at railroad yards. The three use cases are: damage detection in sheet metal flooring, damage detection in wooden load spaces, and character localization and recognition as a foundation for matching wagons with an internal database. The last of the three marks a complexity level beyond previous publications employing vision transformers in the context of AVI. We trained three different transformer-based architectures, DETR, DDETR, and (Retina-)Swin, of which the last two to our knowledge have not been applied to VI before. For comparison we also trained four established CNN-based architectures--F-RCNN, Retina Net, Yolo V3, and RetinaNext--that represent the current SOTA in VI as baselines. Each use case was evaluated on downscaled, full images and patches generated by a sliding window method, due to the large dimensions of the input images. The DDETR model achieves the best results in the detection of holes in sheet metal and the recognition of characters utilizing a sliding window approach with a \(F1\)-\(Score_{adjusted}\) of 92.7% and 77.9%. It seems to be very good at the differentiation between many similarly shaped objects. The Swin model with a Retina detection head delivers the best performance in the windowed version of the wooden load space use case with a \(F1\)-\(Score_{adjusted}\) of 92.7%. It appears its capability to exchange information between the attention windows is especially useful when damages spread across wide areas of the images and have a high level of color variety. Considering the full image versions of the use cases the transformer models cannot surpass the CNN models, because the high memory requirements resulting from the large images limit their potential. After our studies, we can give the clear recommendation to apply vision transformer models to industrial visual inspection scenarios with typical input image sizes as they:  
* Perform better than typically used CNN models;
* Show no significant difference in convergence speed compared to CNNs;
* Handle small datasets commonly utilized in industrial VI well. Yet there is still room for improvement as the models were not modified to specifically fit the intricacies of the presented VI use cases to, e.g., better handle very large-scale input data, improve detection of small damages or become injected with prior knowledge of the typical hierarchical structure of the inspected technical systems. Based on our results it seems the combination of a Swin backbone with a DDETR detection head would achieve optimal VI results and should be evaluated in future work. Since the acquisition of labelled data is generally very expensive, other approaches are needed. Unsupervised learning offers promising concepts, such as masked patch reconstruction [52] and the soft-teacher approach [24], that need to be further explored. Solely supervised training also limits the achievable performance gains by scaling models up, which is why unsupervised pre-training played a key part in unlocking model dimensions and performances that GPT-3 exhibits in NLP, as a prime example model from the transformer family. This also enabled the concept of having one large general-purpose foundation model and use its outputs for zero- or few-shot learning of small task specific adapter models. The adaptation of this concept to VI could also prove highly valuable, as it would alleviate the labeling efforts and extremely shorten or even completely negate training time. Conceptualization, methodology, formal analysis, investigation, writing--original draft preparation N.H.; writing--review and editing, R.M. and T.M. ; project administration, funding acquisition, R.M. ; supervision, resources, T.M. All authors have read and agreed to the published version of the manuscript.