## 3 Materials and Methods
### SimMIM  
Transformers have emerged as state of the art for a wide range of vision-related tasks, primarily due to their exceptional capability in feature extraction and representation learning [37]. This progress inspired our exploration into their application for defect detection in manufacturing processes.  
Furthermore, with the aim of maximizing the capabilities of the Transformers, the selection of a pretraining approach is critical [47]. Considering the need for defect detection models to accurately detect features across multiple resolutions, we selected the SimMIM approach [44] for the pretraining phase on X-ray manufacturing images. This choice was informed by the compatibility of SimMIM with multi-scale Vision Transformers, which is essential for effective feature discernment at various scales. Adopting the masked image modeling approach, SimMIM aims to acquire visual representations by reconstructing randomly masked image patches. This approach allows for a deeper understanding of the underlying visual context.  
The SimMIM approach incorporates an encoder-decoder strategy. The encoder is responsible for extracting latent feature representations from the unmasked sections of image patches. The decoder reconstructs the pixel values of masked patches. \(l_{1}\) loss is used in order to train this model. This process involves down-sampled-resolution feature maps from the encoder being mapped back to original-resolution feature maps through a \(1\times 1\) convolution layer. The \(l_{1}\) loss function is then applied specifically to the masked pixels using the following formula:  
\[\mathcal{L}=\frac{1}{\Omega(x_{M})}\|\mathbf{y}_{M}-\mathbf{x}_{M}\|_{1} \tag{3}\]  
where \(x\) and \(y\) represent the input image and predicted values, respectively, with \(M\) indicating masked pixels and \(\Omega\) the count of images. Figure 3 provides a schematic representation of the model architecture.  
The key aspect of SimMIM is its efficient reconstruction process, which demonstrates that the lightweight prediction head can perform comparably, if not better, than more complex reconstruction decoders [44]. This efficiency not only maintains accuracy but also significantly reduces pretraining time and computational resource demands.  
The masking strategy in SimMIM, crucial for maximizing its effectiveness, involves applying random masking to the image. SimMIM demonstrates that either larger patch sizes or higher masking ratios lead to improved performance in downstream tasks [44]. This is attributed to the reduced correlation between unmasked and masked pixels when larger image areas are masked, compelling the model to learn more semantic information instead of replicating the closest highly correlated pixels. For instance, in ImageNet classification, SimMIM achieves optimal results with a 32-pixel masking patch size and a 60% masking ratio as it forces the model to infer more complex and less obvious image features.  
A notable advantage of the SimMIM strategy over approaches like SimSiam is that it does not require specifically chosen data augmentation for each problem. This characteristic simplifies the application of the method across various scenarios, reducing the need for customized augmentation strategies and thereby optimizing the process of model training and adaptation to different tasks.