## 2 Related Work  
### Manufacturing Defect Detection  
Numerous efforts have been made over the years to automate the task of defect detection in manufacturing using computer vision methods. Currently, manufacturing defect detection is a well-established problem in the field of computer vision and has been widely applied in numerous industrial quality control processes [7; 8; 9]. Initial attempts in defect detection used image comparison [10] and Fourier Transform [11] to identify defects, followed by statistical techniques for feature extraction [8; 12]. These features were then classified using machine learning to distinguish between defective and non-defective items. However, this manual feature extraction was product specific and not universally applicable, limiting the transferability of learned knowledge. The emergence of deep-learning-based approaches has improved the accuracy in manufacturing quality control [3; 4; 5; 13; 14; 15]. Their complex structures are able to retain and automatically learn the information contained in the image, more effectively facilitating the image processing compared to previous techniques. Deep-learning-based models are built in an end-to-end manner so handcrafting processes are not required to extract discriminant features [3; 16]. In fact, the feature extraction process is carried out automatically from raw images, followed by a classifier head that learns the boundary between defective and non-defective features. In recent years, numerous studies have employed deep Convolutional Neural Networks (CNNs) for image-level classification, effectively differentiating between defective and non-defective manufacturing images [1; 8]. These works focused on refining the feature extraction process, considering it crucial for developing an effective defect detector. [PERSON] et al. [1] compared different feature extraction backbones on surface images with sandblasting defects. Additionally, [PERSON] et al. [17] used a self-attention module to extract and classify features from small defects. However, the aim of defect detection is not only to classify a product at image level but also to locate the defects throughout the image when the product is defective. Therefore, research shifted towards the application of object detection models to incorporate location information [3; 13; 14; 15; 18]. [PERSON] et al. [3] applied Faster R-CNN [19] to a GDXray benchmark dataset [20], obtaining satisfactory results. Soon after, they improved their results by applying a Masked R-CNN [4] pretrained on the COCO dataset [21]. [PERSON] et al. [13] proposed several improvements to the Faster R-CNN for defect detection in automobile manufacturing products, including the Feature Pyramid Network [22] (FPN), RoIAlign [23], and data augmentation techniques. Furthermore, [PERSON] et al. [17] were the first to integrate self-attention mechanisms with CNNs, enhancing image defect detection by extracting subtle features from general features. The main challenge facing deep-learning-based approaches is their reliance on a large volume of images to achieve reliable performance [24]. Indeed, applying transfer learning with pretrained models offers a suitable initial step [4]. However, there is a notable disparity between X-ray images and those typically used in ImageNet pretraining; X-ray images are grayscale and depict specific manufacturing parts, contrasting with the RGB images of varied everyday objects and scenes from ImageNet. This discrepancy leads to a significant domain shift. Although there have been significant advancements in applying defect localization to manufacturing defect detection, a notable gap exists in the use of self-supervised learning on X-ray manufacturing images. To the best of our knowledge, this specific approach remains unexplored, representing a promising direction for the enhancement of the accuracy and efficiency of defect detection in industrial settings.