###### Abstract  
Steel surface defect detection is challenging because it contains various atypical defects. Many studies have attempted to detect metal surface defects using deep learning and had success in applying deep learning. Despite many previous studies to solve the steel surface defect detection, it remains a difficult problem. To resolve the atypical defects problem, we introduce a hierarchical approach for the classification and detection of defects on the steel surface. The proposed approach has a hierarchical structure of the binary classifier at the first stage and the object detection and semantic segmentation algorithms at the second stage. It shows 98.6% accuracy in scratch and other types of defect classification and 77.12% mean average precision (mAP) in defect detection using the Northeastern University (NEU) surface defect detection dataset. A comparative analysis with the previous studies shows that the proposed approach achieves excellent results on the NEU dataset. defect detection; deep learning; steel defect detection; RetinaNet model; UNet +
Footnote †: journal: Applied Sciences  
0000-0002-4000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0000-00000-00000-000  
introduction of machine vision, it was evaluated as a groundbreaking technology that could make accurate and consistent decisions, but it gradually showed limitations. As the industry is segmented, more sophisticated technologies are needed. For example, parts and products have been miniaturized and more diversified throughout the manufacturing industry, such as machinery, automobiles, semiconductors, and electronic products. The demand for rapid and accurate technology development for precision (may be precise) parts has also increased rapidly. In particular, as atypical defects are difficult to distinguish by general definition, there is an active movement to introduce artificial intelligence (AI) technology, such as machine learning (ML) and deep learning to automatic vision inspection. Among AI technologies, the deep learning-based vision inspection system, in particular, guarantees high accuracy and efficiency in areas that are difficult to inspect by conventional methods [20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34]. On the other hand, with the development of the automobile, aerospace, and ship-building industries, the importance of the steel industry is also increasing. As a result, the requirements for automated technology for steel surface defect detection to control the quality of steel products are increasing. Steel surface defects contain various atypical defects, such as pitted surface, inclusion, patches, rolled-in scale, crazing and scratches; thus, it remains a difficult problem. Many previous studies have tried to solve the steel surface defect detection problem by using only one ML or deep learning-based methods, such as object detection and image segmentation. Despite the introduction of deep learning technology, it is difficult to detect all of the aforementioned metal surface defects accurately [35; 36; 37]. In this study, we propose a hierarchical approach for steel surface defect detection, which has a hierarchical structure of the binary classifier at the first stage and the object detection and image segmentation algorithms at the second stage. In general, an object can be defined based on the characteristics such as closed boundary, different appearance and uniqueness. Most defective types of steel surfaces, such as pitted surfaces, inclusion, patches, rolled-in scale, and crazing, can be considered objects. However, scratches are very thin and elongated in shape, so they are not included in the general category of objects. We defined scratch detection as an image segmentation problem and other types of defect detection as an object detection problem. In the first stage, the binary classifier classifies whether the input image is an image containing a scratch defect or an image containing other types of defects.