—Visual steel surface defect detection is an essential step in steel sheet manufacturing. Several machine learning-based automated visual inspection (A VI) methods have been studied in recent years. However, most steel manufacturing industries still use manual visual inspection due to training time and inaccura- cies involved with A VI methods. Automatic steel defect detection methods could be useful in less expensive and faster quality control and feedback. But preparing the annotated training data for segmentation and classiﬁcation could be a costly process. In this work, we propose to use the Transfer Learning-based U-Net (TLU-Net) framework for steel surface defect detection. We use a U-Net architecture as the base and explore two kinds of encoders: ResNet and DenseNet. We compare these nets’ performance using random initialization and the pre-trained networks trained using the ImageNet data set. The experiments are performed using Severstal data. The results demonstrate that the transfer learning performs 5% (absolute) better than that of the random initialization in defect classiﬁcation. We found that the transfer learning performs 26% (relative) better than that of the random initialization in defect segmentation. We also found the gain of transfer learning increases as the training data decreases, and the convergence rate with transfer learning is better than that of the random initialization. Index Terms —Automated visual inspection (A VI), DenseNet, ResNet, Surface defect detection, Transfer learning, U-Net I. I NTRODUCTION Steel is one of humanity’s most important building mate- rials. Defect inspection is a critical step of quality control in the steel plates. This step mainly involves capturing images of the steel surface using an industrial camera followed by recognizing, localizing, and classifying the defect, which helps rectify the defect’s cause. Typically, this process is performed manually, which is not reliable and time-consuming. Unreli- able quality control can cause a huge economic problem for manufacturers. Manual detection can be replaced or aided by the automatic classiﬁcation using computer vision methods. The general ﬂow of automatic visual inspection for quality control is shown in Fig. 1. There are two main steps involved in the defect inspection. The ﬁrst step is to classify the defect type from the images, and the second step is to identify the defect location in the image. There are various automatic methods in the literature to address one/both of these steps. Some of the early methods use a handcrafted feature to classify the defect type [1, 2, 3], and few methods ﬁnd the coarse defect locations. The main drawback of these methods is that Figure 1. A Generic Automatic Visual Inspection Outline the features need to be designed by an experts. The designed feature may not generalize to new type of defect. The recent advances in end-to-end deep learning (DL) methods overcame these hand-designed features. It learns to extracts the multi- scale features depending on the task using only the data and labels. The DL method has been shown to outperform the hand-designed features in various computer vision tasks [4]. There are various deep learning methods are used to perform defect classiﬁcation. [5] use the features extracted from the OverFeat, a variant of convolutional neural networks (CNN), to do the defect classiﬁcation. They have also shown that the ﬁxed features from the pre-trained network perform well on some defects and perform poorly on texture kind of defect. These authors have also proposed a structural visual inspection method based on faster region-based CNN (faster R-CNN) to ensure quasi-real-time simultaneous detection of multiple types of defects [6]. [7] proposed to detect weak scratches using deep CNN and skeleton extraction. They have shown that the their method is robust to background noise. [8] proposed a variant of you only look once (YOLO) network to detect surface defects of ﬂat steel in real-time. The CNN is used extensively for a different kind of defect classiﬁcation on different data sets [9, 10]. [11] use a transfer learning ap- Figure 2. Proposed architecture transfer learning method for the joint steel defect classiﬁcation and segmentation. The blue line indicate the skip connection and the orange dotted line indicate the initialization. proach for defect classiﬁcation. They have shown that transfer learning can help achieve a good classiﬁcation accuracy with fewer data samples. [12] use a patch-wise classiﬁcation to do both defect classiﬁcation and segmentation. There are various methods in the literature on defect lo- calization. [13] uses a classical CNN to perform steel defect detection. Authors have explored the effect of regularization and unsupervised pre-training. [14] uses a pre-trained ResNet to extract the multi-scale features, and the features from differ- ent scales are fused using a multilevel feature fusion network (MFN). The fused features and region proposal network are used to classify the defect type and predict the bounding box. The main drawback of the method is that localization is very coarse. [15] use an U-Net and residual U-Net architecture for the ﬁne segmentation of the steel defect. The method’s main drawback is that the networks are trained with random initialization and need a large amount of pixel-level annotation of the defects. The pixel-level annotation process can be very time consuming and expensive. [16] uses a SegNet based semantic segmentation for the steel defect detection. There are various unsupervised and reinforcement learning based methods for steel defect detection. The summary of various methods for defect classiﬁcation, and segmentation can be found in [17]. The authors discuss the taxonomy of defect detection including the statistical, spectral, model based and machine learning methods. In this work, we systematically study the transfer learning effectiveness for steel defect classiﬁcation and localization (SDCL). The transfer learning or domain adaption aims to reuse the feature learned in one domain to improve the learning in the other domain. This is a popular approach in cases where the annotated data is limited. The transfer learning has shown really good application in various tasks such a object detection [18], semantic segmentation [19, 20] etc. It is already shown that the transfer-learning from an arbitrary domain to anotherdomain may not be useful. Transfer learning is most effective when two domains are similar [20, 21]. Hence, it is important to study the effectiveness of transfer learning in the case of SDCL. We consider a baseline architecture of U-Net for steel defect segmentation. U-Net has demonstrated state of the art performance in various image segmentation tasks [22]. It uses an encoder-decoder architecture with skip connections. The encoder learns the images’ features at different scales, and the decoder uses these features to predict the segmentation masks. In this work, we explore two kinds of pre-trained encoder networks– ResNet and DenseNet networks. Both of these networks have been shown to perform well on various computer vision tasks. The networks are pre-trained on the ImageNet data set [23]. We use a linear classiﬁer using the bottleneck representation of U-Net to classify the defect. We ﬁne-tune both the encoder and decoder of the network using the Severstal dataset [24]. The experiments on Severstal data shows that the performance of both segmentation and classiﬁ- cation is better in case of the pre-trained network compared to the random initialization. It is found that performance gain by using the pre-trained networks is even higher if 50% of data is used for training. We also show that the convergence of transfer learning is faster compared to random initialization. II. P ROPOSED TRANSFER LEARNING BASED U-N ET The proposed architecture for joint steel defect segmentation and classiﬁcation is shown in Fig. 2. The architecture takes an input image of dimension HWand classiﬁes each pixel to be one or more type of the defect. It involves mainly four parts (1) The U-Net architecture, (2) the type of initialization (3) classiﬁcation and (4) objective function A. U-net architecture The U-Net is an encoder-decoder architecture with a skip connection. The encoder encodes the image using an encoder block and reduces the resolution using pooling. This helps in Figure 3. The structure of encoder layer Resnet (left) and the Densenet (right). The concatenation of inputs is indicated by (c) and + indicate the add operation. BN+ReLU+Conv2D indicate the batch normalization, Relu activation and convolution of kernel size 3x3 extracting a multi-scale feature of images. The decoder up- samples the representation in every step. The skip connection can enable the decoder to select the feature at a different scale to make a more accurate prediction of the object boundaries. The output of the U-Net is 2561600Nwith sigmoid activation, where N is the types of steel defects. B. Transfer learning We explore two kinds of encoder blocks for transfer learn- ing. Both of these nets are trained using ImageNet data set [23]. We brieﬂy review the features of the two networks in the following subsections. 1) ResNet: The residual networks (ResNet) are very deep convolutions neural networks with skip connection [25]. The vanishing gradient problem is addressed by having a skip connection after each block. Each block contains a two 3x3 convolution with batch normalization and ReLU activation. Fig 3 (left) shows one layer of ResNet. The total parameters of the encoder are 11 million. 2) Densenet-121: Densely connected convolutions neural nets (DenseNets) are stacked convolution networks where the feature map of the L-th layer is concatenated with the feature maps of the previous layer [26]. This has been shown to alleviate the vanishing gradient problem. The network’s representation power is also increased because the deep layer has access to the previous layer feature maps. Fig 3 (right) shows one layer of DenseNet. The total parameters of this encoder are 6 million.C. Classiﬁcation The encoder output encodes the rich abstract representation of the input image. Hence we propose to spatial average pooling the encoder output to extract the image representation. The image representation is passed through a linear classiﬁer with sigmoid activation to enable the multi-label classiﬁcation. D. Objective function The joint segmentation and classiﬁcation problem is formu- lated as a weighted combination of the two losses, as shown below. J=PL k=1PN m=1h BCE (^ykm;ykm) +P256 j=1P1600 i=1BCE (zkijm;^zkijm)i (1) whereBCE indicate the binary cross-entropy loss, kindicate the data point index, mis the defect class index, i;jare the spatial index, ^yindicate the predicted probability and yis the ground truth defect labels. The predicted and ground truth pixel labels are indicated by zand^z. During the test stage, the labels from the probability are obtained using the threshold of 0.5. III. E XPERIMENTS AND RESULTS A. Data-set and Pre-processing The Kaggle Competition - ”Severstal: Steel Defect Detec- tion“ data is used for all the experiments. In each experiment, the input image could contain one or more kinds of defects. The training set includes 12568 images, and 6666 of them include at least one defective region. Ground truth classiﬁca- tion was performed by an expert to provide the defect type classiﬁcation and the annotation of the defective region by visual inspection. The resolution of images is 256x1600 px. We normalize the image using the global mean and standard deviation. We apply a random vertical/horizontal ﬂip as data augmentation. The same augmentation applies to both the original image and the corresponding ground truth masks to pair with the augmented images. B. Experimental setup We use a total of 75% data for the training, 12.5 % data for validation, and 12.5% data for the testing. The U-Net of ﬁve encoders and decoder is used for all the experiments. The network is trained using the objective function in eq. 1 with the batch size 16 and Adam optimizer with learning rate of 5104, 1=0.99 and 2=0.99 [27]. We train the network for 10 epochs with early stopping. We use the U-Net with random initialization as the baseline. We have implemented the network in PyTorch [28] with the PyTorch segmenta- tion library [29]. The ResNet/DenseNet with random initial- ization is indicated by ResNet(Random)/DenseNet(Random). The Imagenet pre-trained counterparts are indicated by ResNet(Imagenet)/DenseNet(Imagenet) respectively. To un- derstand the model’s sample complexity, we also train these networks using 50% of the training data. We refer to the pre- trained initialization as TLU-Net and the random initialization as just U-net. 00.20.40.60.81DICE, 100% data ResNet (Random)ResNet (ImageNet)DenseNet (Random)DenseNet (ImageNet)DICE, 50% data ResNet (Random)ResNet (ImageNet)DenseNet (Random)DenseNet (ImageNet)00.20.40.60.81IoU, 100% data ResNet (Random)ResNet (ImageNet)DenseNet (Random)DenseNet (ImageNet)00.20.40.60.81IoU, 50% data ResNet (Random)ResNet (ImageNet)DenseNet (Random)DenseNet (ImageNet)Figure 4. Boxplot comparison of DICE and IoU for different networks and the initialization. The red line indicate the median, the black dot indicate the mean, the blue box indicate the 75% conﬁdence interval and the black whiskers indicate the 95% conﬁdence intervals. 100% data ResNet (Random)DenseNet (Random)ResNet (ImageNet)DenseNet (ImageNet)85909510050% data ResNet (Random)DenseNet (Random)ResNet (ImageNet)DenseNet (ImageNet) Figure 5. Comparison of average MLA for different networks and initializa- tion using different amount of training data. C. Evaluation metrics We evaluate the steel defect classiﬁcation performance using multi-label classiﬁcation accuracy (MLA) and the average area under receiver operating curve (AUC) across 4 classes. The MLA is deﬁned as the proportion of the predicted correct labels to a total number of labels for instance. We treat the multi-label classiﬁcation as 4 separate binary classiﬁers and compute the average AUC across four classiﬁers. We have used DICE and the intersection of unions (IoU) to evaluate the performance of steel defect segmentation. The DICE metric for each class is deﬁned as follows: DICE =2jX\Yj jXj+jYj(2) The IoU is deﬁned as follows: IoU=jX\Yj jX[Yj(3) whereXandYare the ground-truth and the predicted seg- mentation masks,[indicates the union operation, \indicated the intersection, and j:jindicates the cardinally. D. Results and discussion Fig. 5 shows the MLA(%) comparison for different net- works and the initialization. It is clear from the ﬁgure that the TLU-Net achieves 5%(absolute) improvement in the MLA compared to the random initialization with 100% training data. This indicates that the features learned using ImageNet can 00.20.40.60.81DenseNet(Random):0.93774class 1 class 2 class 3 class 4 DenseNet(ImageNet):0.98565 0 0.2 0.4 0.6 0.8 100.20.40.60.81ResNet(Random):0.93338 0 0.2 0.4 0.6 0.8 1ResNet(ImageNet):0.98396Figure 6. comparison of AUC plot of four classes for different networks and initialization. The title shows the average AUC across four classes. help for steel defect classiﬁcation as well. The MLA gap increases to 8%(absolute) as the training data is reduced to 50%. The performance of TLU-Net does not drop signiﬁcantly as the training data is reduced. This indicates that the TLU- Net is helpful when there is a limited number of annotated data points. The ResNet(ImageNet) and DenseNet(ImageNet) performs best in case of 100% and 50% of training data respectively. This could be because the DenseNet has fewer parameters than the ResNet and hence needs less data but has less representational power. Fig. 6 shows the AUC for different networks and initializa- tion. It is clear from the ﬁgure that the best AUC archived for class 1 in all cases. Class 3 and class 4 defects demonstrate the poorest performace. This is mainly because of the number of samples of the training data for each class. The AUC of all classes improved by using the TLU-Net compared to U-Net. The DeseNet(ImageNet) has the highest AUC. Fig. 4 shows the box plot of the DICE and IoU for different network with the 100% and 50% of the training data. In all Figure 7. Illustration of segmentation mask prediction. (row a) The input images (row b) The ground truth masks (row c) The mask predicted by ResNet(Random) (row d) the mask predicted by ResNet(ImageNet). The corresponding DICE for the prediction is shown in the title of the image. 0 2 4 6 8 10 epochs0.10.20.30.40.50.6DICEDenseNet (Random) DenseNet (ImageNet) ResNet (Random) ResNet (ImageNet) 0 2 4 6 8 10 epochs8486889092949698MLA Figure 8. Comparison of validation loss/MLA evolution over epoch for different networks and initialization. cases, the median and mean performance of the TLU-Net perform better than that of the U-Net. In case of 100% training data, the ResNet(ImageNet) DICE/IoU is better than all the other models and it has the small 75% conﬁdence interval. The TLU-net with ResNet shows an improvement of 26% (relative) compared to U-Net with ResNet. The TLU-net with DenseNet shows an improvement of 5% (relative) compared to U-Net with ResNet. But in the case of 50% training data, the DenseNet with transfer learning shows an improvement of 60%(relative), and the ResNet shows an improvement of 12%(relative). This clearly indicate the gain of using transfer learning is higher as the number of annotated samples are lower. Fig. 8 shows the DICE/MLA metric using validation data during the course of training. This helps us understand the convergence rate of different networks. It is clear from the ﬁgure that the TLU-Net has higher DICE at the beginning of the epochs than the U-Net. The converged DICE value -0.5 0 0.5050100150200ResNet -0.5 0 0.5DenseNetFigure 9. Histogram of difference between DICE obtained using transfer learning and the random initialization. The red line indicate the zero line. for the TLU-Net is higher compared to the U-Net. This clearly indicates that transfer learning is helping in faster convergence of the model. Similar observations are applicable for the MLA plot also. It is interesting to note that the pre- trained model’s starting accuracy is signiﬁcantly higher than the random initialization. This is mainly because the classiﬁer directly uses the encoder output for classiﬁcation, and the encoder is initialized with the pre-trained weights. It implies that the pre-trained features are useful in discriminating against the different steel defects. Fig. 7 shows the four illustrative examples of mask pre- dictions for TLU-Net using ResNet and the U-Net using ResNet. In the ﬁrst column, the TLU-Net can detect the defect more accurately, and the U-Net fails to detect some parts of the defects. In the second column, the TLU-net is showing some false positive detection because of some illumination differences. In the third column, both networks failed to detect some of the defects. This mainly because the training data has few defects belonging to this class. We hypothesize that the TLU-net also fails because the defect share is more complex, and pre-learned features may not help this scenario. In the fourth column, the TLU-Net can detect the defect in the form of a ﬁne line, and the U-Net is failing to detect these lines. Fig. 9 shows the histogram of the DICE difference between the TLU-Net and the U-Net using both ResNet/DenseNet. It is clear from the ﬁgure that the histogram is skewed toward positive values. We observe improvement in the case of 81% images in case ResNet and 63% of images in DenseNet. IV. C ONCLUSION In this work, we propose to use the transfer learning framework for steel defect classiﬁcation and segmentation. We use a U-Net architecture as a base architecture and explore two kinds of encoders: ResNet and Dense Net. We compare these nets’ performance using random initialization and the pre-trained networks trained using ImageNet data set. We found that the performance of the transfer learning is superior both in terms of defect segmentation and classiﬁcation. We also found the performance gap increases as the training data decreases. We also found that the convergence rate with transfer learning is better than that of the random initialization. We have found that transfer learning performance is poor in rare defect types and complex shape defects. As a part of future work, we would like to work on transfer learning to handle more complex shapes using synthetic data and the rare defect type generalization using generative models. We want to explore the semi/weakly supervised learning approaches to reduce the annotated training data requirement.