## 5 Strip Steel Surface Defect Recognition Deep Neural Network Based on Transfer Learning
### A Transfer Learning Deep Neural Network Based on Improved VGG19  
The current VGG19 network's learning rate was a training step method based on the learning rate decay. Since the underlying textures of strip steel are relatively fine, in order to obtain detailed defect information, the network's learning rate was set quite low. This resulted in slow convergence and falling to the local optimal value of the network, while the loss curve shocked greatly on the testing dataset, and achieved poor generalization. For the low learning rate problem, this paper proposed a hierarchical difference learning rate method. This means that underlying layers' learning rates were set low, so that edges and other fine geometry features can be carefully learned and responded to, while the layers' learning rates were set higher to ensure the network learned the images' high-level features faster and to solve the problem of slow convergence on the network. In view of the above difficulties, this paper still used the VGG19 transfer learning network as a backbone in Section 5.2. The first 15 frozen layers are still non-trainable, but the remaining convolution layer and three full connection layers of the original network were discarded. These four network layers were replaced by three convolution network layers. Therefore, the improved VGG19 backbone neural network had 18 layers in total; the network layers set their learning rates as \(10^{-6}\), \(10^{-4}\) and \(10^{-2}\) for different layer areas by the rule of 2:2:6. By observing images of strip steel surface, we found whether the surface defect area had a higher brightness or a darker brightness than the defect-free area. In Section 3.4, the defect feature extraction algorithm proposed in this paper showed poor recognition of high brightness defects. Based on the above problems, a maximum and average feature extraction module was designed in this paper. As shown in Figure 13, this module has three branches, where the original features are processed in different convolutional layers, and finally connected by the concate layer. Figure 12: The accuracy and loss error of the baseline VGG19 network. Branch 1: The original features are sequentially passed through a convolution layer of size \(1\times 1\) and a convolution layer of size \(2\times 2\). This branch is not specially processed so that branch 1 can save the features in the original image as much as possible. Branch 2: The original features are sequentially passed through a convolution layer of size \(1\times 1\) and an average pooling layer of size \(2\times 2\), and finally a ReLU activation layer is connected. Branch 2 uses the averaging pooling layer mainly to filter out the interference information in the original features. Branch 3: The original features are sequentially passed through a convolution layer of size \(1\times 1\) and a maximum pooling layer of size \(2\times 2\), and finally a ReLU activation layer is connected. Branch 3 adopts the maximum pooling layer mainly to extract the features with higher brightness from the original features, so as to better find the defect area. In order to allow the improved VGG19 backbone neural network to extract multi-level feature information, the 17 th, 18 th and 19 th layers were connected with a maximum and average feature extraction module and a convolution layer, respectively. Then, the multi-level features extracted from the three branch layers were combined to connect a global pooling layer and a full connection layer. Finally, the network was connected to the softmax classifier. As shown in Figure 14, the improved transfer learning network was constructed. Figure 14: Improved VGG19 with feature extraction modules. Figure 13: Maximum and Average feature extraction module. In the optimization process of the network it is easy for it to drop into the local optimal problem. We referenced [PERSON]'s work [46], and from this the RAdam (Rectified Adam) optimizer was introduced to avoid the VGG19 network dropping into the local optimal.