## 3 The Framework of the Proposed Method and the Data Required for the Experiment
### Overall Framework
#### 3.1.1 Image Classification in Computer Vision  
(1) Selection of convolutional neural network model  
Image classification, object detection, and image segmentation are the three major tasks in the field of computer vision. Feature extraction is carried out by a feature extractor (convolution kernel). The channels in the shallow layer learn the simple basic features, such as colors and points, at the beginning. With the increase of layers, these channels start to learn the features of line segments and edges. The deeper the layer, the more specific and abstract the features learned. Deep neural network (DNN) is widely used in image classification, detection, and semantic segmentation due to its powerful feature extraction and feature representation in convolutional layers. Feature extraction is the most critical step in DNN. The quality of feature extraction directly affects the accuracy and relevant evaluation standards in image classification, detection, and semantic segmentation. The ResNet network model is a kind of deep convolutional neural network model. In the application of image classification, the image data are convolved and pooled by multiple filters in the network to extract the edge features, such as cracks, sizes, surface defects, and so on. ResNet generally includes a convolution layer, pooling layer, local normalization layer, activation function layer, and full connection layer. At the same time, the essential difference between ResNet and other convolutional neural networks (such as VGG) is residual structure. The residual structure is introduced into ResNet, proposed by [PERSON] et al. [18], which successfully solves the problems of gradient disappearance, gradient explosion, and model degradation. Residual blocks are the powerful part of ResNet network, which uses the shortcut to transfer the high-quality weight \(x\) from the previous layer to the next layer. In this way, even if the middle layer \(F(x)\) is poorly trained and the training error is large, the training result of the upper layer is not affected. The residual output is \(H(x)\), as shown in Equation (5). \[H(x)=F(x)+x \tag{5}\]  
Figure 1: Fusion knowledge representation architecture based on computer vision and natural language technology. According to the different positions of residual structure, there are two main structures: identity mapping and projection mapping. The first structure is shown in Figure 2a, which is adopted when the input dimension and the output dimension are the same, including a convolution layer (Conv) with a 3 \(\times\) 3 convolutional kernel, a batch normalization layer (BN), and an activation function layer (ReLu). The second structure, shown in Figure 2b, is used when the input dimension and the output dimension are different. In this structure, a 1 \(\times\) 1 convolution kernel is added to the shortcut line for channel dimension raising to ensure that the output dimension of the shortcut line is consistent with that of the main line. On the premise of improving model precision, the deepest ResNet network structure reaches more than 1000 layers, and the commonly used ones are ResNet18, ResNet34, ResNet50, Resnet101 and ResNet152. Considering the equipment environment and the difficulty of image training, ResNet101 was used as the classification model in this work. (2) Introduction of deformable convolution modules  
Conventional convolutional kernels are usually fixed in size and shape and have poor adaptability to unknown changes, which limits the modeling ability of the geometric transformation of network models. As shown in Figure 3a, the size and shape of the 3 \(\times\) 3 convolution kernel are fixed, and the capability of geometric modeling in the convolution process is inadequate. According to the actual shape of the feature map, the deformable convolution kernel can learn the offset ability from the parallel convolution layer and focus on the region of interest so as to better extract the input features.