# Deep Transfer Learning Approach for Identifying Slope Surface Cracks
## 2 Methods
### Overview
#### 2.2.3 Pretrained Model Construction  
The pretrained models are constructed by applying six deep neural network models, and a total of seven deep neural network models are built: LeNet5, AlexNet, InceptionA, InceptionE, ResNet, MobileNet, and Vision Transformer. Each of the models is briefly described as follows.  
1. LeNet5: The LeNet5 model is the most original image classification model. The entire neural network has 7 layers (excluding the input layer), each layer has multiple Feature maps, and each Feature map has multiple neurons. The result is output by the Sigmoid function, and Softmax is used as the activation function [23]. The LeNet5 model has a simple structure and a small number of network layers, but the image needs to be scaled to \(32\times 32\) pixels first and then used to train the LeNet5 model.
2. AlexNet: The AlexNet model is the first convolutional neural network model that has attracted wide attention. Compared with LeNet, AlexNet has a deeper network structure, cascading convolutional layers, dropout to suppress overfitting, ReLu to replace Sigmoid as the activation function, and multi-GPU training [24]. A series of optimization measures have been taken to achieve higher accuracy.
3. Inception module: The Inception module was first employed in GoogLeNet, which enhanced the function of the convolution module and further deepened the network depth. Inception modules stack multiple convolutional and pooling layers, and an Inception module can contain several different types of convolutional and pooling operations at the same time [25; 26]. Later, InceptionV2, InceptionV3, and InceptionV4 network structures have been developed. Because the GoogLeNet network is too large, only the InceptionA and InceptionE modules in InceptionV3 are applied to LeNet to further deepen the depth of the LeNet network and improve the accuracy.
4. ResNet: The residual module Residual Block is introduced to alleviate the vanishing gradient problem caused by the network being too deep, thus speeding up the neural network training and improving the model's accuracy [28]. The ResNet structure of the network mainly has ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152, which represent neural networks with different numbers of convolutional layers. In this paper, the ResNet18 network structure is employed for the deep learning model established.
5. MobileNet: This is a lightweight deep neural network model suitable for mobile terminals, embedded devices, and other devices that have the low computing power and require speed and real-time performance. MobileNet mainly applies Depthwise Separable Convolution (Xception variant structure) instead of the traditional standard convolution operation [30]. The main MobileNet network structures are MobileNetV1, MobileNetV2, and MobileNetV3. In this paper, we adopt the MobileNetV1 network structure and employ the deep separable convolution module to reduce the computational parameters and the amount of calculation.
6. Vision Transformer (ViT): Vision Transformer is based on the widely used Transformer model in the field of natural language processing. ViT combines the fields of CV and NLP and does not use the traditional CNN convolutional neural network but creatively deals with the computer vision field based on the Transformer architecture. It can achieve good results and high accuracy [31]. In this paper, the Vison Transformer model is employed for crack identification, and the effect of crack image identification is further tested.