## 2 Experiment and Methods
### Structure of Faster R-CNN and Mask R-CNN  
The network structure of the Faster R-CNN and Mask R-CNN used in this paper was divided into three parts: backbone structure, RPN, and head structure (Figure 3).  
For the pavement defect images, the ResNet and FPN (Figure 4) were used as the backbone network for the Faster R-CNN and Mask R-CNN, which act as a feature extractor and take the generated feature maps into the RPN. A relevant study [55] has shown that the ResNet50 or ResNet101 could solve the problem of gradient disappearance during network training. As defects were usually small relative to the whole image, it was necessary to extract the data information as much as possible in the CNN for the classification, localization, and pixel segmentation of pavement defect detection.  
The FPN was divided into three parts: bottom-up, top-down, and lateral connections (Figure 4). The bottom-up connection represented the forward propagation process of the neural network. The characteristic graph usually becomes smaller and smaller after convolution kernel calculation. The top-down process was to up-sample the more abstract and semantic high-level feature map, while the lateral connection was to merge the up-sampling results with the feature map of the same size generated from the bottom-up. The two layers of the features connected horizontally have the same space size, and the lateral  
Figure 4: Feature pyramid network.  
Figure 3: Network structure of Faster R-CNN and Mask R-CNN.  
connections can make use of the bottom positioning details. The fused feature map was finally used to predict their class and anchor box. The enlarged area in Figure 4 was the lateral connection. The main function of the \(1\times 1\) convolution kernel here was to reduce the number of feature maps without changing their size. Up-sampling was implemented on the low-resolution feature map by double times. Then, by adding element to element, the up-sampling map was combined with the corresponding bottom-up map. Figure 4 shows that the FPN was used to extract the features of the defects at various levels of dimension, as well as to construct multi-scale feature maps to improve the model performance.  
The RPN generated the region proposal boxes on the feature maps. As the input size of the fully connected layer needed to be consistent, the Fast R-CNN used the region of interest (RoI) pooling to transform the feature maps of unequal size into equal-size feature maps after the RPN operation and input them into the fully connected layer for classification. The RoI pooling also sped up the training and testing process and improved the detection accuracy. However, the RoI pooling (Figure 5a) had a shortcoming in segmentation; it would round up the pooling box if the maximum pooling was used(The red numbers are the maximum value in the area), which would cause an error in feature extraction. The RoI align (Figure 5b) was used by the Mask R-CNN to skip the pooling process, and more accurately, floating-point pixel values were obtained by the bilinear interpolation method.  
The defects would be classified, located, and segmented through an FCN eventually.