# Depth Evaluation of Tiny Defects on or near Surface Based on Convolutional Neural Network
## 3 Results and Discussion
### Neural Network Optimization  
To enhance the model's generalization ability and prevent overfitting, regularization techniques were introduced to incorporate additional information into the original model. One such method is dropout, which was proposed in 2012 as an optimization technique for deep neural networks [26]. Its principle involves randomly setting a fraction of weights or outputs in hidden layers to zero during training, reducing the interdependence between nodes and, thereby, achieving the regularization of the neural network. In this study, dropout was applied before both of the two convolutional neural network layers. This process involved randomly dropping out 50% of the neurons to prevent overfitting and simultaneously improve the model's generalization capacity. L2 regularization can be viewed as a penalty term added to the loss function, restricting certain parameters within the loss function. Given the small size of the dataset in this study, a regularization parameter was included in the loss function to further mitigate overfitting. The selection of hyperparameters in a convolutional neural network (CNN) frequently affects the training outcomes of the model, particularly regarding the initial learning rate \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer** & **Type** & **Parameter Settings** \\ \hline L1 & Conv & Filter number = 64, kernel size = 11 \(\times\) 11, stride = 4, activation = “ReLU”, padding = “same” \\ L2 & Max-pooling & Kernel size = 3 \(\times\) 3, stride = 4, padding = “invalid’ \\ L3 & Conv & Filter number = 128, kernel size = 7 \(\times\) 7, stride = 4, activation = “ReLU”, padding = “same’ \\ L4 & Max-pooling & Kernel size = 3 \(\times\) 3, stride = 2, padding = “valid’ \\ L5 & Conv & Filter number = 256, kernel size = 3 \(\times\) 3, stride = 1, activation = “ReLU”, padding = “same’ \\ L6 & Max-pooling & Kernel size = 3 \(\times\) 3, stride = 2, padding = “valid’ \\ L7 & FC & Units = 1024, activation = “ReLU” \\ L8 & FC & Units = 1024, activation = “ReLU” \\ L9 & FC & Units = 5, activation = “Softmax” \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The parameter settings of 2D CNN. Figure 9: The 2D CNN structure incorporated with inputted CWT spectra from raw signals. and batch size. Both parameters are crucial. An epoch represents the complete dataset's single-round forward and backward propagation through the neural network. The number of samples per period might be excessive, and the computation time required for a period calculation might be too long. As a result, the data are often subdivided into several batches for training purposes. The batch size refers to the number of samples in each batch of data. Completing one batch of sample training constitutes one iteration. Therefore, the batch size determines the direction of gradient descent in the model. When the batch size is small, the gradient direction is frequently adjusted, making it challenging to converge due to greater variability among the samples. Conversely, an excessively large batch size can result in a relatively stable gradient direction, potentially causing the model to get trapped in local optima and reducing the model accuracy. The learning rate, on the other hand, is a hyperparameter that determines the step size during the gradient descent process, essentially dictating the size of each step during optimization. If a large initial learning rate is chosen, the optimization process may linger near the global optimum without converging to it, making it difficult to find the global optimum. Conversely, a too-small initial learning rate can slow down the rate of change of the loss function, potentially leading to overfitting and requiring more time to converge. These considerations highlight the delicate balance required in choosing appropriate batch sizes and learning rates during the training of deep learning models to achieve effective convergence and optimal performance.