## 2 Methodology
### Deep Neural Networks' Setup
A fully connected layer expects a 1D vector of numbers as its input so the 3D output of the final pooling or convolution layer must be flattened into a 1D vector of numbers before it becomes the input to the fully connected layer. The most common form of a ConvNet architecture stacks a few convolutional layers (CONV), followed by a (optional) pooling layer (POOL), and repeats this pattern until the image has been reduced spatially to a small size. At this point, it is customary to introduce the fully connected layers (FC). The standard final layer for a multiclass classification problem is a fully connected layer with a number of nodes that corresponds to the number of classes and that uses the softmax function as its activation function that converts the numbers into probabilities. The ConvNet architecture thus appears as:(6)  
where \(N\in[1,3)\), \(M\geq 0,L\in[0,3)\). Typically, ConvNets are trained with the stochastic gradient descent, and its weights are updated using the backpropagation method. The objective function to be minimized (loss function) is defined as the cross-entropy between the training data and the network response. Deep neural networks frequently incorporate a regularization technique called dropout to prevent overfitting [76]. At each training iteration, a neuron is temporarily disabled with probability \(p\) (all the inputs and outputs to it will be disabled). The dropped out neurons are resampled with probability \(p\) at every training step, so a dropped out neuron at one step can be active at the next one. The hyperparameter \(p\) is called the dropout rate, and it is typically a number around 0.5, corresponding to 50% of the neurons being dropped out. A ConvNet model can be thought as a combination of two components: the feature extraction part and the classification part. The convolution and pooling layers perform feature extraction. The fully connected layers act as a classifier on top of the extracted features and assign a probability for the input image representing a class. The lower layers encode/detect simple structures (colors, edges, and simple shapes), and as we go deeper into the network, the layers build on top of each other and learn to encode more complex patterns. One of the problems using deep ConvNets is the requirement to have large annotated image datasets. For some domains, obtaining such data can be difficult, time consuming, and costly. To overcome those difficulties, transfer learning can be used by applying the ConvNets pretrained on large datasets (such as VGG-16, AlexNet [77], GoogLeNet [78], and ResNet) to a new classification task. Networks with architectures that perform well on large scale classification tasks such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [79] have been found to be able to generalize to other tasks of image classification by retraining the fully connected layers that are near the output of the network while keeping the feature extraction part of the network with the pretrained weights ([80; 81; 82]). In this work, we considered three architectures optimized for the ImageNet dataset for our task of pavement distress detection:  
* VGG16 [83], which was the best performing classifier of ILSVRC in 2014 along with GoogLeNet. This architecture has 16 weight layers, 13 of which consist of 3 \(\times\) 3 convolutional filters with a total of 4224 filters, followed by three fully connected layers of length 4096, 4096, and 1000, respectively. In total, it has 15,252,034 trainable parameters. * ResNet34 and ResNet101 [84], which introduced residual blocks to the typical ConvNet architecture and won ILSVRC in 2015. The residual block allows connections from earlier preceding convolutional layers, not only the immediately preceding one. This allows deeper models to be trained while also maintaining information only a shallower network would be able to capture [85].