## 2 Methodology
### Model Average Ensemble of CNN  
Deep neural networks are nonlinear models that learn via a stochastic training mechanism, so they are highly flexible and capable of learning complex relationships between variables and approximating any mapping function for given training data. A drawback of this flexibility is that they are sensitive to the specifics of the training data and random initialization. They may produce a different set of weights each time they are trained. This is especially true with small datasets. The models with these different weights produce different predictions. In other words, neural networks have a high variance. A successful approach to overcome the high variance problem is ensemble learning [35; 36].  
In this research, we apply a model average ensemble to the designed CNN model in Section 2.1. The model average ensemble of neural networks is shown in Figure 3. It consists of multiple neural network models, and each model is trained with the same image data but different random initialization. When classifying an input image, the image is passed to each network, and it classifies the image independently. Final prediction is obtained by the combination of all the predictions from these models by averaging.  
We use model averaging to reduce the variance of the model, reduce the generalization error of the model, and achieve higher accuracy than any single model. In this study, we propose an ensemble of CNNs that includes 10 individual CNNs with the same model architecture. In each CNN model, we used the model design described in the previous section, whose detailed information is shown in Table 1.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**No.** & **Layer** & **Output Size** & **Filter Size** & **Stride Size** & **Dropout** \\ \hline
1. & Input & \(125\times 125\times 3\) & - & - & - \\
2. & Convolution 1 & \(125\times 125\times 32\) & \(3\times 3\) & - & - \\
3. & Relu & \(125\times 125\times 32\) & - & - & - \\
4. & Max pooling & \(62\times 62\times 32\) & - & \(2\times 2\) & - \\
5. & Convolution 2 & \(62\times 62\times 64\) & \(3\times 3\) & - & - \\
6. & Relu & \(62\times 62\times 64\) & - & - & - \\
7. & Max pooling & \(31\times 31\times 64\) & - & \(2\times 2\) & - \\
8. & Convolution 3 & \(31\times 31\times 64\) & \(3\times 3\) & - & - \\
9. & Relu & \(31\times 31\times 64\) & - & - & - \\
10. & Max pooling & \(15\times 15\times 64\) & - & \(2\times 2\) & - \\
11. & Convolution 4 & \(15\times 15\times 16\) & \(3\times 3\) & - & - \\
12. & Relu & \(15\times 15\times 16\) & - & - & - \\
13. & Max pooling & \(7\times 7\times 16\) & - & \(2\times 2\) & - \\
14. & Flatten & \(1\times 1\times 784\) & - & - & - \\
15. & Fully connected & \(1\times 1\times 784\) & - & - & - \\
16. & Relu & \(1\times 1\times 784\) & - & - & - \\
17. & Dropout & \(1\times 1\times 784\) & - & - & 0.25 \\
18. & Fully connected & \(1\times 1\times 64\) & - & - & - \\
19. & Relu & \(1\times 1\times 64\) & - & - & - \\
20. & Dropout & \(1\times 1\times 64\) & - & - & 0.5 \\
21. & Fully connected & \(1\times 1\times 5\) & - & - & - \\
22. & Softmax & \(1\times 1\times 5\) & - & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Detailed parameters of the designed model.