# Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
## 2 Materials and Methods
### Proposed Model
#### 2.2.2 Classification and the Proposed AWV Algorithm
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**CNN** & \multicolumn{4}{c}{**ASL Alphabet**} & \multicolumn{4}{c}{**ASLA**} \\ \cline{2-9}
**Methods** & **Accuracy** & **Precision** & **Recall** & **F1-Score** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline M-LeNet5 & 94.45\% & 94.76\% & 94.67\% & 94.71\% & 93.53\% & 95.11\% & 94.27\% & 94.69\% \\ M-AlexNet & 95.56\% & 94.46\% & 94.38\% & 94.42\% & 94.61\% & 94.11\% & 93.87\% & 93.99\% \\ M-GoogleNet & 96.07\% & 91.84\% & 90.41\% & 91.12\% & 95.54\% & 90.87\% & 90.92\% & 90.89\% \\ M-VGGNet & 97.39\% & 93.48\% & 93.74\% & 93.61\% & 96.86\% & 93.78\% & 92.69\% & 93.18\% \\ M-ResNet & 97.76\% & 95.42\% & 96.13\% & 95.77\% & 97.27\% & 95.61\% & 96.33\% & 95.97\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance evaluation of the five CNNs on both datasets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Final Fusion** & \multicolumn{4}{c}{**ASL Alphabet**} & \multicolumn{4}{c}{**ASLA**} \\ \cline{2-9}
**Classifiers** & **Accuracy** & **Precision** & **Recall** & **F1-Score** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline SVM & 98.35\% & 97.67\% & 97.12\% & 97.39\% & 98.27\% & 97.59\% & 97.31\% & 97.45\% \\ RF & 97.96\% & 96.32\% & 96.27\% & 96.29\% & 97.89\% & 97.56\% & 96.22\% & 96.89\% \\ AdaBoost & 98.11\% & 96.12\% & 95.78\% & 95.95\% & 98.04\% & 96.24\% & 96.51\% & 96.37\% \\ Soft voting & 98.46\% & 97.19\% & 96.51\% & 96.85\% & 98.41\% & 96.74\% & 96.84\% & 96.79\% \\ AWV (\(\alpha\) = 2, \(\lambda\) = 5) & 98.83\% & 97.98\% & 96.94\% & 97.46\% & 98.79\% & 97.61\% & 97.59\% & 97.60\% \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Performance evaluation of five classifiers on two datasets. \begin{table}
\begin{tabular}{c c c} \hline \hline
**\(\alpha\)** & **Accuracy on ASL Alphabet** & **Accuracy on ASL** \\ \hline
1 & 98.7843\% & 98.7432\% \\
2 & 98.8319\% & 98.7943\% \\
3 & 98.8317\% & 98.7937\% \\
4 & 98.8317\% & 98.7942\% \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Accuracies by \(\alpha\) in ARS-MA model. the five classifiers for the two datasets are depicted in Figure 5c,d. The AWV algorithm achieved the highest AUCs, of 0.93 and 0.91, on the ASL Alphabet and ASLA datasets, respectively. The AUCs of all single M-CNNs were lower than that of AWV in the ARS-MA model, which proves the proposed ARS-MA model works well for both datasets. To obtain the best model in the feature extraction step, three types of multiple deep CNNs were designed: CNN-3, CNN-4, and CNN-5. CNN-3 uses M-LeNet, M-AlexNet, and M-GoogleNet; CNN-4 adds M-VGGNet to CNN-3; and CNN-5 adds M-ResNet to CNN-4. Figure 6 shows the performance of the ARS-MA models with the three types of CNNs. It shows that increasing the diversity of the higher-depth CNN models (M-GoogleNet, M-VGGNet, and M-ResNet) in the feature extraction step improves recognition accuracy. The accuracies and AUC values of the five classifiers increase as the number of M-CNNs increases, as shown in Figure 6a,b. CNN-5 with the AWV algorithm performed better than other types of CNN models on both datasets. When \(w_{i,j}\) is calculated in Equation (2), two parameters must be assigned. \(\alpha\) is related to the relative accuracies among the M-CNN models. Table 8 shows the accuracy of the ARS-MA model by \(\alpha\) when \(\lambda\) is 5. Table 9 shows the accuracy and average test time for an image in ARS-MA as \(\lambda\) varies from 1 to 9 when \(\alpha\) is 2. An optimal value of 5 is assigned to \(\lambda\) because it reduces the training time, despite accuracy and average test time being nearly the same over the range of 3 to 7. The proposed ARS-MA model performed best when \(\alpha\) = 2 and \(\lambda\) = 5, as shown in Table 7.

Figure 5: The ROC curves and AUC values: (**a**) M-CNN models on ASL Alphabet, (**b**) M-CNN models on ASLA, (**c**) classifiers on ASL Alphabet, and (**d**) classifiers on ASLA.