# Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
## 2 Materials and Methods
### Proposed Model
#### 2.2.2 Classification and the Proposed AWV Algorithm
\begin{table}
\begin{tabular}{c c c c} \hline
**CNN Model** & **Architecture** & **Kernel Size** & **Output Shape** \\ \hline Convolution\_11 & 3 \(\times\) 3, 512, stride = 1 & 6 \(\times\) 6 \(\times\) 512 \\ Convolution\_12 & 3 \(\times\) 3, 512, stride = 1 & 6 \(\times\) 6 \(\times\) 512 \\ Convolution\_13 & 3 \(\times\) 3, 512, stride = 1 & 6 \(\times\) 6 \(\times\) 512 \\ Maxpooling\_5 & 3 \(\times\) 3, stride = 2 & 3 \(\times\) 3 \(\times\) 512 \\ Flatten & - & 4600 \\ Fully connected\_1 & - & 4096 \\ Fully connected\_2 & - & 1000 \\ Output & - & 29 \\ \hline \end{tabular}
\end{table} 
Table 4: _Cont._ \begin{table}
\begin{tabular}{c c c c} \hline
**CNN Model** & **Architecture** & **Kernel Size** & **Output Shape** \\ \hline Input & - & 224 \(\times\) 224 \(\times\) 1 \\ Convolution\_1 & 7 \(\times\) 7, 64, stride = 2 & 112 \(\times\) 112 \(\times\) 64 \\ Maxpooling\_1 & 3 \(\times\) 3, stride = 2 & 56 \(\times\) 56 \(\times\) 64 \\ Residual module\_1 & 3 \(\times\) 3, 128, stride = 13 & 56 \(\times\) 56 \(\times\) 128 \\ \(\times\)2 & \(\times\) 3, 128, stride = 1 & 56 \(\times\) 56 \(\times\) 128 \\ Residual module\_2 & 3 \(\times\) 3, 256, stride = 23 & 28 \(\times\) 28 \(\times\) 256 \\ \(\times\)1 & \(\times\) 3, 256, stride = 1 & 28 \(\times\) 28 \(\times\) 256 \\ Residual module\_3 & 3 \(\times\) 3, 256, stride = 13 & 28 \(\times\) 28 \(\times\) 256 \\ \(\times\)1 & \(\times\) 3, 256, stride = 1 & 28 \(\times\) 28 \(\times\) 256 \\ M-ResNet & Residual module\_4 & 3 \(\times\) 3, 256, stride = 23 & 14 \(\times\) 14 \(\times\) 256 \\ \(\times\)1 & \(\times\) 3, 256, stride = 1 & 14 \(\times\) 14 \(\times\) 256 \\ Residual module\_5 & 3 \(\times\) 3, 256, stride = 13 & 14 \(\times\) 14 \(\times\) 256 \\ \(\times\)1 & \(\times\) 3, 256, stride = 1 & 1 & 14 \(\times\) 256 \\ Residual module\_6 & 3 \(\times\) 3, 512, stride = 23 & 7 \(\times\) 7 \(\times\) 512 \\ \(\times\)1 & \(\times\) 3, 512, stride = 1 & 7 \(\times\) 7 \(\times\) 512 \\ Avgpooling\_1 & 7 \(\times\) 7, stride = 1 & 1 \(\times\) 1 \(\times\) 512 \\ Fully connected\_1 & - & 512 \\ Output & - & 29 \\ \hline \end{tabular}
\end{table} 
Table 5: The architecture of M-ResNet. ## 3 Results and Analysis

The performance results for the five M-CNN models are shown in Table 6. The accuracies for all M-CNNs are higher than 93%, which means that each M-CNN model effectively extracts useful features for gesture prediction in ASL letter images. The performance scores of the M-CNNs decrease in the ASLA dataset, which has complex background environments, but they are no more than 1% lower than the ASL Alphabet dataset, which proves that the M-CNN models achieve almost the same results on both datasets. M-ResNet with the residual module had the best recognition performance for both datasets of all five M-CNNs because high-depth models with high-depth feature maps are well suited to extracting semantic information for prediction. The recognition scores of the five different classifiers in the classification step are shown in Table 7. The accuracy, precision, recall, and F1-scores range from 97.89% to 98.83%, 96.12% to 97.98%, 95.78% to 97.59%, and 95.95% to 97.60%, respectively. Comparing Tables 7 and 8 shows that the classification accuracies of the fusion classifiers are higher than those in a single CNN, proving that the proposed ensemble models using multiple deep CNNs are superior to single CNN models. The recognition accuracies for the two datasets in Table 7 are nearly the same, proving that the proposed ARS-MA effectively removes the noise in a complex background. The proposed AWV model was the most accurate, at 98.83% and 98.79% for the ASL Alphabet and ASLA datasets, respectively, so it was chosen as the classifier in the classification step. The ROC curves and AUC values of the five M-CNNs for the two datasets are shown in Figure 5a,b, respectively; M-ResNet achieved the highest AUCs of 0.89 on the ASL Alphabet dataset and 0.88 on the ASLA dataset. The ROC curves and the AUC values of