# Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
## 2 Materials and Methods
### Proposed Model
#### 2.2.2 Classification and the Proposed AWV Algorithm  
The final classifier in the ARS-MA model was selected from among five algorithms: SVM, RF, AdaBoost [35], soft voting, and the AWV algorithm. These obtained final results from the aspects of four classification methods: nonlinear mapping (SVM), an ensemble tree (RF), a weighted ensemble tree (AdaBoost), and voting (soft-voting and the AWV algorithm). The keys of the SVM algorithm establish the maximum margin hyperplane for classification and have a good generalization ability. The RF algorithm creates new training samples, which are used to train several different decision trees. Then, the final result is aggregated from the various decision trees. Boosting is a machine-learning approach that aims to create a highly accurate model by combining many less accurate models. AdaBoost, the most widely used boosting algorithm, combines many decision trees and gives greater weights to the higher-accuracy tree classifiers. The new dataset ND1 is used for SVM, RF, and AdaBoost to fuse the CNNs. In soft voting, the probabilities of five CNN models for each class are averaged to predict the result. The new dataset ND2 is used for the soft-voting algorithm, which compares the 29 average probabilities from the five CNN models for each class, that is, the comparison of [(\(P_{1,1}+P_{2,1}+P_{3,1}+P_{4,1}+P_{5,1})/5\),..., (\(P_{1,29}+P_{2,29}+P_{3,29}+P_{4,29}+P_{5,29})/5\)]. However, a class result having a high probability from a CNN model does not mean that the class is correct. For example, AlexNet performs better than LeNet in general. When both are used in an ensemble together, LeNet incorrectly predicts Class A gestures into other classes, while AlexNet accurately predicts them in Class A, but the predicted probability of Class A in LeNet is coincidentally higher than in AlexNet. In this case, soft voting only considers the probability, which makes it difficult to help the ensemble model correct the error. Soft voting makes the final recognition of the gesture class result with the highest prediction probability but does not always consider different CNN models. Therefore, a new method AWV is proposed to consider accuracy when voting for the gesture class. This method is called Accuracy-based Weighted Voting (AWV). The soft-voting algorithm only focuses on the probabilities for the final recognition of the ensemble model, but the AWV algorithm considers the CNN prediction accuracies and the probabilities corresponding to each class, which allows the ensemble model to fuse the results of the CNNs more accurately. The new dataset ND3 is used for the AWV algorithm. The weights in the proposed AWV algorithm are defined in Equation (2):  
\[w_{i,j}=\frac{ACC_{i,j}\kappa}{\lambda}, \tag{2}\]  
where \(w_{i,j}\) is the weight value of the \(j\)th class of the dataset in the \(i\)th CNN classifier  
chosen by the max() function. Label() is defined to take the class number belonging to the largest value as the final gesture class prediction result, meaning the final ASL letter recognition result is determined by comparing 29 values: \([w_{1,1}*P_{1,1}+w_{2,1}*P_{2,1}+w_{3,1}*P_{3,1}+w_{4,1}*P_{4,1}+w_{5,1}*P_{5,1})/5\),..., \((w_{1,29}*P_{1,29}+w_{2,29}*P_{2,29}+w_{3,29}*P_{3,29}*P_{4,29}*P_{5,29})/5]\). The workflow of the AWV algorithm is shown in Figure 3.  
In the AWV algorithm, the five probabilities for each sign language class are predicted by five different CNN models, and the weights of each class are determined by the accuracy of each corresponding CNN model. The total number of weights is \(5\times 29\) in the ARS-MA model, recognizing 29 gesture classes in the ASL Alphabet and ASLA datasets. AWV makes the CNN models better complement their drawbacks after independent probability predictions. The execution process of the AWV algorithm with \(\alpha=2\), \(\lambda=5\) is shown in Algorithm 1. The proposed weighted voting is accuracy-based, where a CNN model with higher recognition accuracy is given greater weight in the AWV to improve the accuracy of the ARS-MA model.