# Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
## 1 Introduction
### Literature Review  
Some machine-learning methods have been used in sign language recognition. [PERSON] et al. [9] proposed a static sign language recognition method based on Principal Component Analysis (PCA) and Support Vector Machine (SVM) to recognize the five vowels in English, with an accuracy of 80%. PCA and SVM were used for hand feature extraction and classification, respectively. [PERSON] et al. [10] used the Leap Motion sensor to collect 26 letters in the ASL alphabet. The \(k\)-nearest neighbor algorithm and SVM were used for classification, with accuracies of 72.78% and 79.83%, respectively. [PERSON] et al. [11] used skin-color detection and contour-extraction techniques to detect sign language in videos, and the Camshift and hidden Markov models (HMM) algorithms were used for hand tracking and classification, respectively. [PERSON] et al. [12] proposed a skin-color-based detection method to binarize the input data to obtain the face and hand regions, and they calculated the similarity between training and testing data by the dynamic time warping (DTW) algorithm. The DTW algorithm did not use a statistical model framework for training because it was difficult to connect the semantic information of the context. Consequently, DTW had disadvantages in problem solving such as large data volumes and complex gestures. However, most machine-learning methods must convert the original data into feature vectors that are suitable for operation because the image information is too complex and diverse. Thus, the incomplete feature vectors limit the image recognition performance of machine learning methods. Some deep-learning approaches have been used for sign language recognition. [PERSON] et al. [13] used a CNN to recognize ASL, achieving 97.62% accuracy. [PERSON] et al. [14] proposed a CNN-based sign language recognition method to recognize 20 words in the ChaLearn dataset. This model extracted hand and upper-body features from two sets of input data. [PERSON] et al. [15] proposed a multi-channel, multi-modality framework based on a 3D-CNN. The multiple channels contained color, depth, and optical-flow information, and the multiple modalities included gestures, facial expressions, and body poses. [PERSON] et al. [16] proposed a 3D-CNN network model based on multimodal input, including color, depth, and skeleton-joint point information. However, these methods are limited to the single-stream feature extraction portion, and obtaining sufficient feature information to distinguish similar gestures in a single-stream CNN is very difficult due to the high similarity of gestures in ASL. Gated recurrent unit-relative sign transformer [17], STMC-Transformer [18], and a full transformer network [19] have been successfully used in sign language recognition. The image frame features from sign language videos were extracted and combined into a standard encoder to boost gesture sequence attention and improve recognition performance. However, transformer methods are not appropriate for simple image recognition, such as ASL letters, because these techniques are designed to handle lengthy and complicated sign language videos, so much processing power is wasted. Some researchers have focused on combining the advantages of different deep-learning models to improve image-recognition performance. [PERSON] et al. [20] proposed a three-dimensional recurrent convolutional neural network (3 DRCNN), which combined a 3D-CNN with a fully connected recurrent neural network (FC-RNN). The 3D-CNN learned from color, optical flow, and depth channels. The FC-RNN obtained timing information of sequence segmentation from the video. [PERSON] et al. [21] used deep ensemble learning to decompose body poses automatically and perceive its background information. [PERSON] et al. [22] proposed two methods for automatically constructing ensembles with various architectures, using different architectures to achieve feature diversity. These methods allow image features to be extracted more comprehensively.