# Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
## 2 Materials and Methods
### Proposed Model
#### 2.2.2 Classification and the Proposed AWV Algorithm
``` \(\text{X}=[P_{1,1},\)\(P_{1,2},\)\(\ldots,\)\(P_{i,j},\)\(\ldots,\)\(P_{5,28},\)\(P_{5,29},\)\(ACC_{1,1},\)\(ACC_{1,2},\)\(\ldots,\)\(ACC_{i,j},\)\(\ldots,ACC_{5,28},\)\(ACC_{5,29}]\) Output: Y Process:
1) Initialize the weights matrix for memory \(\text{weights}=\textbf{W}[w_{i,j}]\) \(w_{i,j}=1\) for i = 1, 2, \(\ldots,5\) and for j = 1, 2, \(\ldots,29\)

Figure 3: The process of the proposed accuracy-based weighted voting (AWV) algorithm. #### 2.2.3 CNN Algorithms

The main structures in LeNet [37], AlexNet [38], and VGGNet [39] are shown in Figure 4a-c. The inception structure of GoogleNet [40], whose width increases by branching and merging to improve the accuracy of the model, is shown in Figure 4d. The residual module of ResNet [41] in Figure 4e generates an output F(\(x\)) + \(x\) from an input \(x\) and reduces the gradient-vanishing problem, improving the model performance in the prediction task. Five M-CNN models are designed based on the above models: M-LeNet, M-AlexNet, M-GoogleNet, M-VGGNet, and M-ResNet. Figure 4: The structure of the convolutional neural network (CNN) methods: (**a**) LeNet5, (**b**) AlexNet, (c) VGGNet, (**d**) the inception structure in GoogleNet, and (**e**) the residual module in ResNet. Table 1 shows the structure of M-LeNet in detail. The size of the kernels is modified, and one convolution layer is reduced to decrease the consumption time. Table 2 shows the structure of M-AlexNet. Batch normalization [42] is added after each convolution layer in M-AlexNet to reduce the impact of unstable gradients. Table 3 shows the structure of M-GoogleNet in detail, where MP indicates a max-pooling layer, the convolution stride is one, and there is one MP layer in the inception block. Table 4 shows the structure of M-VGGNet in detail, where the 3 \(\times\) 3 convolution kernel and 2 \(\times\) 2 max-pooling size are used in the entire network to improve the performance by continuously deepening the network structure. Batch normalization is added behind each convolution layer. The layer configuration in M-VGGNet is similar to LeNet and AlexNet. The purpose of using M-LeNet, M-AlexNet, and M-VGGNet is to help the ARS-MA model extract gesture semantic information at different depths with a similar layer configuration in the feature extraction step. Table 5 shows the structure of M-ResNet in detail, where the number of convolution layers is reduced to obtain better performance for the ASL Alphabet and ASLA datasets. M-GoogleNet, M-VGGNet, and M-ResNet are all high-depth models with different layer configurations. Hence, the ARS-MA model gains the ability to obtain more high-depth information from the feature extraction because the gesture semantic information is easily captured in the high-depth feature maps [43] and deeper feature information categories are obtained. In all models, dropout is used to reduce overfitting, the activation function in the convolution layers is the rectified linear unit (ReLu) function, and the softmax function is used for multi-class prediction. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**CNN Model** & **Architecture** & **Kernel Size** & **Output Shape** \\ \hline \multirow{8}{*}{M-LeNet} & Input & - & \(227\times 227\times 1\) \\  & Convolution\_1 & \(5\times 5\), 64, stride = 2 & \(75\times 75\times 64\) \\  & Maxpooling\_1 & \(2\times 2\), stride = 2 & \(37\times 37\times 64\) \\  & Convolution\_2 & \(5\times 5\), 128, stride = 2 & \(11\times 11\times 128\) \\ \cline{1-1}  & Maxpooling\_2 & \(2\times 2\), stride = 2 & \(5\times 5\) \(\times 128\) \\ \cline{1-1}  & Flatten & - & 3200 \\ \cline{1-1}  & Fully connected\_1 & - & 640 \\ \cline{1-1}  & Fully connected\_2 & - & 128 \\ \cline{1-1}  & Output & - & 29 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The architecture of M-LeNet.