# Control Chart Concurrent Pattern Classification Using Multi-Label Convolutional Neural Networks
## 3 Experimental Analysis
\begin{table}
\begin{tabular}{c c c c c c} \hline
**Layer** & **Type** & **Filter Size** & **\#Filters** & **Stride** & **Padding** \\ \hline
1 & Convolution/ReLU & \(1\times 5\) & 16 & 1 & same \\
2 & Max pooling & \(1\times 2\) & 1 & 2 & - \\
3 & Dropout (10\%) & - & - & - & - \\
4 & Convolution/ReLU & \(1\times 3\) & 32 & 1 & same \\
5 & Max pooling & \(1\times 2\) & 1 & 2 & - \\
6 & Dropout (10\%) & - & - & - & - \\
7 & Fully connected (32) & - & - & - & - \\
8 & Dropout (10\%) & - & - & - & - \\
9 & Sigmoid (7) & - & - & - & - \\ \hline \end{tabular}
\end{table} 
Table 1: Convolutional neural network architecture for experiment 1. Figure 6: Effect of the number of neurons on convolutional neural network performance. For SVC, this study chose the radial basis function (RBF) as the kernel function. The hyperparameters \(C\) and \(\gamma\) of SVC were chosen using the grid search method. Figure 8 illustrates the performance of SVC with different parameter settings. The \(C\) value had a small effect on the accuracy rate (or \(F_{2}\) score). The best performance of SVC was achieved when the parameters \(C\) and \(\gamma\) were set to 4 and 0.03125, respectively. The best \(\gamma\) value (0.03125) obtained from the grid search was equivalent to 1/the number of input features, which was 1/32. Because RF and CNN are stochastic, this study took the average of multiple runs (10 times) when evaluating model performance. The randomness of CNN comes from the influence of initial weights. The randomness of RF arises from bootstrap sampling and feature selection. The performance measures are summarized in terms of the mean and standard deviation across all runs. All performance metrics presented hereafter are expressed in decimal form. Table 2 summarizes the performance of the multi-label CNN classifier and other traditional classifiers. The multi-class CNN developed by [PERSON] et al. (2016) achieved an accuracy of 0.9235. The proposed multi-label CNN was the best performing classifier. Of the traditional classifiers evaluated, SVC performed better than RF. The recall rate and precision of RF were somewhat different, but other classifiers had very similar recall rates and precision. When partially correct matches were considered, CNN achieved the lowest Hamming loss (0.0198), indicating its superiority in classification. Figure 8: Effect of parameters on support vector classification performance. Figure 7: Effect of the number of decision trees on random forest performance. Using the procedure described in experiment 1, the number of trees for RF was 900. SVC used RBF kernel function. After a grid search study, the parameters \(C\) and \(\gamma\) were set to 16.0 and \(2^{-4}\), respectively. Table 4 summarizes the classification results for the second experiment. According to the results in Table 4, CNN achieved the highest overall classification accuracy (0.9876), followed by SVC (0.9760) and RF (0.8652). The SVC method used in this study obtained the same classification accuracy as that reported by [PERSON] and [PERSON] [51]. The proposed multi-label CNN still performed the best in terms of the other metrics. Table 5 provides a detailed comparison between the proposed multi-label CNN and the feature-based SVC developed by [PERSON] and [PERSON] [51]. It is clear that the proposed \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metric\(\backslash\)Algorithm** & **RF** & **SVC** & **CNN** \\ \hline Accuracy & 0.8652 (0.0034) * & 0.9760 & 0.9876 (0.0016) \\ Recall & 0.8888 (0.0030) & 0.9877 & 0.9930 (0.0016) \\ Precision & 0.9073 (0.0039) & 0.9910 & 0.9937 (0.0018) \\ \(F_{1}\) & 0.8953 (0.0032) & 0.9882 & 0.9927 (0.0011) \\ \(F_{2}\) & 0.8910 (0.0031) & 0.9876 & 0.9927 (0.0012) \\ Hamming loss & 0.0232 (0.0007) & 0.0047 & 0.0032 (0.0008) \\ Jaccard\_score & 0.9073 (0.0028) & 0.9814 & 0.9917 (0.0008) \\ \hline \hline \end{tabular}  
* Numbers in parentheses are standard deviations.  
\end{table} 
Table 4: Classification performance in experiment 2.