# Control Chart Concurrent Pattern Classification Using Multi-Label Convolutional Neural Networks
## 3 Experimental Analysis
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **[PERSON] and [PERSON]** [51] & \multicolumn{2}{c}{**This Work**} \\ \hline
**Pattern** & **Accuracy** & **Label** & **Accuracy** \\ \hline NOR & 1.0 & (1,0,0,0,0,0) & 1.0 \\ CYC & 0.98 & (0,1,0,0,0,0) & 1.0 \\ UT & 0.98 & (0,0,1,0,0,0) & 1.0 \\ DT & 1.0 & (0,0,0,1,0,0) & 1.0 \\ US & 1.0 & (0,0,0,0,1,0) & 0.98 \\ DS & 0.98 & (0,0,0,0,0,1) & 0.98 \\ CYC+UT & 0.98 & (0,1,1,0,0,0) & 1.0 \\ CYC+DS & 1.0 & (0,1,0,0,0,1) & 0.98 \\ UT+DS & 0.96 & (0,0,1,0,0,1) & 0.98 \\ CYC+UT+DS & 0.88 & (0,1,1,0,0,1) & 0.96 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: A detailed comparison between the proposed multi-label CNN and the feature-based SVC developed by [PERSON] and [PERSON] [51]. The third experiment addressed cases in which the basic constituents of concurrent patterns may have different change points (cases in which multiple non-random patterns do not appear at the same time). Little research has been conducted on such cases [24,32]. [PERSON] and [PERSON] [51] set the change point of the non-random patterns to a fixed point in time. The shift position (\(P\)) of shift patterns was set to be 2; shift patterns were present when \(t>P\) and other non-random patterns started from \(t=1\). In the present experiment, shifts can occur at different locations in the window. The shift position was set to \(P\in\{\)2, 6, 10, 14, 18\(\}\). With this setting, a concurrent pattern that involves a shift may have different change points. Table 6 summarizes the results of each classification model. The hyperparameters and the training process were similar to those used in the second experiment. A comparison of the results in Tables 4 and 6 reveals that the performance of traditional classifiers decreased dramatically. It can be seen from the table that CNN's classification accuracy and  
Figure 10: The results of the multi-label 1D CNN. Figure 9: Test patterns for experiment 2.  
other performance metrics were better than those of traditional classifiers. What makes CNN different from other classification algorithms is its translation invariance property. The results demonstrate that CNN maintains high accuracy when basic constituents of concurrent patterns may not appear at the same point in time. The final experiment explored a more rigorous evaluation of the proposed method using a different setting of \(P\) in the training set and the test set. For the training dataset, the shift position was \(P\in\{2,6,10\}\); for the test dataset, it was \(P\in\{10,14,18\}\). By manipulating the shift position \(P\), the non-random patterns encountered in the test stage can differ from those learned in the training stage. The hyperparameters and the training process were determined using the same method as that used in the second experiment. Table 7 summarizes the results of each classification model. A degradation of performance for all classifiers can be observed. Performance degradation might be due to the setting of the shift position \(P\). Different settings for \(P\) creates variants that are not seen in the training stage. As evident in Table 7, CNN still performed the best among the classifiers that were compared. To extent the analysis, this study swapped the training and testing sets. As indicated in Table 8, CNN outperformed other classifiers in all evaluation metrics. This experiment reveals that CNN is much more resilient against the variation in change points of the basic constituents of concurrent patterns. RF performed the worst out of all the methods. This is because RF takes raw data as inputs and each observation is treated as an individual feature. This makes RF very sensitive to the location of features.