# Control Chart Concurrent Pattern Classification Using Multi-Label Convolutional Neural Networks
## 2 Methods
### Performance Metrics  
The performance metrics for multi-label classification models are different from those for multi-class classification problems. A recent survey of multi-label metrics and unified view of them is given by [PERSON] and [PERSON] [50]. In this subsection, the metrics used in this study are reviewed. Let \(y_{i}\) be the true set of labels for instance \(i\), \(\hat{y}_{i}\) be the predicted set of labels. Suppose \(m\) is the number of instances. The symbol \(|\cdot|\) represents the 0 norm (\(L_{0}\) norm), which refers to the number of non-zero elements in the vector. All of the following evaluation metrics are defined on a per instance basis and the aggregate value is an average over all instances.  
1. Hamming loss
2. Hamming loss is the fraction of the incorrect labels to the total number of labels, i.e., \[\frac{1}{mL}\sum_{i=1}^{m}\sum_{j=1}^{L}I(y_{ij}\
eq\hat{y}_{ij})\] (3)  
where \(y_{ij}\) and \(\hat{y}_{ij}\) are the \(j\)th element of the true label set and predicted label set for the \(i\)th instance, respectively. \(I\) is the indicator function. The quantity \(L\) is the total number of classes. The smaller the value of Hamming loss, the better the performance of the learning algorithm. The optimal value of Hamming loss function is 0. Figure 3 illustrates an example of Hamming loss. The loss in this example is calculated as 4/20. It can be interpreted as the fraction of wrong labels to the total number of labels.  
2. Jaccard index  
The Jaccard index is defined as the number of correctly predicted labels divided by the union of the true and the predicted labels. It ranges from 0 to 1, and 1 is the perfect score. \[\frac{1}{m}\sum_{i=1}^{m}\frac{|y_{i}\cap\ \hat{y}_{i}|}{|y_{i}\cup\ \hat{y}_{i}|}\] (4) c. Precision and recall Precision is defined as follows: \[\frac{1}{m}\sum_{i=1}^{m}\frac{|y_{i}\cap\ \hat{y}_{i}|}{|\hat{y}_{i}|}\] (5) Recall is defined as follows: \[\frac{1}{m}\sum_{i=1}^{m}\frac{|y_{i}\cap\ \hat{y}_{i}|}{|y_{i}|}\] (6) d. \(F_{\beta}\) score The harmonic mean of precision and recall is called the \(F_{1}\) score. The metric of classification accuracy can be used when the number of samples in each class is balanced. For an unbalanced dataset, \(F_{1}\) score is more suitable for evaluating the performance of a model, which is defined as follows: \[F_{1}=\frac{2}{\frac{1}{\text{precision}}+\frac{1}{\text{recall}}}\] (7) Another indicator is the \(F_{\beta}\) metric, which is defined as follows: \[F_{\beta}=\left(1+\beta^{2}\right)\times\frac{\text{precision}\times\text{ recall}}{(\beta^{2}\times\text{precision})+\text{recall}}\] (8)  
A higher \(\beta\) value indicates that \"recall\" is more important than \"precision\". When \(\beta=1\), it means that \"recall\" and \"precision\" are equally important. When \(\beta=0\), \(F_{\beta}\) is equivalent to \"precision\", and when \(\beta=\infty\), \(F_{\beta}\) is equivalent to \"recall\". For multi-label classification problems, the number of labels is not necessarily equal; therefore, the \(F_{\beta}\) metric is more suitable for evaluating the performance of the model than the classification accuracy. The most commonly used \(\beta\) value is 2 (hereinafter referred to as \(F_{2}\)), which means that the importance of \"recall\" is twice that of \"precision\".  
2. Exact match  
Exact match is also known as subset accuracy (hereinafter abbreviated as Acc.). It is the most rigorous performance indicator and represents the fraction at which the labels of each instance can be correctly classified: \[\frac{1}{m}\sum_{i=1}^{m}I(y_{i}=\hat{y}_{i})\] (9)  
Figure 3: Example of Hamming loss.  
where \(I\) is the indicator function. Figure 4 illustrates an example of the accuracy of the subset. In this case, the accuracy is calculated to be \(1/4\).