# Control Chart Concurrent Pattern Classification Using Multi-Label Convolutional Neural Networks
## 2 Methods
### Convolutional Neural Network (CNN)  
A typical CNN comprises of a convolution layer, a pooling layer, and one or more fully connected (FC) layers. The convolution layer is the key element in the CNN architecture. Its function is to perform feature extraction, including linear and non-linear operations (i.e., convolution operations and activation functions). The combination of convolution layer, activation layer, and pooling layer is usually referred to as the convolution block. In brief, the CNN network extracts features using the convolution blocks and performs classification using the fully connected layers. They join to form an end-to-end classifier that can reduce traditional feature engineering demands. Figure 2 illustrates a typical CNN architecture. In the convolution block, the 1D convolution layer is used to extract feature maps and different numbers of 1D convolution filters of the same size are applied in each layer. Deeper convolution blocks have more convolution filters. The distance between two consecutive filter positions is called a stride and is usually set to 1. During the convolution operation, zero padding is typically used to avoid the loss of spatial dimensions. When a convolution layer is applied, the hyperparameters that must be determined include filter size and filter number. The output of the convolution operation is passed to a non-linear activation function. Some commonly used functions include the identity, sigmoid, hyperbolic tangent (tanH), and rectified linear unit (ReLU) functions. The pooling layer is equivalent to a non-linear form of down sampling. Pooling comes in many forms, of which max pooling with a size of \(1\times 2\) and a stride of 2 is most commonly used. The pooling layer can reduce the dimensionality of the feature map, and thus can exhibit translation invariance for small offsets and deformations. The output feature map of the last convolutional layer or pooling layer is flattened, that is, it is converted into a 1D vector and connected to one or more fully connected layers. The number of neurons in a fully connected layer is typically determined through experimentation. In a classification problem, the number of neurons in the last layer of the network is equal to the number of classes. Softmax is usually chosen as the activation function of the last layer of the model. The Softmax function normalizes output values from the last fully connected layer to target class probabilities, where each value ranges between 0 and 1 and all values sum to 1. Each value in the output of the Softmax function can be interpreted as the probability of membership for each class. The class label of the maximum probability is selected as the output. The predicted output can be written as follows:  
\[\hat{y}=\arg\max_{i}\left(\frac{e^{z_{i}}}{\sum_{j=1}^{C}e^{z_{j}}}\right) \tag{1}\]  
where \(z\) is the activation value of the last layer neurons, the activation value of neuron \(i\) is \(z_{i}\), \(C\) is the number of classes and the number of last layer neurons, and \(\hat{y}\) is the label of the predicted class. During training, the network finds the weights of the filter and the weights in the fully connected layer to minimize the difference between the true class label of the sample in the training set and the predicted value of the network. The most critical elements of training are the loss function and the optimization algorithm. Classification loss is calculated by comparing the class probabilities (\(p_{i}\)) and true labels of samples (\(y_{i}\)). For multi-class classification, the loss calculated by the commonly used categorical cross-entropy loss function can be expressed as follows:  
\[\text{Loss}=-\sum_{i=1}^{C}y_{i}\log(p_{i}) \tag{2}\]  
Gradient descent is often used as an optimization algorithm to iteratively update the learnable parameters (referring to the weights of the filter and the weights of the fully  
Figure 2: Typical convolutional neural network architecture. connected layer) in the network to minimize the loss.