# Multivariate Statistical Process Control Using Enhanced Bottleneck Neural Network
## 2 Materials and Methods
### Multivariate Statistical Process Control
#### 2.1.2 Enhanced Bottleneck Neural Network  
Auto-associative neural networks are powerful tools for mapping inputs to outputs, especially when this mapping is nonlinear. In this work, an enhanced bottleneck neural network (EBNN) topology is used that includes five layers with three hidden layers: mapping layer, bottleneck layer, and de-mapping layer, the input and output variables [13; 22; 23]. Once the general structure is defined, it remains to determine the necessary number of neurons in each hidden layer. This number is generally determined by cross-validation. In addition, the classification using feedforward artificial neural networks is an effective tool to produce a specific output pattern for a specific input pattern. For this purpose, a small artificial neural network classifier (ANNC) is included in the output layer of EBNN (part dedicated to estimating the probability rates of normal operating modes) as shown in Figure 2. ANNC is used because of its ability to estimate the probability rates of the monitored variables, leading to their classification according to their operating modes. In the training phase, the number of normal operating modes to be extracted is assumed to be known.  
The training of the whole network includes two steps:  
* Encoding process (compression): For an input vector \(x_{i}\), the encoding process (compress inputs) can be described as: \[t_{k}=\sum_{j=1}^{h}w_{jk}^{(2)}\sigma\left(\sum_{i=1}^{n}w_{ij}^{(1)}x_{i}+ \theta_{i}\right),k=1,2,\ldots,q\] (7) When the bottleneck layer forces the EBNN to compress the inputs (variables to be monitored).  
Figure 2: EBNN for variables estimation and modes classification.  
- \(n\) is the number of neurons in the input layer.
- \(h\) is the number of neurons in the mapping layer.
- \(q\) is the number of neurons in the bottleneck layer.
- \(w\) is the weight values of the network.
- \(\theta_{i}\) is the threshold value for the \(i\)th node of the mapping layer.
- \(m\) is the number of neurons in the output layer of the operating modes classification part.
- \(\sigma\) is the sigmoid transfer function, where the transfer function in the mapping and de-mapping layers is sigmoid, and in bottleneck and output layers is linear. Exceptionally, in the output classification part is log-sigmoid in order to generate a posterior probability rates vary between 0 and 1.  
The gradient descent algorithm is adopted to train the BNN and to find the optimal values [24; 25]. This is done iteratively by changing the weights \(w\) and threshold \(\theta\) (randomly initialized before the training) according to the gradient descent so that the difference between the ideal output and the real output is the smallest. The training process is finished when the cost function \(E\) for all samples is minimized, which is calculated as:  
\[E=\sum_{n=1}^{N}\sum_{i=1}^{n}\left(x_{i}^{n}-\hat{x}_{i}^{n}\right)^{2}+\sum_ {n=1}^{N}\sum_{j=1}^{m}\left(\rho_{j}^{n}-\hat{\rho}_{j}^{n}\right)^{2} \tag{10}\]  
where  
- \(N\) is the number of training samples.  
- \(n\) is the number of neurons in the output layer of the estimation part.  
- \(m\) is the number of neurons in the output layer of the modes classification part.  
_the variables estimation part:_  
- \(x_{i}^{n}\) is the \(n\) desired values of the \(i\)th output neuron.  
- \(\hat{x}_{i}^{n}\) is the \(n\) actual outputs of that neuron.  
_the modes classification part:_  
- \(\rho_{j}^{n}\) is the probability rate already obtained with GMM of the \(j\)th mode.  
- \(\hat{\rho}_{j}^{n}\) is the actual output of \(j\)th neuron which corresponds to \(j\)th mode.  
It should be noted that the number of neurons in the output layer of the modes classification part corresponds to the extracted normal operating modes.