# Multivariate Analysis of Transient State Infrared Images in Production Line Quality Control Systems
## 2 Fundamentals of Principal Component Analysis  
The contemporary real-world systems become more and more complicated so one needs more and more variables to describe it. A set of these variables is just a point in the measurement space. If the dimension of this space is high then methods of its reduction become very important as they simplify the analysis, help to uncover the relationship between observations and variables revealing groups or trends and enable visualization of the data. PCA is one of these methods. It consists in the projection of the data into space in which the first axis of the new coordinate system is aligned with the direction where the data has the largest variance. The next axes are ordered according to the value of variance of the data for the direction they represent. Moreover, this principal coordinate system is orthogonal. From the mathematical point of view the original coordinate system and principal coordinate system are related by an affine transform, i.e., translation and rotation. So PCA is a statistical procedure that performs a linear subspace projection. It enables data compression while preserving the major features of the data. PCA involves minimization of the mean square error between the projected data (onto an \(m\)-dimensional subspace) and the original data (given for the \(n\)-dimensional space, \(n>m\)).  
The PCA is based on the following theorem of linear algebra [4]. If an \(n\)-dimensional vector \(\mathbf{x}\) has the covariance matrix \(\mathbf{C_{x}}\), then there exists an orthogonal transform:  
\[\mathbf{y}=\mathbf{Q^{T}x}, \tag{1}\]such that the covariance matrix \(\mathbf{C_{y}}\) of the vector \(\mathbf{y}\) is:  
\[\mathbf{C_{y}}=\left[\begin{array}{cccc}\lambda_{1}&0&\ldots&0\\ 0&\lambda_{2}&\ldots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\ldots&\lambda_{n}\end{array}\right], \tag{2}\]  
where \(\lambda_{1}\geq\lambda_{2}\geq\ldots\geq\lambda_{n}\) are the eigenvalues of the matrix \(\mathbf{C_{x}}\) and the columns of \(\mathbf{Q}\) are the corresponding eigenvectors.  
The components of \(\mathbf{y}\) are statistically uncorrelated. Moreover, each component has the maximum variance among all normalized linear combinations that are uncorrelated with the other components. It can therefore be said that \(\mathbf{y}\) is the vector of principal components of \(\mathbf{x}\) and the method is called PCA.  
The projection of the data onto a smaller dimensionality space \(m\) (\(m<n\)) is commonly called feature extraction. What is important, such projection always results in data distortion. The error of approximation of the data set by only m features is equal to the sum of the eigenvalues that have not been taken into account:  
\[\delta=\sum\limits_{i=m+1}^{n}\lambda_{i}. \tag{3}\]  
Principal components (PCs) can be calculated not only with the help of classical algorithms based on eigenvector calculation, but also by means of more sophisticated ones like Karhunen-Loeve transform or neural networks. One can use also Singular Value Decomposition (SVD) which allows us to express a given \(m\)-by-\(n\) real matrix \(\mathbf{A}\), with \(m>n\), as:  
\[\mathbf{A}=\mathbf{US_{o}V^{T}}, \tag{4}\]  
where \(\mathbf{U}\) and \(\mathbf{V^{T}}\) are orthogonal matrices, \(\mathbf{S_{o}}\) is \(n\)-by-\(n\) diagonal matrix with elements on the diagonal in decreasing order. These elements are called singular values of the matrix \(\mathbf{A}\). The time of calculation is the main drawback of using SVD to calculate PCs in the case of high dimensional spaces. Moreover, usually only the first few PCs are important and SVD calculates all \(n\) PCs. A method that enables us to determine only PCs that are needed is called the deflation method. It resembles the well-known recursive orthogonalization procedure by Gram-Schmidt.