## 2 Classification Algorithms Theory
### Linear Discriminant Analysis  
Linear Discriminant Analysis or LDA, introduced by [PERSON] [41] in 1936, is a linear transformation algorithm used for dimensionality reduction and binary classification. The main objective of this algorithm is to project the data into a lower-dimensional space that maximizes the between-class variance while minimizing the within-class variance [42]. Some of the applications of LDA on Industry 4.0 include state of product detection [43], malfunction monitoring systems [44], EEG hand movement classification [45], etc. That being said, let us explore the theory behind LDA. Let us suppose that we have one independent and one dependent variable, in which the latter is a binary one. The distribution of this variable with respect to the independent variable \(X1\) is as follows:  
In Figure 3, the blue and purple dots represent the different classes of the dependent variable. As we can see, it is not possible to divide these classes with a linear divider because they are scattered around the \(X1\) axis. Then, we can conclude that a single independent variable is not enough to explain the behavior of our dependent variable. Hence, it may be a good idea to add a new explanatory variable \(X2\) to our problem, as is shown in Figure 4.  
Now, we can see that, with the new feature \(X2\), it was easier to separate the blue and purple dots using a linear divider. However, this might not always be the case, and if we keep increasing the dimensional space, the problem may turn very complicated to deal with. In these cases, LDA becomes useful. Figure 4: The 2-D distribution of dependent variables. Figure 3: The distribution of dependent variables. Linear Discriminant Analysis does an analysis over the entire feature space of the problem to create new axes in a lower-dimensional space. In the example we are dealing with, the two feature spaces can be reduced to a 1-D space using LDA to transform the original \(X1\) and \(X2\) axes to a new axis that ensures maximum between-class separability while minimizing within-class variance. It is important to have in mind that both between-class and within-class variance are important for LDA to work correctly. This can be seen in Figures 5 and 6, where Figure 5 only maximizes the between-class variance (arrow between means) while ignoring within-class variance (dotted ellipses), which leads to poor separation of classes. On the other hand, Figure 6 shows a new axis that maximizes between-class variance and minimizes within-class variance, leading to good class separation. Now that we have a visual idea of how LDA works, let us explore the mathematics behind it. The general idea of LDA is to project a high-dimensional space into a line, and this projection must maximize the between-class variance and minimize the within-class variance. To find this projection, the linear discriminant finds a weight vector \(w\) that maximizes the [PERSON]'s criterion, defined as:  
\[J(w)=\frac{(\tilde{\mu}_{1}-\tilde{\mu}_{2})^{2}}{\tilde{S}_{1}^{2}+\tilde{S}_ {2}^{2}} \tag{2}\]  
where \(\tilde{\mu_{i}}\) represents the means of projections of classes 1 and 2 (\(\tilde{\mu_{i}}=w^{T}\mu_{i}\)), \(\mu_{i}\) are the means of classes 1 and 2, and \(\tilde{S}^{2}\) represents the variance (\(\tilde{S}_{i}^{2}=\sum_{y_{i}\in\text{Class}_{i}}(y_{i}-\tilde{\mu_{i}})^{2}\)), letting \(y_{i}\) be the projected samples \(y_{i}=w^{T}x_{i}\). Using these equalities, [PERSON]'s criterion can be written as a function of \(w\) as:  
\[J(w)=\frac{w^{T}S_{B}w}{w^{T}S_{w}w} \tag{3}\]  
Figure 5: (**a**) The projection of data points on a new “bad” axis. (**b**) The resultant separation with the new axis. Figure 6: (**a**) The projection of data points on a new “good” axis. (**b**) The resultant separation with the new axis. where \(S_{B}\) measures the separation between the means of two classes, and \(S_{w}\) measures the within-class scattering.