## 4 Methodology
### Classification and Feature Selection Algorithms  
Three tree-based classifiers (random forests, extra trees, and adaboost), as well as three linear models (passive aggressive algorithm, support vector classifier, and ridge classifier) are utilized to find the best features. All algorithms used in this research are examples of embedded feature selection methods.  
Random forest (RF) is a batch of decision trees [49] trained on the randomly selected subspaces of training data [50]. At each tree node, the following steps are repeated: choose random features from the set of all features and split, among the best ones, into the daughter nodes [51]. The best feature is selected according to the outcome of the criterion function. This action is finished when the boundary conditions are met, e.g., reaching the number of different class samples in a node. Similarly, in the testing process, each tree is tested on a random part of the testing data. The outcome of the classifier is nothing but aggregating all outcomes of particular trees. Extra trees (ET) have much in common with random forests except for two conditions. Firstly, each tree is trained on all training data. Then, in each tree node, the splitting is performed randomly. The remaining parts of the algorithm are ideally the same as in the previously mentioned random forests [52]. The Adaboost (AB) classifier brings together a batch of weak decision trees, where each one is trained on a subspace of training data. Each data sample misclassified by a tree obtains a higher weight than other samples [53]. Then, each subsequent decision tree is built on a base of previously updated weights. On top of that, each tree receives its score, which indicates its importance in the final result of the classification process [54]. In tree-based classifiers, the importance of the particular feature is a summed importance measure from all trees' nodes where this feature occurs [51].  
The passive aggressive algorithm (PA) is very similar to the concept of the perceptron algorithm, where the model's weight vector is updated with an analysis of every new training sample [55]. The noticeable difference during training is that in the passive aggressive model, the weight vector is not only updated when a new sample is badly classified. The algorithm's aggressiveness also updates the weight vector when the result is correct, but the value of the loss function is not equal to zero. On the other hand, when the value of the loss function is equal to zero, the weights are not altered. This mechanism is the so-called passiveness [56]. Additionally, for noisy data, a special parameter is introduced that controls the trade-off between these two terms [57]. The support vector classifier (SVM) converts the space of features and finds linear boundaries to differentiate different classes of samples [51]. To improve the performance of the classifier, a popular optimization technique of stochastic gradient descent [58] is used. The ridge classifier (R) is a special case of ridge regression. This method converts input data into a feature space, in which the linear regression function is utilized [59]. This method is based on the SVM concept [60]. The feature importance measure in linear classifiers is obtained much differently from the tree-based ones. In this case, one should analyze the feature weights calculated during the algorithms' training phase. The higher the absolute value of a feature's weight in the model, the more important the feature [61].