## 2 Materials and Methods
### Random Forest  
Random Forest was proposed by [PERSON] (1984) as an ensemble classifier or regression tree based on many decision trees. Each of the trees is based on a bootstrap sample from the original training dataset using a tree classification procedure. Bootstrapping is a metric or test that depends on random sampling with replacement. A random selection of the whole variable set is used as variables for splitting the tree nodes. After formation of the forest, a new object which needs to be classified is noted for classification by each of the trees in the forest. A vote is cast by each of the trees to indicate the tree's decision about the group or class of the object. The group or class with the majority of the votes is chosen by the forest.  
According to Neural Designer [PERSON] (1984), Random Forest is one of the most famous algorithms used by data scientists. In Random Forest, each tree is influenced by the values of a random vector sampled independently as shown in Figure 4.  
Random Forest algorithm follows as:  
Select number of trees to grow (ntree)  
For i = 1 to ntree  
Randomly sample with replacement, same size as original dataset (bootstrap).  
Grow a tree.  
For every split of tree.  
Randomly select mtry predictors.  
Grow trees till stopping criteria is reached.  
Each tree then casts a vote for the most famous class, and the class with most votes wins.  
The formal definition of the random forest in our case is given as:  
Assume training set (Kyphosis) of microarrays  
\(D=\left\{\left(\mathbf{X}_{1}\,\ y_{1}\right),\ \ldots\ldots,\ \left( \mathbf{X}_{n},\ y_{n}\right)\right\}\), where \(D\) is the full dataset, is drawn from a random probability distribution \(\left(\mathbf{X}_{i}\,\ y_{i}\right)\sim\left(\mathbf{X}_{i}\,\ \mathbf{Y}\right)\). The goal is to build a classifier which predicts \(y\) (target: kyphosis disease) from X (input features: Age, Number, Start) based on the dataset \(D\).  
Therefore, given ensemble of classifiers \(h=\left\{h_{1}(\mathbf{X}),\ldots\ldots\ \ h_{\mathbf{K}}(\mathbf{X})\right\}\), if each \(h_{\mathbf{k}}(\mathbf{X})\) is a decision tree, then the ensemble is a random forest. We, therefore, define the parameters of the decision tree for classifier \(h_{\mathbf{k}}(\mathbf{X})\) to be \(\Theta_{k}=\left(\theta_{k1},\ \theta_{k2}\ \ldots\,\ \theta_{kp}\right)\). The relationships between the classifiers and the parameters are sometimes given as \(h_{\mathbf{k}}(\mathbf{X})=h(\mathbf{X}\ |\ \Theta_{k})\). That is, decision tree k leads to a classifier \(h_{\mathbf{k}}(\mathbf{X})=h(\mathbf{X}\ |\ \Theta_{k})\). The appearance of the features which are selected in the nodes of the \(k^{th}\) occur at random, according to parameters \(\Theta_{k}\), that are randomly chosen from a model variable \(\Theta\).  
The final classification \(f(\mathbf{X})\) combines the classifiers \(\left\{h_{\mathbf{k}}(\mathbf{X})\right\}\), each tree then casts a vote for the most famous class at input X, and the class with the most votes wins.  
Figure 4: Structure of a Random Forest Breiman (1984).  
The RF model was achieved by importing the RF algorithm from the sklearn library. We further performed a grid search to help automate the selection of the best parameters to produce a model with the highest performance. The selected parameters for the grid search are as follows:  
* n_estimators (100, 150, 200, 250, 300): this correspond to the number of trees in the forest.
* Criterion (gini, entropy): The criterion is the function used to measure the quality of the split. By default, gini is selected for the Gini impurity and entropy is for the information gain. They are in a string format.
* Bootstrap (true, false): The bootstrap is the random sampling with replacement. The bootstrap samples are used by default (bootstrap = True) whereas the default strategy for extra-trees is to use the whole dataset (bootstrap = False)