## 6 Dimensionality Reduction
### Effective Unbiased Automatic Feature Selection  
AFS techniques try to find the optimum feature set based on the corresponding training set. Interchange of the training and test sets or even smaller changes in the data can dramatically effect the selection. Thus, the issue of selection stability arises, in particular for small sample cases, which heretofore has been largely unanswered. For this reason, in this approach the AFS is augmented by cross-validation (CV), which is a well known method for classifier error estimation. The aim of applying CV is to perturb the training set to obtain selection statistics and information on most frequently used (stable) features [16,50]. For instance, the leave-one-out (LOO) method is implanted within the AFS procedure, where the LOOFS will take place based on (_N - I_) samples from the training set for N runs. First and second order statistics can be generated from these N selection results by incrementing the bins \(\rho_{i}\) for the selected features, or \(\rho_{ij}\) for selected feature pairs, respectively. The first and second order statistics of features is normalized by N. Three methods have been introduced in [50], namely, Highest Average Frequency (HAF), Elimination Low Rank Feature (ELRF), and Neighborhood-Linkage Feature (NLF). The first two approaches (HAF and ELRF) determine the unbiased features based on first order statistics, and the NLF is based on first and second order statistics. The HAF approach is to seek the optimal feature subset among \(N\) solutions produced by LOOFS, where each solution of active features is multiplied with their probability of the first order statistic and normalized by the number of active features. A solution is selected from the collection of solutions found by LOOFS in N runs, if its average frequency is the highest. Let \(F_{n}=(f_{n,1},f_{n,2},\)\(\ldots,f_{n,M})\) be a solution of consisting \(M\) features and \(F_{n}\) be a binary-vector, where \(n=\{\)1, 2,..., _N_\(\}\). The average frequency  
is defined as:  
\[P_{n}=\frac{\sum_{j=1}^{M}f_{n,j}\ \rho_{i}}{A} \tag{30}\]  
where A denotes the number of active features. The ELRF approach is to seek the unbiased feature subset, where the probability of selected features is above a computed threshold. The ELRF ranks the features based on the frequency of first order histogram from highest to lowest frequency and recomputes the first order histogram as follows:  
\[R_{i}=\frac{\sum_{j=1}^{i}\rho_{j}^{\prime}}{i} \tag{31}\]  
where \(\rho_{j}^{\prime}\) is the probability value of the j-th feature after ranking arrangement and \(i\) is the rank index. The threshold is computed as follows:  
\[T=\frac{\sum_{i=1}^{M}R_{i}}{M} \tag{32}\]  
where M is the number of features. In the NLF approach, first order statistics is used as a reference to determine the rank of the features. In this approach the highest rank feature will be selected or activated automatically. The rest of lowerrank features will be evaluated by referring to the active higher rank features. If the evaluation of one feature fails to even a single of its active higher rank features, the feature will be discarded, otherwise it will be selected and participated into evaluating process for the next lower rank features. This selection process is described in more detail in Figure 22.  
The evaluation process is determined by the following equation:  
\[\rho^{(2)}_{i,j}\geq S_{f}\ \ \rho_{j} \tag{33}\]  
where \(\rho^{(2)}_{i,j}\) are the 2\({}^{\text{nd}}\) order frequencies of selected the \(j\)-th feature when the \(i\)-th feature is selected, and S\({}_{\text{f}}\) denotes the selection stability measurement [16]. Here, the selection stability function is modified to give proportional assessment value for all possible cases. The selection stability measurement is defined by following equation:  
\[S_{f}=\big{(}\rho_{\text{max}}-\rho_{\text{min}}\ \big{)}\frac{U\ +\ f_{i}}{B+f_{h}\ +\ f_{z}} \tag{34}\]  
where the value of \(S_{f}\) is in the range between 0 (the worst case situations) and 1 (the best stability).