# Leveraging Feature Extraction to Perform Time-Efficient Selection for Machine Learning Applications
## 2 Dimensionality Reduction
### Feature Selection  
Feature selection is a critical step in both data engineering and machine learning workflows. This process is recognized as the selection of a subset from an initial feature set, guided by specified selection criteria [16,17]. By creating a smaller/more concise subset of relevant features using a suitable selection criterion, it is viable to reduce a problem's scale (the quantity of data involved) while  
Figure 1: Dimensionality reduction concept hierarchy. simplifying the process of analyzing it. This results in improving learning accuracy and potentially speeding up the learning process [16,18]. These outcomes are desirable since the volume of high-dimensional data has caused significant issues in the field. Due to the significant increases in sample size and dimensionality in the majority of ML use cases (e.g., the constant need to scale up large language models), this leads to the need for more computation time, more intricate models, and potentially more expensive hardware [16,18]. The main problems caused by this dimensional issue are related to the existence of noisy, redundant, or unnecessary dimensions. By creating representative subsets of the main datasets that eliminate the main sources of noise/redundancy, it is expected that the preceding issues would be solved [18]. Feature selection methods can be classified by taking into account different standards [15,16,18,19]:  
* Label Information--This standard relates to the presence of label information in the data provided to the method. Supervised approaches use label information to select relevant features to adequately distinguish samples from different classes. Semi-supervised methods also attempt to find relevant features to differentiate between classes; however, due to the limited availability of labeled data, they exploit both labeled and unlabeled data. In general, it is considered that feature selection for unsupervised problems is a more challenging task than in the two previous cases [13,20,21]. The purpose of feature selection for unsupervised learning is to locate the feature subsets that reveal natural clusters in the processed data, according to the given selection criterion. * Learning Method Relation--Feature selection techniques can be categorized as filter, wrapper, or embedded methods, based on their integration with the learning algorithm. Wrapper approaches assess the optimal feature subset using the underlying ML algorithm. The performance metric used to assess the final model may serve as the selection criterion, and multiple passes may be used to reach an optimal feature set. Filter methods select the relevant features through statistical characteristics intrinsically present in the data. They typically follow a two-step approach, with feature selection occurring before the machine learning algorithm (regardless of the objective being clustering or classification). In embedded methods, feature selection is inherent to the process of model construction. Various hybrid approaches fall under this classification. * Evaluation Criterion--Feature subsets are selected based on various evaluation criteria. Commonly, the metrics used for evaluation are the Euclidean distance, consistency, dependence, and an information measure. * Search Strategy--This standard is based on the general approach taken by the feature selection method. It divides techniques into forward selection, backward elimination, random, and hybrid models. Briefly, forward selection techniques build a subset by iteratively adding more relevant features to an initially empty one. Backward elimination, on the other hand, starts from the full feature set and iteratively eliminates features that are not considered relevant. Random approaches normally select a number of random feature subsets and then find the one with the best performance. Hybrid approaches combine any of the aforementioned approaches, or may even just take steps to improve them by including different techniques or algorithms in their process [22].