# Leveraging Feature Extraction to Perform Time-Efficient Selection for Machine Learning Applications
## 2 Dimensionality Reduction
### Feature Extraction  
Feature extraction is the process of converting high-dimensional data into a useful representation with reduced dimensionality. The dimensionality of the reduced representation should ideally match the intrinsic dimensionality of the data, which may be defined as the smallest number of factors required to explain its observable qualities [27; 28]. This reduction is of benefit to the entire ML process. Feature extraction methods are commonly divided into linear and non-linear; however, more standards may be applied [14; 27; 29]:  
* Label Information--Methods may be considered to be supervised or unsupervised. Supervised approaches take into consideration label information while processing data, while unsupervised ones infer solely from the data themselves. Most relevant feature extraction methods tend to be unsupervised. * Data Linearity--According to this standard, methods are divided according to how data are combined on processing time. Linear techniques embed data into a lower-dimensional linear subspace for dimensionality reduction. Non-linear techniques transform a higher-dimensional space into a lower one non-linearly. The most common strategies to achieve this end are preserving global properties, preserving local properties, and performing global alignment of a mixture of linear models. Traditionally, linear techniques were used, but they struggle with complex non-linear data. Non-linear techniques offer a distinct advantage in modeling real-world data, which frequently involve complex, non-linear relationships. * Data Topology--Certain techniques are designed to maintain the intrinsic relationships between data points during extraction. Random projection methods, for instance, use a randomly generated transformation matrix to project data into a lower-dimensional space while approximately preserving Euclidean distances. This is typically achieved by normalizing the matrix columns to unit length. In contrast, manifold learning approaches assume that high-dimensional data lie on a lower-dimensional, non-linear manifold. These methods, often unsupervised and graph-based, aim to uncover and embed this manifold in a way that preserves local neighbourhood relationships. Both types of techniques generate new data representations based on structural properties, complementing the traditional linear versus non-linear classification with a topology-aware view. * Intrinsic Nature--Feature extraction techniques can also be characterized by the nature of the optimization problems they solve. In particular, convex approaches optimize objective functions that do not contain local optima, allowing these methods to reliably find a global solution. In contrast, non-convex methods may converge to one of multiple local optima, which can affect the stability and quality of the extracted features. While convexity pertains to the optimization process rather than the feature extraction technique itself, understanding this distinction provides insight into the computational complexity and solution reliability associated with different methods. Regarding data linearity, in most cases, non-linear approaches appear to perform better than linear ones and manifold-based outperform random projection-based approaches, as seen in [14]. However, this is not always the case. In some cases, linear techniques (namely, principal component analysis) may outperform non-linear ones, as seen in [27]. Some relevant techniques in this area include the following [14; 29; 30]:  
* Principal Component Analysis (PCA): PCA is a feature extraction technique used to reduce dimensionality by transforming features into a new set of uncorrelated variables, called principal components, which are ordered by the amount of variance they capture from the data. This technique identifies the directions in which the data varies the most and projects the data onto them. The key point of PCA is that it maximizes variance, capturing the most important aspects of the data structure.