Feature Selection — Exhaustive Overview
By: Danny Butvinik
Tags: Feature Engineering, Feature Selection, Dimensionality Reduction, Data Science, Machine Learning

Photo by Dominik Scythe on Unsplash

Content

Introduction What is Feature Selection What Makes Some Feature Better Than Others Feature Selection by Label Information Supervised Feature Selection Unsupervised Feature Selection Feature Selection — Data Perspective Characteristics of Feature Selection Algorithm Feature Stability Feature Representation in Financial Crime Reference

Introduction

As a dimensionality reduction technique, feature selection aims to choose a small subset of the relevant features from the original features by removing irrelevant, redundant, or noisy features. Feature selection usually can lead to better learning performance, higher learning accuracy, lower computational cost, and better model interpretability. This article focuses on the Feature selection process and provides a comprehensive and structured overview of feature selection types, methodologies, and techniques from data and algorithm perspectives.

Figure 1: High-level taxonomy for feature selection

This article is focused on the feature selection process. The problem is important, because a high number of features in a dataset, comparable to or higher than the number of samples, leads to model overfitting, which in turn leads to poor results on the validation datasets. Additionally, constructing models from datasets with many features is more computationally demanding.

Feature selection is based on the terms of feature relevance and redundancy concerning the goal. In feature selection, a subset of features is selected from the original set of features based on features redundancy and relevance. Based on the relevance and redundant features, the feature subset classified according to four types:

noisy and irrelevant redundant & weakly relevant weakly relevant and non-redundant strongly relevant

The weakly relevant feature may not always be necessary for an optimal subset, this may depend on certain conditions. Irrelevant features are not necessary to include at all.

The feature which did not require for predicting accuracy is known as an irrelevant feature. The relevancy of the feature is measured based on the characteristics of the data not by its value. Statistics is one technique that shows the relationship between the features and their importance.

Redundant features are those that are weakly relevant but can be completely replaced with a set of other features such that the target distribution is not disturbed. Redundancy is thus always inspected in multivariate cases (when examining feature subset), whereas relevance is established for individual features.

The distortion of irrelevant and redundant features is not due to the presence of un-useful information; it is because the features did not have a statistical relationship with other features. Individually any feature may be irrelevant but it is relevant when joined with other features.

A strongly relevant feature is always necessary for an optimal feature subset; it cannot be removed without affecting the original conditional target distribution.

The aim of feature selection is to maximize relevance and minimize redundancy

Feature selection methods can be used in data pre-processing to achieve efficient data reduction. This is useful for finding accurate data models. Since the exhaustive search for optimal feature subset is infeasible in most cases, many search strategies have been proposed in the literature. The usual applications of feature selection are in classification, clustering, and regression tasks.

In general, the feature selection process can be categorized based on strategies of searching and label information.

Figure 2: Feature selection category

What Is Feature Selection

All machine learning workflows depend on Feature engineering which comprises feature extraction and feature selection that are fundamental building blocks of modern machine learning pipelines. Even though feature extraction and feature selection processes share some overlap, often these terms are erroneously equated. Feature extraction is the process of using domain knowledge to extract new variables from raw data that make machine learning algorithms work. The feature selection process is based on selecting the most consistent, relevant, and non-redundant features.

Feature selection, as a data preprocessing strategy, has been proven to be effective and efficient in preparing high-dimensional data for machine learning and pattern recognition problems.

The feature selection process is based on selecting the most consistent, relevant, and non-redundant feature subset from feature vectors. It is not only reduces training time and model complexity, but it eventually helps to prevent overfitting.

The objectives of feature selection techniques include:

simplification of models to make them easier to interpret by researchers/users

shorter training times

avoid the curse of dimensionality

enhanced generalization by reducing overfitting

The abundance of data in contemporary datasets demands the development of clever algorithms for discovering important information. Data models are constructed depending on the data mining tasks, but usually in the areas of classification, regression, and clustering. Often, pre-processing of the datasets takes place for two main reasons:

reduction of the size of the dataset to achieve more efficient analysis

adaptation of the dataset to best suit the selected analysis method.

Dataset size reduction is more important nowadays because of the plethora of developed analysis methods that are at the researcher’s disposal, while the size of an average dataset keeps growing both concerning the number of features and samples.

Dataset size reduction can be performed in one of two ways: feature selection or sample set reduction.

Figure 3: Illustration of feature selection and data size reduction in tabular data

What Makes Some Feature Representations Better Than Others?

Regardless of the technological approach to feature representation, there is a common question that haunts data scientists in most machine learning workflows: What makes some feature representations better than others?

This might seem like an insane question considering modern machine learning problems were using hundreds of thousands or even millions of features that are impossible to interpret by domain experts.

While there is no trivial answer to our target questions there are some general principles that we can follow. In general, there are three key desired properties in feature representations:

Disentangling of causal factors

Easy to model

Works well with regularization strategies

Fundamentally, solid representations include features that correspond to the underlying causes of the observed data. More specifically, this thesis links the quality of representations to structures in which different features and directions correspond to different causes in the underlying dataset, so that the representation can disentangle one cause from another.

Another leading indicator of good representation is the simplicity of modeling. For a given machine learning problem/dataset, we can find many representations that separate the underlying causal factors but they could be brutally hard to model.

The final quality of solid knowledge representations is related to structures that work well with regularization strategies. This is super important as machine learning axioms such as the ‘no free lunch theorem’ show that regularization is often needed to achieve knowledge generalization. From that perspective, a representation structure that can be used in regularization strategies is unlikely to lead to generalization. In general, there are some properties of feature representations such as smoothness, linearity, or causal factors that are good indicators of the effectiveness of regularization methods.

The way selected features are structured and represented is critical to both quality of the data and the ability to leverage that data for automated decision-making via machine learning.

Feature Selection by Label Information

In terms of availability of label information, the feature selection technique can be roughly classified into three families:

supervised methods, semi-supervised methods, and unsupervised methods.

The availability of label information allows supervised feature selection algorithms to effectively select discriminative and relevant features to distinguish samples from different classes. There are supervised feature selection algorithms that identify the relevant features for best achieving the goal of the supervised model (e.g. classification or a regression problem)and they rely on the availability of labeled data.

Unsupervised feature selection techniques ignore the target variable, such as methods that remove redundant variables using correlation. Supervised feature selection techniques use the target variable, such as methods that remove irrelevant variables.

When a small portion of data is labeled, we can utilize semi-supervised feature selection which can take advantage of both labeled data and unlabeled data. Most of the existing semi-supervised feature selection algorithms rely on the construction of the similarity matrix and select those features that best fit the similarity matrix. Due to the absence of labels that are used for guiding the search for discriminative features, unsupervised feature selection is considered a much harder problem.

Supervised Feature Selection Methods

Supervised feature selection methods are classified into four types, based on the interaction with the learning model as Filter, Wrapper, Hybrid, and Embedded Methods.

Figure 4: Extended taxonomy of supervised feature selection methods and techniques

Filter Methodology

In the Filter, method features are selected based on statistical measures. It is independent of the learning algorithm and requires less computational time. Information gain, chi-square test, Fisher score, correlation coefficient, and variance threshold are some of the statistical measures used to understand the importance of the features.

Filter methodology uses the selected metric to identify irrelevant attributes and also filter out redundant columns from the models. It gives the option of isolating selected measures that enrich a model. The columns are ranked following the calculation of the feature scores.

By choosing and implementing the right features, the accuracy and efficiency of classification models potentially can be improved.

Missing value ratio

Missing value ratio equals the number of missing values divided by a total number of observations and multiplied by 100 per column; decisions need to be taken on the threshold and drop all features that have a missing value ratio more than this threshold.

Information Gain

Calculation of reduction in entropy from the transform. It evaluates the information gain of each variable in the context of the target variable.

Chi-square Test

Applied on categorical features; it is calculated between each feature and selects features with the best Chi-squared score. There are three conditions: (1) feature must be categorical (2) sampled independently (3) expected frequency of the feature above 5.

Fisher’s Score

The features selected such that in the data space spanned by the selected features, distances between samples in different classes are as large as possible, while the distances between samples in the same class are as small as possible; Features with high quality should assign similar values to instances in the same class and different values to instances from different classes.

Correlation Coefficient

Features should be correlated with the target but should be uncorrelated among themselves.

Low Variance Filter

Low variance filter removes all zero-variance (that have the same value in all samples) features by default. The assumption is that features with higher variance may contain more useful information. Relationships between feature or feature and target variables are not taken into account, which is one of the drawbacks of this filter method.

Mean Absolute Difference (MAD)

The higher the MAD, the higher the discriminatory power.

Dispersion Ratio

Dispersion ratio is the ratio between the arithmetic mean and geometric mean AM/GM. Higher dispersion implies a higher value of AM/GM, thus a more relevant feature.

Permutation Feature Importance

Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is tabular. This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature.

Relief

Relief’s filter method approach is notably sensitive to feature interactions. It was originally designed for application to binary classification problems with discrete or numerical features. Relief calculates a feature score for each feature which can then be applied to rank and select top-scoring features for feature selection.

Alternatively, these scores may be applied as feature weights to guide downstream modeling. Relief feature scoring is based on the identification of feature value differences between nearest-neighbor instance pairs. If a feature value difference is observed in a neighboring instance pair with the same class (a ‘hit’), the feature score decreases.

Alternatively, if a feature value difference is observed in a neighboring instance pair with different class values (a ‘miss’), the feature score increases. Original Relief algorithm has since inspired a family of Relief-based feature selection algorithms (RBAs).

RBAs have been adapted to

perform more reliably in noisy problems generalize to multi-class problems generalize to numerical outcome (i.e. regression) problems make them robust to incomplete (i.e. missing) data

Here are some extensions for Relief-based feature selection algorithms: Iterative Relief, I-RELIEF, Tuned ReliefF, VLSReliefF, SURF, and ReliefSeq.

Iterative Relief

This method was implemented to address bias against non-monotonic features. Neighbors are uniquely determined by a radius threshold and instances are weighted according to their distance from the target instance.

I-RELIEF

The iterative method of Relief injects sigmoidal weighting based on distance from the target instance. All instance pairs (not just a defined subset of neighbors) contributed to score updates. This method introduces local-learning updates between iterations for improved convergence and is proposed as an online learning variant.

Tuned ReliefF

The tuned version of ReliefF is specifically meant to address noise in large feature spaces through the recursive elimination of features and the iterative application of ReliefF.

VLSReliefF (aka Very Large Scale ReliefF)

Significantly improves the efficiency of detecting two-way feature interactions in very large feature spaces by scoring random feature subsets rather than the entire feature space.

SURF

SURF identifies nearest neighbors (both hits and misses) based on a distance threshold from the target instance defined by the average distance between all pairs of instances in the training data. Results suggest improved power to detect two-way epistatic interactions over ReliefF.

ReliefSeq

ReliefSeq presents a feature-wise adaptive k parameter for more flexibly detecting univariate effects and interaction effects.

Fast Correlation Based Filter (FCBF)

Fast Correlation Based Filter (FCBF) algorithm, is a Filter method that uses the idea of ‘predominant correlation’. It selects features with high correlation with the target variable, but little correlation to other features.

Notably, the correlation used here is not the typical Pearson, Kendall, or Spearman you may be used commonly. It’s known as Symmetrical Uncertainty (SU), which is based on information theory, drawing from the concepts of Shannon Entropy and Information Gain.

The algorithm initially selects features correlated above a given threshold by SU with the class variable. After this initial filtering, it detects predominant correlations of features with the class. The definition is that for a predominant feature, for example, feature X, no other feature is more correlated to X than X is correlated to Y.

Methodology

Creates groups of the features as per different criteria Creates a benchmark for each group Tests the correlation of features inside the group, compared to the predetermined group benchmark Keeps only the features that are less correlated to each other than to the group benchmark

Wrapper Methodology

The Wrapper methodology considers the selection of feature sets as a search problem, where different combinations are prepared, evaluated, and compared to other combinations. A predictive model is used to evaluate a combination of features and assign model performance scores.

The performance of the Wrapper method depends on the classifier. The best subset of features is selected based on the results of the classifier.

Wrapper methods are computationally more expensive than filter methods, due to the repeated learning steps and cross-validation. However, these methods are more accurate than the filter method. Some of the examples are Recursive feature elimination, Sequential feature selection algorithms, and Genetic algorithms.

Boruta Algorithm

For this demonstration, I’ve chosen to implement the Boruta algorithm, with XGBoost as our wrapper classifier. By doing so, we found it to be better on the performance and efficiency fronts. The classifier tries to capture all important and interesting features that may be hiding in your data set concerning an outcome variable.

Methodology (Boruta)

Creating duplicate features and shuffle their values in each column. These features are called shadow features Trains a classifier (XGBoost) several times, on the Dataset and calculates all feature importance at all iterations For all shadow features, we create a benchmark based on the mean importance and algorithm configuration parameter. Then, the algorithm checks for each of your real features if they have higher importance. That is, whether the feature has an importance greater than the benchmark. and keeps only the ones greater.

Figure 5: Schematic flow of Boruta algorithm

Forward Feature Selection

Iteratively select the best performing feature against the target. Next, select another feature that gives the best performance in combination with the first selected feature. Construct model till preset criteria are achieved.

Backward Feature Elimination

Select all features and removes the least significant feature at each iteration which improves the performance of the model. It constructs the next model with remained features until all features are exhausted.

Exhaustive Feature Selection

Brute-force evaluation of each feature subset. This means that it tries every possible combination of features and returns the best-performing subset.

Recursive Feature Elimination

Given weights estimator for features (coefficient of the linear model), the goal is to select features by recursively considering smaller sets of features; Estimator trained and importance of each feature obtained and then least important features pruned from the feature set. This process repeats recursively on pruned set until the desired num of features reached.

Hybrid Methodology

The process of creating hybrid feature selection methods depends on what you choose to combine. The main priority is to select the methods you’re going to use, then follow their processes. The idea here is to use these ranking methods to generate a feature ranking list in the first step, then use the top k features from this list to perform wrapper methods. With that, we can reduce the feature space of our dataset using these filter-based rangers to improve the time complexity of the wrapper methods.

Embedded Methodology

In the Embedded method, there are ensemble learning and hybrid learning methods for feature selection. Since it has a collective decision, its performance is better than the other two models. Random forest is one such example. It is computationally less intensive than wrapper methods. However, this method has a drawback specific to a learning model.

In embedded techniques, the feature selection algorithm is integrated as part of the learning algorithm. The most typical embedded technique is the decision tree algorithm. Decision tree algorithms select a feature in each recursive step of the tree growth process and divide the sample set into smaller subsets.

LASSO Regularization L1

Shrinks some of the coefficients to zero. Therefore, that feature can be removed from the model.

Random Forest Importance

Random Forests inherently rank by how well they improve the purity of node, decreasing impurity (Gini impurity) over all trees. Nodes with the greatest decrease in impurity happen at the start of trees, while nodes with the least decrease in impurity occur at the end of trees. Thus, by pruning trees below a particular node, we can create a subset of the most important features.

Unsupervised Feature Selection Methods

Due to the scarcity of readily available labels, unsupervised feature selection methods are widely adopted in the analysis of high-dimensional data. However, most of the existing UFS methods primarily focus on the significance of features in maintaining the data structure while ignoring the redundancy among features. Moreover, the determination of the proper number of features is another challenge.

There are two issues involved in developing an automated Feature Subset Selection Algorithm for unlabeled data:

the need for finding the number of clusters in conjunction with feature selection, and

the need for normalizing the bias of feature selection criteria concerning the dimension

Unsupervised feature selection methods are classified into four types, based on the interaction with the learning model as Filter, Wrapper, and Hybrid methods.

Figure 6: Extended taxonomy of unsupervised feature selection methods and techniques

Filter Methodology

Unsupervised feature selection methods based on the filter approach can be categorized as univariate and multivariate. Univariate methods, aka ranking-based unsupervised feature selection methods use certain criteria to evaluate each feature to get an ordered ranking list of features, where the final feature subset is selected according to this order. Such methods can effectively identify and remove irrelevant features, but they are unable to remove redundant ones since they do not take into account possible dependencies among features. On the other hand, multivariate filter methods evaluate the relevance of the features jointly rather than individually. Multivariate methods can handle redundant and irrelevant features; thus, in many cases, the accuracy reached by learning algorithms using the subset of features selected by multivariate methods is better than the one achieved by using univariate methods.

Univariate Filter Methods

Within the univariate filter methods, two main groups can be highlighted:

methods that assess the relevancy of each feature based on Information methods that evaluate features based on Spectral Analysis using the similarities among objects.

In methods based on information, the idea is to assess the degree of dispersion of the data through measures such as entropy, divergence, mutual information, among others, to identify cluster structures in the data.

On the other hand, methods based on Spectral Analysis — Similarity, aka Spectral Feature Selection methods, follow the idea of modeling or identifying the local or global data structure using the eigensystem of Laplacian or normalized Laplacian matrices derived from an object similarity matrix.

Information-based Methods

Sequential Backward Selection Method for Unsupervised Data

SUD is a filter method that weighs features using a measure of the entropy of similarities based on distance, which is defined as the total entropy induced from a similarity matrix, where the elements of this matrix contain the similarity between pairs of objects in the dataset. The idea is to measure the entropy of the data based on the fact that when every pair of objects is very close or very far, the entropy is low, and it is high if most of the distances between pairs of objects are close to the average distance. Therefore, if the data has low entropy, there are well-defined cluster structures, while there are not when the entropy is high. The relevance of each feature is quantified using a leave-one-out sequential backward strategy jointly with the entropy measure above mentioned. The final result is a feature ranking ordered from the most to the least relevant feature.

SVD-Entropy

The central idea is to select those features that best represent the data, measuring the entropy of the original data matrix through its singular values. This entropy varies between 0 and 1, in such a way that when the entropy is low (close to zero), well-formed clusters are generated, since the spectrum of the data matrix is not uniformly distributed;

When the entropy is high, the spectrum is uniformly distributed, and the cluster structure is not well-defined. Through a leave-one-out comparison, the contribution of each feature to the entropy (CE) is evaluated, and the features are sorted according to their respective CE values.

Representation Entropy

This method rank features using information theory. The aim is to weigh each feature using the concept of Representation Entropy. Representation Entropy is a measure of information compression in a dataset, and it is computed from the entropy of eigenvalues of the covariance matrix of the data. Representation Entropy ranges from 0 to 1, where 1 represents the maximum compression, and 0 is the minimum one.

Spectral Similarity-based Methods

Laplacian Score

One of the most referenced and relevant univariate filters in unsupervised feature selection methods based on Spectral Feature Selection is Laplacian Score. In Laplacian Score, the importance of a feature is evaluated by its variance and its power of locality preserving. This method assigns high weights to those features that most preserve the predefined graph structure represented by the Laplacian matrix. This idea comes from the observation that two objects are probably related to the same cluster if they are close to each other; in such a way that those features that take similar values for close objects, and dissimilar values for the far away ones are the most relevant.

Spectrum Decomposition

SPEC evaluates the relevance of a feature by its consistency with the structure of the graph induced by the similarities among objects. This method consists of three steps: (1) building the object similarity matrix as well as its graph representation; (2) evaluating features using the eigensystem of the graph by measuring the consistency between each feature and those nontrivial eigenvectors of the Laplacian matrix. And (3), ranking features in descending order in terms of their feature relevance and consistency. In fact, SPEC is a generalization of the Laplacian Score.

Unsupervised Spectral Feature Selection

Unsupervised Spectral Feature Selection Method applied on mixed data. It assesses features by analyzing the changes in the spectrum distribution (spectral gaps) of the first non-trivial eigenvalues of the Normalized Laplacian matrix when each feature is excluded from the whole set of features separately. Features are sorted in descending order according to their respective spectral gap values.

Multivariate Filter Methods

Multivariate filter methods can be divided into three main following groups: Statistical / Information, Bio-inspired, and Spectral / Sparse Learning-based methods.

Statistical/Information group includes unsupervised feature selection methods that perform the selection using statistical and/or information theory measures such as variance-covariance, linear correlation, entropy, mutual information, among others.

The bio-Inspired group includes unsupervised feature selection methods that use stochastic search strategies based on the swarm intelligence paradigm for finding a good subset of features, which satisfies some criterion of quality.

Spectral/Sparse learning group includes unsupervised feature selection methods that are based on spectral analysis or a combination of spectral analysis and sparse learning. This group of methods is sometimes considered embedded methods because feature selection is achieved as part of the learning process, commonly through the optimization of a constrained regression model. But embedded methods also could be considered as a sub-category of filter, wrapper, and hybrid approaches.

Statistical / Information Group

Feature Selection using Feature Similarity

FSFS method introduces a statistical measure of dependency/similarity to reduce feature redundancy; this measure called Maximal Information Compression Index (MICI) is based on the variance-covariance between features. The idea of this method is partitioning the original set of features into clusters, such that those features in the same cluster are highly similar, while those in different clusters are dissimilar. Feature clustering is done iteratively based on the kNN principle as follows: in each iteration, FSFS computes the k-nearest features of each feature (using MICI). Then, the feature with the most compact subset of k-nearest features (determined by the distance to its farthest feature among the k-nearest) is selected, and its k nearest features are discarded. This procedure is repeated for the remaining features until all of them are either picked or discarded.

Relevance Redundancy Feature Selection

RRFS is a supervised/unsupervised feature selection method, which selects features in two steps. First, the features are sorted according to a relevance measure (variance for the unsupervised version and the Fisher’s Ratio or mutual information for the supervised one). Then, in the second step, following the order generated in the previous step, the features are evaluated using a feature similarity measure to quantify the redundancy between them. Afterward, the first p features with the lowest redundancy are selected.

Maximum Projection and Minimum Redundancy

MPMR is an unsupervised feature selection method based on a criterion of maximum projection and minimum redundancy. The core idea is to select a feature subset such that all original features are projected into a feature subspace (applying a linear transformation) with minimum reconstruction error while aiming to maintain low redundancy, a term for quantifying the redundancy among features, estimating redundancy rate with added Pearson correlation coefficient.

Information based

In information-based, the basic idea is to select features using a measure of the entropy of similarities based on distance.

Minimum Dependency

Min Dependency is a multivariate statistical-based filter method that holds the objective to remove redundant features using the concept of minimization of the feature dependency. The idea is to find independent features (relevant) by choosing a set of coefficients such that the linear dependency of features (expressed by the error vector E) could be close to zero. At each iteration, the feature with the largest absolute coefficients is removed, and the effect of its removal is updated. This process is iterated until all the remaining error vectors E are smaller than a threshold fixed by the user.

Bio-Inspired Methods

Unsupervised Feature Selection based on Ant Colony Optimization

UFSACO’s main objective is to select feature subsets with low similarity among features (low redundancy). In this case, the search space is represented as a complete undirected graph, where the nodes represent the features and the weights of the edges represent the similarities between features. This similarity is computed using the cosine similarity function. The idea is that if two features are similar, then these features are redundant. Each node in the graph has a desirability value called pheromone, which is updated by agents (ants) in the function of its current value, a pre-specified decay rate, and the number of times that a given feature has been selected by an agent. The agents traverse the graph iteratively preferring high pheromone values and low similarities until a pre-specified stop criterion (number of iterations) is reached. Finally, those features with the highest pheromone value are selected. Consequently, it is expected to pick feature subsets with low redundancy.

Microarray Gene Selection based on Ant Colony Optimization and Relevance-Redundancy Feature Selection based on ACO (ant colony optimization)

In both MGSACO and RR-FSACO, in addition to quantifying the feature redundancy as in the UFSACO method, they also measure the relevance of each feature through its variance. Therefore, the main objective of all these methods is to select features that minimize redundancy and at the same time maximize relevance.

Spectral / Sparse Learning Methods

Some multivariate methods based on spectral analysis derived from the SPEC and the Laplacian Score.

Minimum-Redundancy Spectral Feature Selection

In mR-SP that combines the SPEC ranking and the minimum redundancy optimality criterion. The basic idea of this method is to add a way for controlling the feature redundancy in SPEC, by introducing an evaluation measure for quantifying the similarity of each pair of features through a modified cosine similarity function.

Laplacian Linear Discriminant Analysis-based Recursive Feature Elimination

LLDA-RFE method extends the Linear Discriminant Analysis (LDA) to the unsupervised case using the similarities among objects; this extension is called LLDA. The idea is to recursively remove features with the smallest absolute values of the discriminant vectors of the LLDA to identify features that potentially reveal clusters in the samples. LLDA-RFE is closely related to Laplacian Score; the main difference is that LLDA-RFE is a multivariate method, which allows selecting features that in combination contribute to discrimination.

Multi-Cluster Feature Selection

MCFS consists of three steps: (1) spectral analysis, (2) sparse coefficient learning, and (3) feature selection. In the first step, spectral analysis is applied to the dataset to detect the cluster structure of the data. Then, in the second step, since the embedding clustering structure of the data is known, through the first k eigenvectors of the Laplacian matrix, MCFS measures the importance of the features by a regression model with an L1-norm regularization. In the third step, after solving the regression problem, MCFS selects d features based on the highest absolute values of the coefficients obtained through the regression problem.

Minimize the feature Redundancy for Spectral Feature Selection

MRSF evaluates the features all together to eliminate redundant features. The idea is to formulate the feature selection problem as a multi-output regression problem, and the selection is performed by enforcing the sparsity applying the norm L2, instead of the L1 norm.

Unsupervised Discriminative Feature Selection Algorithm

UDFS performs feature selection by simultaneously exploiting discriminative information contained in the scatter matrices and feature correlations. This method proposes to address the feature selection problem taking into account the trace criterion into the regression problem. Furthermore, UDFS adds some additional constraints to the regression problem and proposes an efficient algorithm to optimize it. UDFS ranks each feature according to the corresponding weight value in descending order, and the top-ranked features are selected.

Joint Embedding Learning and Sparse Regression

JELSR operates on the same objective function as MRSF, and it only differs in the construction of the Laplacian graph, since in this work, locally linear approximation weight is used to measure local similarity for building the Laplacian graph.

Nonnegative Discriminative Feature Selection

NDFS like UDFS and MRFS, performs feature selection exploiting the discriminative information and feature correlations in a unified framework. First, NDFS uses spectral analysis to learn pseudo-class labels (defined as non-negative real values). Then, a regression model regularization is formulated and optimized through a special solver also proposed in this work. The main difference between NDFS and UDFS is that NDFS adds a non-negativity constraint to the regression problem, since removing this constraint NDFS becomes UDFS.

Feature subset with Sparsity and Low Redundancy

FSLR employs spectral analysis to represent the data in a lower dimension and introduces a novel regularization term into the objective function with a non-negative constraint.

Structured Optimal Graph Feature Selection

SOGFS simultaneously performs feature selection and local structure learning, which was proposed. SOGFS adaptively learns local manifold structure by introducing a similarity matrix in a sparse optimization model based on minimization on both loss function and regularization. Features are selected according to the corresponding weights once the proposed model has been optimized.

Clustering-Guided Sparse Structural Learning

CGSSL is a general method for feature selection that jointly exploits nonnegative spectral analysis and structural learning with sparsity. The idea is to use the cluster indicators (learned with nonnegative spectral clustering) in a linear model to provide label information for structural learning.

Robust Unsupervised Feature Selection

RUFS’s objective is to achieve both robust clustering and robust feature selection. Unlike the unsupervised feature selection methods above mentioned such as MCFS, UDFS, and NDFS, RUFS learns the pseudo cluster labels via local learning regularized robust nonnegative matrix factorization.

Robust Unsupervised Feature Selection via Matrix Factorization

RUFSM selects features by performing discriminative feature selection and robust clustering simultaneously. The main difference between RUFS and RUFSM is that the latter uses the cluster centers as an objective concept rather than the pseudo labels of the data.

Regularized Self-Representation Model for Unsupervised Feature Selection

RSR reflects the idea that if a feature is important, then it will participate in the representation of most of the other features. The feature selection is done by the minimization of the self-representation error for the characterization of residuals, and the most representative features (those with high feature weights) are selected.

Structure-Preserving Non-negative Feature Self-Representation

SPNFSR method takes into account both the self-representation and the structure-preserving ability of features by optimizing a model. The general idea of this method methods is to optimize a model (objective function) take into account three aspects: (1) the self-representation of features; (2) the local manifold geometrical structure of the original data using a graph-based norm regularization term; 3) a regularization term W to reflect the importance of each feature. The optimization problem is solved through an efficient iterative algorithm. At the final stage, each feature is sorted according to the corresponding W values in descending order and the top p ranked features are selected.

Locally Linear Embedding

LLE is a set of non-convex sparse regularization functions in sparse learning models. The idea is to characterize the intrinsic local geometric through an LLE graph-based instead of the typical pairwise similarity matrix jointly with a structure regularization term. For each feature, a feature-level reconstruction score based on the LLE graph is defined, and the final feature subset is selected according to this score.

Wrapper Methodology

Unsupervised feature selection methods based on the wrapper approach can be divided into three broad categories

according to the feature search strategy: sequential, bio-inspired, and iterative. In sequential methodology, features are added or removed sequentially. Methods based on sequential search are easy to implement and fast.

On the other hand, bio-inspired methodology tries to incorporate randomness into the search process, aiming to escape from local optima.

Iterative methods address the unsupervised feature selection problem by casting it as an estimation problem and thus avoiding a combinatorial search.

Wrapper methods evaluate feature subsets using the results of a specific clustering algorithm. Methods developed under this approach are characterized by finding feature subsets that contribute to improving the quality of the results of the clustering algorithm used for the selection. However, the main disadvantage of wrapper methods is that they usually have a high computational cost, and they are limited to be used in conjunction with a particular clustering algorithm.

Sequential Methods

Maximum Likelihood

Two feature selection criteria are evaluated: the criterion of ML and the scatter separability criterion. This method searches through the space of feature subsets, evaluating each candidate subset as follows: First, Expectation-Maximization (EM) or KMeans clustering algorithms are applied on the data described by each candidate subset. Then, the obtained clusters are evaluated with the ML or T separability criteria. The method uses a forward selection search for generating subsets of features that will be evaluated as described above. The method ends when the change in the value of the used criterion is smaller than a given threshold.

Category Utility by COBWEB

This method is based on a measure called category utility, which is used to measure the quality of the clusters found by the COBWEB hierarchical clustering algorithm. This method generates subsets of features with two search strategies: forward selection and backward elimination. Feature selection is performed by running the COBWEB algorithm using the subset of features generated by the search strategy and evaluating the category utility for this feature subset. The process ends when no higher category utility score can be obtained in the backward or forward selection.

Simplified Silhouette Sequential Forward Selection

SS-SFS method selects a feature subset that provides the best quality according to the simplified silhouette criterion.

In this method, a forward selection search is used for generating subsets of features. Each feature subset is used to cluster the data using the k-means clustering algorithm, and the quality of the feature subset is evaluated through the quality of the clusters measured with the simplified silhouette criterion. The feature subset that produces the best value of this criterion in the forward selection is selected.

Bio-Inspired Methods

Evolutionary Local Selection Algorithm

ELSA method searches feature subsets as well as the number of clusters based on the KMeans and Gaussian Mixture clustering algorithms. Each solution provided by the clustering algorithms is associated with a vector whose elements represent the quality of the evaluation criteria, which are based on the cohesion of the clusters, inter-class separation, and maximum likelihood. Those features that optimize the objective functions in the evaluation stage are selected.

Multi-Objective Genetic Algorithm

MOGA method proposes a multi-objective fitness function that minimizes the intra-cluster distance (uniformity) and maximizes the inter-cluster distance (separation). Each chromosome represents a solution, which is composed of a set of k cluster centroids (cluster center for continuous features and cluster mode for categorical features) described by a subset of features. The number of features used for each centroid in each chromosome is randomly generated, and the cluster centers and cluster modes of chromosomes in the initial population are created by generating random numbers, and feature values from the same feature domain, respectively. Then, for re-assigning cluster centroids, MOGA uses the k-prototypes clustering algorithm which obtains its inputs from the initial population generated in the previous step. Afterward, the crossover, mutation, and substitution operators are applied, and the process is repeated until a prespecified stop criterion is met. In the final stage, this method returns the feature subset that optimizes the fitness function jointly with the clusters that they produced.

Iterative Methods

Feature Salience

The idea is to estimate a set of weights (real values in [0 − 1]) called feature saliences (one for each feature) to quantify the relevance of each feature. This estimation is carried out by a modified EM algorithm derived for the task. The method returns the parameters of the density functions that model the components (clusters), as well as the set of feature salience values. Then, the user can consider those feature saliencies that best discriminate between different components (those with the highest values).

Local Learning-Based Clustering

LLC framework formulates the final ridge regression model. Feature selection is done by introducing a binary feature selection vector τ to the local discriminant function of the model. In the end, after the convergence, the output is the vector τ along with a discretized cluster indicator matrix.

Embedded Unsupervised Feature Selection

EUFS method directly embeds the feature selection in the clustering algorithm via Sparse Learning. In this method non-convex sparse regression model uses a loss function based on L2 norm and optimizes through an Alternating Direction Method of Multipliers (ADMM). EUFS uses the KMeans clustering algorithm to initialize a pseudo cluster indicator matrix U and a latent feature matrix V (used for indicating feature weights) in the final model. Once the model has converged, the output is a feature ranking sorted according to the final values of the latent feature matrix along with the pseudo clusters indicators.

Dependence Guided Unsupervised Feature Selection

DGUFS method simultaneously performs feature selection and clustering using a constraint model. The model is optimized using a modified algorithm based on the iterative Alternating Direction Method of Multipliers (ADMM).

Gaussian Mixture Models

In GMM the idea is to apply feature selection and clustering simultaneously, using a Gaussian mixture model. The objective is to optimize the Gaussian mixture model via the Expectation-Maximization clustering algorithm, where the maximization step of this algorithm was reformulated as an L1-constraint LASSO problem. The method returns the clusters as well as the coefficients of the model; the coefficients indicate the relevance of each feature.

Hybrid Methodology

Hybrid methods try to exploit the qualities of both approaches, filter, and wrapper, trying to have a good compromise between efficiency (computational effort) and effectiveness (quality in the associated objective task when using the selected features).

To take advantage of the filter and wrapper approaches, hybrid methods, in a filter stage, the features are ranked or selected applying a measure based on intrinsic properties of the data. While, in a wrapper stage, certain feature subsets are evaluated for finding the best one, through a specific clustering algorithm. We can distinguish two types of hybrid methods: methods based on ranking and methods non-based on the ranking of features.

It is worth noting that, in the literature, some hybrid unsupervised feature selection methods like (Jashki et al. 2009; Hu et al. 2009; Yang et al. 2011a; Yu 2011) designed specifically for handling data in specific domains also have been proposed.

Fuzzy Evaluation Index

In FFEI method exponential entropy measure is combined with the fuzzy evaluation index for feature ranking and feature subset selection, respectively. The method employs sequential search considering subsets of features based on the generated ranking and using the fuzzy evaluation index as a quality measure. In the wrapper stage, to select even a smaller feature subset, the fuzzy-c-means algorithm and the scatter separability criterion are used to select a compact subset of features.

Calinski-Harabasz Index

In this method, spectral feature selection is combined with the Calinski-Harabasz index for selecting a relevant feature subset. The feature selection is divided into two stages: (1) Feature ranking and, (2) feature subset selection. In the first stage, the idea is to identify those features that preserve the data structure computing for each feature the Laplacian Score. This produces a feature ranking. After, in the second stage, taking advantage of the ranking generated in the previous stage and using forward or backward selection search, feature subsets are evaluated through a modified internal evaluation index called Weighted Normalized Calinski-Harabasz index (WNCH). The feature subset with the highest WNCH value is selected.

Bayesian Filter KMeans

BFK is a non-biased ranking method that combines KMeans and a Bayesian filter. This method, unlike all the above-mentioned hybrid methods, begins with the wrapper stage, by running the KMeans clustering algorithm on the dataset with a range of clusters specified by the user. The clusters are evaluated with the simplified silhouette criterion and the one with the highest value is selected. Subsequently, in the filter stage, using the concept of Markov blanket, a feature subset is selected through a Bayesian network, where each cluster represents a class, the nodes represent features, and the edges represent relationships between features.

Least Square Estimation using Sequential Forward Selection

LSE-SFS is a non-based on ranking method that removes both irrelevant and redundant features. This method performs feature selection in two steps: in the first step, a subset of features is founded by applying the least-square estimation LSE-based evaluation. The second step works only with those features identified in the first step, and by using a Sequential Forward Selection search the best feature subset that maximizes the clustering performance (using a modified version of the EM clustering algorithm).

Entropy Measure

This method one of the first ranking unsupervised hybrid feature selection methods that are based on the entropy measure where filter stage joints with the internal scatter separability criterion (wrapper stage). In the filter stage, each feature, one by one, is removed from the whole set of features, and the entropy generated in the dataset after the elimination of the feature is computed. This produces a sorted list of features according to the degree of disorder that each feature generates when it was removed from the whole set of features. Once all features have been sorted, in the wrapper stage, a forward selection search is applied jointly with the k-means clustering algorithm to build clusters that are evaluated using the scatter separability criterion. This method selects the feature subset that reaches the highest value for the separability criterion.

Feature Selection — Data Perspective

Figure 7: Feature Selection — Data Perspective

Big data presents challenges for the traditional feature selection task.

Streaming Data and Features

Streaming data and features have become more widespread in real-world applications. This poses a significant challenge to traditional feature selection algorithms, which assume static datasets with fixed features. It is impractical to apply traditional batch-mode feature selection algorithms to find relevant features at each round when new data or new feature arrives. In addition, the volume data may sometimes be too large to be loaded into memory directly with a single data scan. This is especially a problem when a second pass is either unavailable or very expensive. Therefore, it is more appealing to apply feature selection in a streaming fashion to dynamically maintain the best set of features from all features and data seen up to that point.

Feature Selection Algorithms with Feature Streams

Supervised Grafting Algorithm (Perkins and Theiler, 2003) Supervised Alpha-Investing Algorithm (Zhou et. al., 2005) Supervised Online Streaming Feature Selection Algorithm (Wu et. al., 2010, 2013) Supervised Streaming Feature Selection with Group Structures (Wang et. al., 2013, Li et. al., 2013) Unsupervised Streaming Feature Selection in Social Media (Li et. al., 2015)

Feature Selection Algorithms with Data Streams

Supervised Online Feature Selection (Wang et. al., 2014) Unsupervised Feature Selection on Data Streams (Huang, et. al., 2015)

Heterogeneous Data

Most existing algorithms of feature selection are designed to handle tasks with a single data source and always assume that the data is independent and identically distributed (i.i.d). However, multi-source data is quite prevalent in many domains. For example, in the social media or financial crime domain, data come from heterogeneous sources such as tabular data, text, images, tags.

Figure 8: Multi-source data versus multi-view data feature selection

Feature Selection Algorithms within Multi-view Data

Adaptive Multi-view Unsupervised Feature Selection (Feng, et.al., 2013) Unsupervised Feature Selection for Multi-view Data (Tang et. al., 2013) Multi-view Clustering and Feature Learning via Structured Sparsity (Wang et. al., 2013)

In addition, linked data is omnipresent and presents itself in various forms such as user-post relations and user-user relations. The availability of multiple data sources brings unprecedented opportunities as we can leverage shared intrinsic characteristics and correlations to find more relevant features. However, challenges are also definitely presented with additional data sources. For instance, with the existence of link information, the widely adopted i.i.d. assumption in most machine learning algorithms does not hold. How to appropriately utilize link information for feature selection is still a challenging problem.

Structures Between Features

Sometimes, features can exhibit certain types of structures in many real-world applications. Some well-known structures among features are group structure, tree structure, graph structure, etc. When performing feature selection, if the feature structure is not taken into consideration, the intrinsic dependencies may not be captured and the selected features may not be suitable for the data. Incorporating the prior knowledge of feature structures can help select relevant features to greatly improve the learning performance.

Figure 9: Categorization of traditional feature selection approaches

Characteristics of Feature Selection Algorithms

The purpose of feature selection algorithms is to identify relevant features according to a definition of relevance. However, the notion of relevance in machine learning has not yet been rigorously defined on a common agreement. A primary definition of relevance is the notion of being relevant concerning an objective.

There are several considerations in the literature to characterize feature selection algorithms. Because of them, it is possible to describe this characterization as a search problem in the hypothesis space as follows:

Search Organization: General strategy with which the space of hypothesis is explored

Generation of Successors: Mechanism by which possible variants (successor candidates) of the current hypothesis are proposed

Evaluation Measure: Function by which successor candidates are evaluated, allowing to compare different hypotheses to guide the search process.

Figure 10: Characterization of the feature selection algorithm

Search Organization

This strategy is about the portion of the hypothesis explored for their total number. The search procedure examines only a part of the search space. When a specific state has to be visited, the algorithm uses the information of the previously visited states and eventually heuristic knowledge about non-visited ones.

Generation of Successors

Up to five different operators can be considered to generate a successor for each state: Forward, Backward, Compound, Weighting, and Random.

Forward

This operator adds features to the current solution, among those that have not been selected yet. In each step, the feature that makes the evaluation measure greater is added to the solution.

Backward

This operator removes features from the current solution, among those that have not been removed yet.

In each step, the feature that makes the evaluation measure greater is removed from the solution.

Compound

The idea of this tactic is simple: apply N consecutive forward steps and K consecutive backward ones.

If N > K the net result is a forward operator, otherwise it is a backward one. An interesting approach is to perform the forward or the backward steps, depending on the respective values of the evaluation measure. This allows discovering new interactions among features.

Weighting

In the weighting operators, the search space is continuous, and all of the features are present in the solution to a certain degree. A successor state is a state with a different weighting. This is typically done by iteratively sampling the available set of instances.

Random

This group includes those operators that can potentially generate any other state in a single step. The rest of the operators can also have random components, but they are restricted to some criterion of “advance” in the number of features or in improving the measure evaluation measure at each step.

Evaluation Measure

There are several approaches to evaluate the goodness of a feature subset. It is clear to observe that the relevance of a feature is solely a function of this measure and not a characteristic of the feature itself. Another important consideration is the fact that the range and scale of evaluation measures are immaterial. What counts is that the relative values assigned to different subsets reflect their greater or lesser relevance to the objective function. Among the reviewed measures the probabilistic and the interclass distances, together with consistency, are measures of class separability. Further, the interclass distance, consistency, entropy, and estimations of the probability of error may not require the explicit modeling of probability distributions.

Probability of Error

Provided the ultimate goal is to build a classifier able to correctly labeling instances generated by

the same probability distribution, minimizing the (Bayesian) probability error of the classifier seems to be the most

natural choice. Therefore, it is also a clear choice for evaluation measures.

Divergence

These measures compute a probabilistic distance or divergence among the class-conditional probability densities (Chernoff, Bhattacharyya, Kullback-Leibler, Kolmogorov-Smirnov, Matusita, Patrick-Fisher)

Dependence

These measures quantify how strongly two features are associated with one another, in the sense that knowing the value of one it is possible to predict the value of the other. In the context of feature selection, a feature is better evaluated the better it predicts the class. The correlation coefficient is a classical measure that still finds application.

A somewhat different approach is to estimate the divergence between the class-conditional and the unconditional densities. Any unweighted probabilistic distance measure serves this purpose.

Interclass Distance

These measures are based on the assumption that instances of a different class are distant in the instance space. The most usual distances belong to the Euclidean family. These measures do not require the modeling of any density function, but their relation to the probability of error can be very loose.

Information or Uncertainty

Similarly to the probabilistic dependence, we may observe an instance and compute the a posteriori probabilities to determine how much information on the class of instance has been gained, for its prior probability. If all the classes become roughly equally probable, then the information gain is minimal and the uncertainty (entropy) is maximum. Measures derived from generalizations of Shannon’s entropy as Rényi’s entropy and the entropy of degree alpha. Entropy can also be used without knowledge of the densities as is done in the induction of decision trees, where the information gain is typically computed independently for each feature in the induction process.

Consistency

An inconsistency in X and Y is defined as two instances in Y that are equal when considering only the features in X and that belong to different classes. The aim is thus to find the minimum subset of features leading to zero inconsistencies.

Feature stability

Stability Measures

Feature stability indicates the reproducibility power of the feature selection method. We can say that feature has high stability if multiple feature selection processes agree that this feature is important and low stability when multiple feature selection processes disagree that this feature is important.

Figure 11: Stability measures for feature selection algorithms

Stability by Index

The selected subset of features is represented as a binary vector of size m where 0 represents absence and 1 represents the presence of the feature. The stability is calculated by the amount of overlap between the overall subset of selected features.

Stability by Rank

The correlation between features is evaluated to quantify the stability of the feature selection method using feature ranking. The main drawback of these measures is they cannot handle subsets of features with different cardinality.

Stability by Weight

These measures consider the weight of feature set f while calculating the robustness of the feature selection algorithm. It takes the two sets of weight Wi and Wj for the complete feature set in datasets and returns the correlation between them as the stability. The main drawback of these measures is they cannot deal with the subsets of features of different sizes.

Solution to Instability

Feature Information Strategy

The feature information strategy measures the significance of each feature based on some assessment standards like the accurate measure of the class variables. Then stable features have been selected from these highly important features.

Feature Relatedness

Feature relatedness computes the connection between the features. Covariance-lasso (C-LASSO) calculates the resemblances between features with the help of the feature covariance matrix. It addresses the instability of the L1 norm.

Sample Weighting

The main idea behind this technique is to assign different weights to each sample based on the influence of the sample on the feature relevance. The local profile of feature relevance is used to measure the influence of every sample. Then the feature selection algorithm train on the weighted training set.

Parameter Optimization

To solve the instability problem, the feature selection method is used in the course of the parameter optimization process. The idea is to select an active set of parameters that optimize the current optimization process. The nonlinear regression model investigated with the squared error function and the logistic regression model with the cross-entropy error function.

Intensive Search Approach

Intensive search approaches include parallel search strategies and a Genetic Algorithm (GA). Most of the search processes are stuck at potentially different local maxima, therefore, different search techniques improve the stability by increasing the scope of the search by considering most candidate masks at each decision point. In GA, the initial element is the binary vector of length N containing either 0 or 1. A strong set of features is produced by doing the reproduction of the highest performing variables. This process continues until the best set of features for the search problem is achieved.

Group Feature Selection

The core principle behind this technique is to group the highly correlated features present in high-dimensional datasets which are resistant to the variations of training samples. The stability of the selection process can be improved, if this group is considered as a single entity.

There are two key features in group feature selection:

Feature Group Generation - identifies the groups of associated features. This can be done by Knowledge-driven methods or Data-driven methods. The knowledge-driven method requires deep domain knowledge to form groups and data-driven methods use the information contained in input data for group formation.

Feature Group Transformation: produces a transparent picture of the feature group

Data-driven Group Generation

Data-driven group generation recognizes a group of features using either cluster analysis or density estimation. Instead of relying on domain knowledge of biology, they form groups based on information contained in input data.

Ensemble Feature Selection

The ensemble learning technique works upon the concept of collective intelligence where the large sources of information (people, databases, dictionaries, etc.) collectively combine the knowledge. Ensemble learning paradigm effectively produces a robust and accurate learning solution. Ensemble learning technique based on the bootstrap aggregation that takes the average of several learning techniques builds from random subsamples of the original dataset. An ensemble solution to instability helps in stabilizing the process. It incorporates stability consideration into the designing stage of the algorithm.

Data Perturbation

The recursive feature elimination approach classifies the samples by iteratively eliminating the least influential features. Performance benchmark is done by Precision, Recall, and F-Score. Cross-validation is generally used for error estimation but it shows large variance when applied to small-sized data, therefore, it creates ambiguity in the result.

Function Perturbation

In function perturbation, averaging the outcome of different feature selectors on the same input dataset gives the final result. Different feature selection techniques, instance-level perturbation, feature-level perturbation, stochasticity in the feature selector, etc. incorporate the variation in the feature selectors. Weighted voting can be used for the aggregation of different selection techniques.

No Clear Winner

It should come as no surprise that there is no clear winner in this contest. Moreover, if you try to compute several stability measures you might find the results unstable. This means you might get different values of stability scores, which may actually confuse you more than help you decide the best feature set to eventually select. So, like any other machine learning problem, it takes some experience, judgment, and common sense.

Adding Process Stability

Doing feature selection right is no easy task by itself. Adding the dimension of feature stability on top of it doesn’t make it any easier. However, it is worth paying attention not only to the different types of feature selection strategies that provide the best model score, but also the features that have a strong stability score. I’ve seen more than one project that performs monitoring of post-prediction processes to evaluate how the features that got selected perform over a long period of time. Adding process stability measurements over time might shed more light big picture and give a better feedback loop to perform a better feature selection on your next iteration.

Feature Representation in Financial Crime Domain

Some of NICE Actimize research in data science and machine learning is within feature selection and feature representation. These steps are crucial for the implementation of any machine learning system in general, and in the financial crime domain in particular. In application areas such as fraud detection, these tasks are made more complicated by the diverse, high-dimensional, sparse, and mixed-type data.

Without relying on domain knowledge, selecting the right set of features from data of high-dimensionality for inducing an accurate classification model is a tough computational challenge. It is almost an NP-hard problem as the combinations of features escalate exponentially as the number of features increases.

Unfortunately in the data mining and the financial crime domain, some data are described by a long array of features. Many feature subset selection algorithms have been proposed in the past, but not all of them are effective. Since it takes seemingly forever to use brute force in exhaustively trying every possible combination of features, stochastic optimization may be a solution.

Thus, the time and context-specific nature of the financial data requires domain expertise to properly engineer the features while minimizing any potential information loss. In addition, there is no industry standard of metrics in the financial crime domain. That makes the process of developing feature extraction and feature selection extremely difficult. Especially when defining an objective function for a machine learning model.

Most common metrics in various domains incorporate Euclidean distance which makes sense for spatial geometric data. This has frequent usage in image processing, computer vision, agriculture, healthcare, signal processing, etc.

The financial crime feature space vectors cannot be projected into a geometric plan because they won’t hold any underlying logic. The question is, how should one define the distance between two financial transactions (two high-dimensional, mixed type vectors)

It is necessary to apply domain knowledge while engineering and learning features:

Extract complex indicative features in addition to the simple, raw, self-representative features with absolute values. In more complex cases, when there are categorical values that carry out underlying information about the essence of the financial operation, distance metrics must be defined the way it reflects the real context of the transaction. The human mind can make assumptions on many things in life, such as evaluate time, distance, compare ontologies. It is because human intelligence can understand the context. Machines can not (yet). That’s why the process of feature engineering in financial crime is impossible without having business or product analysts working in synergy with data scientists and machine learning engineers. Once the process of engineering complex indicative features is over, feature space needs to be reduced and refined to come up with only the most representative features to feed the model. The automation process of feature selection might achieve extremely good results, but this is not always the case. The feature selection process can find an optimal subset of features in absolute terms, but there is no guarantee that weak engineering features from (1) were not a part of the initial set of features. The conclusion is inescapable — specific features might exhibit a great impact on the model and be highly influential, but they might poorly reflect the underlying logic of the domain.

Data Scientists at NICE Actimize research hard problems because the financial crime domain is one of the unique domains that imposes difficult challenges with a large scope. That means, our data scientists dare to tackle intricate problems on the daily basis and that makes our work extremely challenging and interesting. It is also rewarding because of the moral satisfaction of knowing you are part of the ‘good guys’ that fight against crime.

I invite you to take that challenge and think, on

i. how would you define proper metrics between two sparse, heterogeneous feature vectors (or tensors) having different cardinalities.

ii. What mechanism would guarantee the verification for valid features. The valid feature is the feature that both has great importance and represents well domain logic.

Reference

A. Jovic et al., (2015) A Review of Feature Selection Methods with Applications

B. Dadaneh et. al., (2016) Unsupervised probabilistic feature selection using ant colony optimization.

B. McKinney, (2013) A Gene-Wise Adaptive-KNN Feature Selection Tool for Finding Gene-Gene Interactions

C. Mohana (2016) A survey on feature selection stability measures

G. Chandrashekar (2014) A survey on feature selection methods

G. Forman(2003) An extensive empirical study of feature selection metrics for text classification

H. Liu et. al., (2013) A hybrid feature selection scheme for mixed attributes data

H. Peng (2005) FS based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy.

Hua Wang, et. al., (2013) Multi-view clustering and feature learning via structured sparsity.

I. Kamkar, et. al., (2015) Exploiting Feature Relationships Towards Stable Feature Selection.

I. Kamkar, et. al., (2015) Stable feature selection for clinical prediction

J Bins, et. al., (2001) Feature Selection from Huge Feature Sets

J. Guo (2018) Dependence guided unsupervised feature selection.

J. Moore, (2007) Tuning ReliefF for Genome-Wide Genetic Analysis.

J. Tang and Huan Liu. (2012) Feature selection with linked data in social media.

J. Zhou, et. al., (2015) A stable feature selection algorithm.

J. Zhou, et. al., (2005) Streaming feature selection using alpha-investing.

L. Molina et al., (2002) Feature Selection Algorithms Survey and Experimental Evaluation

L. Xinwang, (2014) Global and local structure preservation for feature selection.

L. Yu (2004) Efficient Feature Selection via Analysis of Relevance and Redundancy

M. Bennasar, et. al., (2015) Feature selection using Joint Mutual Information Maximization.

M. Ca et. al., (2002) Unsupervised feature selection using feature similarity

M. J. Epstein et. al., (2008) Very large scale ReliefF for genome-wide association analysis

M. Qian (2013) Robust unsupervised feature selection

R. Panthong. (2015) Wrapper FS Subset for Dimension Reduction Based on Ensemble Learning Algorithm.

R. Urbanowicz et. al., (2017) Benchmarking Relief-Based Feature Selection Methods for Bioinformatics Data Mining

S. Alelyani (2013) Feature Selection for clustering: a review

S. Fernandez et al., (2020) Review of Unsupervised Feature Selection Methods

S. Tabakhi (2014) An unsupervised feature selection algorithm based on ant colony optimization.

T. Jirapech-Umpai, (2005) Feature Selection and Classification for Microarray Data Analysis

V. Roth (2004) Feature selection in clustering problems.

X. Wu, et. al., (2013) Online feature selection with streaming features.

Y. Chen, et. al., (2006) Survey and Taxonomy of Feature Selection Algorithms in Intrusion Detection System.

Y. Han et. al., (2012) A variance reduction framework for stable feature selection

Y. Li, et. al., (2017) Recent advances in feature selection and its applications

Y. Sun et. al., (2010) Local-Learning-Based Feature Selection for High-Dimensional Data Analysis

Z. Li (2014) SCAMS: Simultaneous clustering and model selection.

Z. Li (2015) Unsupervised feature selection via nonnegative spectral analysis and redundancy control.

Z. Li, et. al., (2012) Unsupervised Feature Selection Using Nonnegative Spectral Analysis.

Z. Zhao and Huan Liu. (2007) Spectral feature selection for supervised and unsupervised learning