Exploring the Gap Between Variable Selection and Dimensionality Reduction

The Problem: This project addresses the gap between variable selection algorithms and dimensionality reduction algorithms. Variable selection algorithms are designed to produce sparse solutions where only few variable are marked as relevant variables. This is not suitable for highly correlated data such as gray values of an image. Dimensionality reduction algorithms (e.g PCA) tend to combine all variables, which makes selection of the significant variables difficult. For this reason, it is not appropriate to apply PCA on a set of complex informative features (e.g wavelets). The problem is how to create a hybrid system which can select from complex features and still collapse the data to only few relevant directions. Motivation: A recently introduced unsupervised variable selection algorithm , the [5] algorithm, was designed to robustly select few relevant variables (”features”) out of many irrelevant ones. It was shown to work better than other unsupervised algorithms, and comparably well to supervised algorithms on synthetic datasets and on real vision and genomics datasets [6]. The algorithm is not suitable for application to raw image data nearby pixel values are highly correlated, each one containing a very limited amount of information. In such cases, the algorithm does not prefer one pixel over another. Similarly to supervised algorithm, the deals best with more informative features such as wavelets [2]. In the unsupervised case most algorithms in general do not gain from using more informative variables. Principle component analysis (PCA), and other dimensionality reduction algorithms, are based on combining variables to maximize some score. On a set of more informative variables these algorithms would still combine variables. In fact, no benefit is gained by just using informative features. Hence, and PCA, as representatives of larger classes of ”variable selection” and ”variable combination” algorithms have opposing advantages and disadvantages. PCA thrives of correlation between variables, but fails to select informative features from a set of more complex features. fails where all variables are correlated, but does well with informative variables. We seek to construct a method which enjoys advantages of both. Previous Work: Second generation dimensionality reduction algorithms (e.g. [4]) give more attention to preserving the structure of the local neighborhood around each data point, but are still largely based on eigendecomposition of some affinity matrix. In building these affinity matrices all variables are considered equal. Variable selection is usually done in the supervised setting, where it is easy to compute a score for each variable independently of other variables. Scores vary from simple mutual information scores to probabilistic ones (e.g. [3]). The algorithm uses spectral properties of the affinity matrix [1] in order to weight the relevancy of the variables in the unsupervised setting. It thus belongs to a very successful family of ”spectral algorithms”. These are used in many areas, from approximation to NP-hard problems to clustering. The way assigns weights to the variables is based on optimizing a criteria which is directed toward making the data well clustered into coherent and compact clusters. In this work we take a related, but different approach. We want to select our variables as to empower the next stage of dimensionality