BACKGROUND
high-dimensional data sets, where the observed variables often by far outnumber the samples, are becoming increasingly prevalent in many scientific domains.
as an example, microarrays are used extensively to measure a variety of genomic attributes such as gene expression and dna copy numbers.
a peculiar feature of many high-dimensional data sets is that they are often collected with an exploratory aim, without a specific hypothesis in mind.
this means, among other things, that a data set may contain many variables that are not really informative, while the informative structure is contained in a small subset of the variables.
the presence of non-informative variables can have detrimental effects on the possibility to extract relevant biological knowledge from the observed data, and the performance of many machine learning algorithms may drastically improve if the number of variables is reduced before or in conjunction with the application of the algorithm.
the exploratory perspective on high-dimensional data means that visualization methods, providing a graphical representation of the data, can be of great assistance.
one of the most commonly used visualization methods for microarray data is principal component analysis   <cit>  which provides coupled, low-dimensional representations of the samples and variables in a data set.
pca is applicable also to very high-dimensional data sets, but the resulting visualization can be severely confounded and the truly informative structure can be concealed by the presence of a large number of non-informative variables.
hence, variable selection can be of great importance to improve the interpretability of pca visualizations.
in the microarray context, one commonly used approach to reduce the dimensionality is to simply exclude the variables with the lowest variances before applying pca  <cit> , implicitly assuming that they do not contain any useful information concerning the samples in the data set.
such variable filtering often provides more informative and easily interpretable pca representations.
other ways to reduce the dimension before pca are also common, such as excluding the variables with the lowest signal intensities  <cit>  or including only those variables that are highly related to a specific response  <cit> .
naturally, the resulting visualization may be highly dependent on the number of included variables and, hence, on the variance or significance threshold used as the inclusion criterion.
to our knowledge, there exist no well-motivated, objective criteria that are useful for obtaining good stopping rules in such variable filterings, and therefore most studies apply some kind of ad-hoc criterion.
in this paper, we present the projection score, which is a straightforward, intuitively appealing measure of the informativeness of a variable subset with respect to visualization.
intuitively, the projection score compares the variance captured by a pre-defined set of principal components to the expected value under the assumption that the variables are independent, that is, that there is no non-random structure in the data.
the projection score can be universally applied to provide a stopping rule for a variable filtering, and hence it can be used to systematically find the most interpretable pca visualizations in practical exploratory analysis applications, for example in microarray data analysis.
moreover, as we will see in this paper, the projection score can be applied in more general contexts than those indicated above, to compare the informativeness of any variable subsets with respect to visualization by pca.
related work
many variable selection techniques have been devised for use with supervised learning algorithms, such as classifiers or predictors.
in the supervised learning context, measuring the informativeness of a variable subset is fairly straightforward.
probably the most common approach is to use some kind of cross-validation scheme to train the classifier on the samples in a training set and define the informativeness of the variable subset based on the performance of the obtained classifier in an independent test set  <cit> .
for unsupervised learning methods, such as visualization and clustering, which do not make use of any class labels or other response variables, this approach is not applicable.
variable selection for model-based clustering, where the data are assumed to come from a mixture of several subpopulations, has been considered  <cit> .
in this case, the value of the objective function depends on how well the data comply with the assumed mixture model.
for visualization purposes however, it is less clear what constitutes an optimal result.
the projection score defined in this paper measures the informativeness of a variable subset based on the variance accounted for by a subset of the principal components of the variable subset, without using any side information such as the class memberships of samples.
variable selection in the context of pca has been considered for many years, but often with the aim of approximating the original data set as well as possible with only a subset of the variables.
moreover, the proposed methods do not in general address the problem of finding a suitable stopping criteria for variable filtering.
the potential advantage of sparse components, in terms of interpretability, were recognized already in the 1970s  <cit> .
one common approach to variable selection for pca has been to fix the desired number of variables in advance and search for the optimal variable subset of this pre-selected cardinality  <cit> .
to comply with the original goals of pca, the optimal variable subset is often defined as the one containing the largest amount of variance, or providing the best approximation to the original data set.
hence, one searches for a subset of the variables containing essentially the same information as the entire variable set.
this goal is common also to many of the recently proposed sparse pca methods  <cit> .
these methods often take their starting point in one of the optimality properties of ordinary pca and introduce a penalty  to limit the number of non-zero principal component weights.
the algorithms are in general developed for a fixed penalty  and the optimal penalty is determined by cross-validation, trying to approximate a test set as well as possible, or by ad-hoc methods.
in contrast to these methods, the main objective of the approach proposed in this paper is to compare subsets of different sizes, such as, for example, subsets obtained by variance filtering with different inclusion thresholds.
our aim is not explicitly to approximate the original data set as well as possible, but rather to find informative structures which may not be apparent by considering the entire data set.
by filtering the variable set with respect to a specific factor  we obtain sparse principal components where the included variables are all related to the same factor.
these may be easier to interpret than general sparse principal components.
with our approach, it is also possible to compare the informativeness contained in different factors, by filtering with respect to the association with each one of them and comparing the best projection scores obtained for each factor.
another related approach for variable selection, or variable clustering, using pca is the gene shaving procedure  <cit> .
this procedure starts by computing the first principal component of the data.
in each step, the variables with the smallest loadings are shaved off, and a new principal component is computed from the remaining variables.
this yields a nested sequence of variable subsets.
for each subset, a quality measure is computed as the ratio between the variance of the mean value of the expression levels of the genes in the subset and the total variance of the subset.
then, this quality measure is compared to what would be expected from random data, using the gap statistic .
the variable subset with the highest value of the gap statistic is considered to be the optimal variable subset .
we will show that one way to use the projection score is to obtain another quality measure of the variable subsets that can be used in such a shaving approach, and that the optimal variable set is not necessarily the same as with the method described in  <cit> .
biclustering methods based on sparse matrix decompositions have been proposed by several authors .
here, the aim is to find a subset of the variables which are correlated across a subset of the samples and hence sparsity is induced for both samples and variables.
the biclustering methods use different measures to evaluate subsets of a given data set.
for example, in  <cit>  the quality of a submatrix is defined based on the average expression value in the submatrix.
biclustering methods have been shown to be useful for finding informative patterns which are not necessarily present across the entire data set.
however, they are not explicitly optimized for visualization, and, as the sparse pca methods, they are not used to find stopping criteria for variable filtering.
another variable selection criterion for pca was described in  <cit> .
for each variable, the authors compute the difference between the entropy of the entire data set and the entropy of the data set with the variable removed.
this is used as a measure of the contribution of each of the variables, and the variables are ranked according to their contributions.
the optimal variable subset is considered to consist of all variables whose contributions are more than one standard deviation higher than the mean value of all contributions.
in contrast to this method, we compute the projection score for a large number of variable subsets and select the one with the highest value as the most informative in a visualization context.
RESULTS
in this section, we will first apply different filtering techniques, with varying inclusion criteria, to generate a collection of variable subsets from each of three microarray data sets.
the projection score will be applied to find the most informative variable subset in each example.
it is important to note that the informativeness is measured with respect to unsupervised, exploratory analysis by pca, where the aim often is generation of new hypotheses rather than validation of existing hypotheses.
this means that we could use the projection score as a quality measure even in the absence of any side information about, for example, sample groups.
it can also be used to quantify and compare the informativeness of variable subsets obtained by supervised methods, for example subsets consisting of the variables most highly related to a given response.
regardless of how the variable subsets are obtained, the projection score evaluates their informativeness from an unsupervised perspective, based on the variance contained in a pre-defined subset of the principal components of the respective variable subsets.
in all cases, when we vary the inclusion criterion , the projection score traces out a reasonably smooth curve, often with a clear maximum, which means that it is reasonable to say that there is indeed a maximally informative subset in the proposed sense.
in this article, this curve will be referred to as the projection score curve for the filtering parameter.
in the first example, we filter the variable set by variance and find an informative subset of variables with high variances, providing a graphical sample representation which is more easily interpretable than the one obtained from all variables.
in the second example, we filter the variable set by the association with given responses, and show that the optimal projection score and the shape of the projection score curve obtained from filtering with respect to a certain response capture the overall evidence in the data for a significant association with that response.
we also show the results from a combined variance and response-related filtering.
in the third example, we apply a shaving procedure to generate variable subsets, and evaluate the obtained subsets by their projection scores.
then, we validate the use of the projection score with synthetic examples, where we show that the variable subset with the highest projection score is often the one containing the non-random structure in the data.
finally, we discuss some issues regarding the estimation of the projection score and warn against a potential pitfall.
in all examples, we use the projection score to compare the informativeness of variable subsets with respect to visualization.
assume that we are given a data set with n samples and p variables, represented by a rank-r matrix x ∈ ℝp×n, and a collection of m variable subsets rm ⊆ { <dig>  ..., p} for m =  <dig>  ..., m. we define selection functions ϕm in such a way that  consists only of the rows in x with indices in rm.
to compute the projection score, we must also select an s ⊆ { <dig>  ..., r}, essentially representing the number of degrees of freedom we expect in the data.
letting Λx =  denote the vector of non-zero singular values of x in decreasing order, the fraction of the variance in x which is accounted for by the principal components with indices in s can be computed as
the projection score of rm is then defined as
here,  denotes the inferred distribution of ϕm under the assumption of independence among the original samples and variables in x. we compute the projection score for each of the m variable subsets, and the subset with the largest projection score is considered the most informative variable subset for visualization.
for further details, see the methods section.
to obtain a visualization which reflects the correlation structure between the variables of the data set instead of the individual variances, we consistently extract the singular values from standardized data matrices, where each variable is mean-centered and scaled to unit variance.
this standardization is indeed commonly used  <cit> .
if we had not standardized the data, the correlations between the variables would be less influential when computing the principal components, and the variances of the individual variables could potentially have a very large impact.
since we define the projection score by comparing the observed data to data generated under a null model defined by assuming independence between the variables, we may argue that the non-random structure that is detected with the resulting score is that corresponding to correlations between variables and that therefore, the standardized data are better suited for our purposes.
note that for the variance- and response-related filterings, the variable rankings are extracted from the original, unstandardized data.
variance filtering of a lung cancer data set
we first use the projection score to find particularly informative variable subsets among those obtained by applying variance filters to the lung cancer data set studied in  <cit> .
the data set contains gene expression measurements for  <dig>  genes in  <dig> subjects.
the subjects belong to four groups: normal , pulmonary carcinoid tumors , small cell carcinoma  or colon metastases .
for a given set of variance thresholds  , we define rm as the collection of variables with variance exceeding θm.
the variance thresholds are specified as fractions of the highest individual variable variance in the data set, meaning that the variables included in the variable subset rm are those with variances exceeding θm · varmax where varmax denotes the maximal variance among the variables in the data set.
we choose s = { <dig>   <dig>  3}, that is, we search for a variable subset providing an informative three-dimensional sample representation, which is reasonable in a visualization context.
figure  <dig> shows the three-dimensional sample representation obtained by applying pca only to the variables in the most informative subset.
the representation based on the most informative subset is considerably more easily interpretable than the representation based on the entire set of variables, which is shown in figure  <dig>  in the representation based on the most informative variable subset, the pulmonary carcinoid tumors  appear to cluster into several subgroups, an effect which was also noted in  <cit> .
figure  <dig> shows the projection score curve for the filtering parameter θ. very small variable subsets, corresponding to high values of the variance threshold, do not support the chosen s .
the projection score attains its maximal value of τ =  <dig>  for a variance threshold of  <dig> % of the maximal variance, corresponding to a variable subset consisting of the  <dig> variables with highest variances.
in figure  <dig>  we show a heatmap for the expression matrix corresponding to the most informative variable subset.
clearly, the variables with the highest variances contain much information about the four sample groups, which is not surprising.
response-related filtering of the nci- <dig> data set
in this section, we construct ϕm to consist of the variables which are most highly related to a given response variable.
in the studied examples the response variable indicates the partition of the samples into different groups.
given such a partition, we calculate the f-statistic, contrasting all these groups, for each variable.
for a given set of significance thresholds , all variables which are significantly related to the response at the level αm  are included in ϕm.
for each randomized data set x* used to estimate   we define the significance thresholds  in such a way that the resulting variable subsets have the same cardinalities as the corresponding subsets from the original data set.
the choice of s is guided by the underlying test statistic.
when we contrast only two groups, we do not expect the optimal variable subset to support more than a one-dimensional sample configuration, and therefore we choose s = {1} in this case.
when contrasting more than two groups, the choice of s must be guided by other criteria.
this is because the variables with the highest f-score may in this case very well characterize many different sample groups, not all of which can simultaneously be accurately visualized in low dimension.
the nci- <dig> data set  <cit>  contains expression measurements of  <dig>  genes in  <dig> cell lines from nine different types of cancers.
we first filter the variable set with respect to the association with the partition defined by all the nine cancer types, using s = { <dig>   <dig>  3}.
this gives a most informative subset consisting of  <dig> variables, with a projection score τ =  <dig> .
the resulting sample representation, obtained by applying pca to the most informative variable subset, is shown in figure  <dig>  the representation based on all variables is shown in figure  <dig> and the projection score is shown in figure  <dig> as a function of log <dig>  figure  <dig> shows the p-value distribution, which indicates that there are indeed variables which are truly significantly associated to the response.
figure  <dig> shows a heatmap for the most informative variable subset ) where it can be seen that the samples are reasonably well clustered according to cancer type based on these  <dig> variables only.
next, we set out to study how informative the contrasts between one cancer type and all the others are.
we filter the variable set using the association with the contrast of one cancer type vs the rest, using s = {1}.
figure  <dig> shows some of the projection score curves.
first, we can note that the range of p-values, as well as the range of obtained projection scores, are highly different for the different contrasts.
the highest projection scores in the respective cases are  <dig>  ,  <dig>  ,  <dig>   and  <dig>  .
apparently, for each of the melanoma, leukemia and renal contrasts, a small subset of the variables related to the respective response contains a lot of non-random information.
however, for the nsclc contrast the full variable set  is the most informative.
this can be understood from figure  <dig>  which shows a histogram over the p-values obtained from the f-test contrasting the nsclc group with the rest of the samples.
the p-values are essentially uniformly distributed, indicating that there are no truly differentially expressed genes in this case.
hence, in the filtering process we do not unravel any non-random structure, but only remove the variables which are informative in other respects.
figure  <dig> shows the p-value distribution for the melanoma contrast.
in this case, there are indeed some differentially expressed genes, which means that in the filtering process, we purify this signal and are left with an informative set of variables.
the projection scores obtained from the different contrasts are consistent with figure  <dig>  in the sense that the highest projection scores are obtained from the contrasts corresponding to the cancer types which form the most apparent clusters in this sample representation, that is, the melanoma samples and the leukemia samples.
the nsclc samples do not form a tight cluster and are not particularly deviating from the rest of the samples in figure  <dig>
combined variance and response-related filtering of the nci- <dig> data set
in some cases, the variable set is first filtered by variance before a statistical test is applied.
removing supposedly non-informative variables in this way may be beneficial in terms of statistical power, since if the number of performed tests is high, we need to do a rather heavy correction for multiple comparisons.
also with the approach proposed in this paper, we can combine different filtering procedures to find a variable subset which is informative from more than one perspective.
here, we show how to combine variance filtering with response-related filtering for the nci- <dig> data set.
in this way, we exclude all variables that obtain small p-values from the f-test mostly due to their low variances.
as before, we define a collection of variance thresholds θm and a number of significance thresholds αm, for the f-test contrasting all nine cancer types.
we choose s = { <dig>   <dig>  3} as before.
now, the projection score can be represented as a surface, shown in figure  <dig>  for varying values of the variance and p-value thresholds.
the curves traced out for log <dig> =  <dig> and θ =  <dig> correspond to the projection score curves for the individual variance and response-related filterings, respectively.
as can be seen in figure  <dig>  we get the maximal projection score for a variable subset obtained by a combination of variance filtering and response-related filtering.
this subset includes all variables with a variance exceeding  <dig> % of the maximal variance, and with a p-value below  <dig>  · 10- <dig>  in total  <dig> variables.
the maximal projection score is τ =  <dig> .
this can be compared to the purely response-related filtering, which gave a maximal projection score of τ =  <dig>  by including the  <dig> variables with p-values below  <dig>  · 10- <dig>  figure  <dig> shows the sample configuration obtained by applying pca to the most informative variable subset from the combined filtering.
shaving of a leukemia data set
here, we will use the projection score to evaluate the informativeness of variable subsets obtained by the shaving procedure described in  <cit> .
briefly, for a given fraction π ∈ , we define a function ξ : 2{ <dig> ...,p} → 2{ <dig> ...,p} by letting ξ consist of the fraction  <dig> - π of the variables in r having the highest loadings in the first principal component of the data set consisting of the variables in r. the selection functions ϕm are then defined by letting ϕm contain all variables in .
we use π =  <dig> , hence in each step shaving off 2% of the variables, continuing until only one variable remains.
we compute the projection score for each of the variable subsets, as well as the gap statistic proposed in  <cit>  .
we apply the described method to the leukemia data set studied in  <cit> .
the data set contains gene expression measurements from  <dig>  genes in  <dig> samples with either aml  or all .
computing the projection score and the gap statistic for the variable subsets obtained by shaving gives optimal variable subsets containing  <dig> variables  and  <dig> variables , respectively.
figure  <dig> shows the two informativeness measures as functions of the variable subset cardinality.
both curves are smooth and have clear extreme points.
figures  <dig> and  <dig> show heatmaps for the optimal variable subsets from the two methods.
as can be seen in these figures, the two subsets contain much the same information about the samples in the data set.
the variable subset selected by the projection score is larger than the one selected by the gap statistic.
in some situations, a small number of included variables may be beneficial.
however, as will be shown , the gap statistic tends to underestimate the number of variables in the optimal subset , which may potentially be the case also in this example.
this would then lead to a number of "false negatives", that is, variables falsely excluded from the optimal combination.
the median and range  of the number of non-zero elements included in the optimal variable subset found by different methods.
each underlying component contains  <dig> non-zero elements.
spc - sparse pca  <cit> .
ssvd - sparse svd  <cit> .
the gap statistic is defined as in  <cit> .
the median and range  of the number of non-zero elements included in the optimal variable subset found by different methods.
each underlying component contains  <dig> non-zero elements.
spc - sparse pca  <cit> .
ssvd - sparse svd  <cit> .
the gap statistic is defined as in  <cit> .
validation on synthetic data
in this section, we will validate the projection score by using it to find informative variable subsets from different filtering processes applied to synthetic data sets.
variance filtering
we let
and generate a synthetic data set with  <dig>  variables and  <dig> samples by letting
the only non-random structure in the data is contained in the first  <dig> variables, discriminating between two groups of  <dig> samples each.
by varying σ <dig> we obtain data sets with difference variance properties.
with σ <dig> =  <dig> , the informative variables and the non-informative variables have comparable variances.
with σ <dig> =  <dig> , the informative variables obtain a lower variance than the non-informative variables and with σ <dig> =  <dig>  the informative variables are also those with the highest variances.
we take s = {1}, since no other choice of s is supported by any variable subset.
this is also consistent with the structure of the data matrix.
across  <dig> realizations, the mean number of variables in the subset giving the best projection score are  <dig>  for σ <dig> =  <dig>  ,  <dig>  for σ <dig> =  <dig>   and  <dig>  for σ <dig> =  <dig>  .
the projection score correctly indicates that when σ <dig> =  <dig> , the informative structure in the data is indeed related to the variables with lowest variances, and hence all variables are included in the most informative subset .
note that the association between the variables within each sample group is very strong when σ <dig> =  <dig> .
if the variables with lowest variance had been routinely filtered out in this example, we would lose the informativeness in the data.
it can also be noted that when the number of variables is below a certain threshold  in the σ <dig> =  <dig>  case, not even s = {1} is supported by the data since we have filtered out all the informative variables.
for σ <dig> =  <dig> , the optimal number of variables is highly dependent on the specific data set since the filtering removes both informative and non-informative variables uniformly.
response-related filtering
in this example, we simulate a data matrix containing two group structures .
the data set consists of  <dig> samples which are divided into four consecutive groups of  <dig> samples each, denoted a, b, c, d. we define
the data matrix is then generated by letting
we perform a two-sided t-test contrasting a ∪ c and b ∪ d and order the variables by the absolute value of their t-statistic for this contrast.
in this case, since we compare only two groups we are essentially searching for a one-dimensional separation, so we choose s = {1}.
figure  <dig> shows the structure of the data set.
the data set contains one very strong factor, encoded by the first  <dig> variables, and one weaker factor, the one we are interested in, which is related to the next  <dig> variables.
figures  <dig> and  <dig> show the projection score for different thresholds on the significance level α and for different variable subset cardinalities, respectively.
the optimal projection score  is obtained for variable subsets containing slightly less than  <dig> variables .
hence, even though there is indeed much information contained in the entire variable set, it is possible to obtain an even more informative variable subset by filtering.
projecting onto the first principal component based only on the variables in the most informative subset clearly discriminates between a ∪ c and b ∪ d, as shown in figure  <dig>  as above, we can compare the maximal projection score corresponding to filtering by the association with different responses.
filtering with respect to the association with the contrast between a ∪ b and c ∪ d, that is, the stronger factor in the data set, gives a maximal projection score around  <dig> .
hence, this correctly suggests that this factor is more informative than the previously studied.
filtering with respect to the variance, still using s = {1}, gives a maximal projection score of  <dig> , obtained for approximately  <dig> variables .
this implies that the variables with high variance in this case are even more informative than the variables closely associated with one of the responses, in the sense that the encoded information deviates more from what would be expected from the highly varying variables in a randomized data set.
applying variance filtering with s = { <dig>  2} provides a most informative variable subset containing  <dig> variables, capturing parts of both the two informative variable groups in the data .
s = { <dig>   <dig>  3} is not supported by any variable subset.
detecting sparsity in principal components
in this example, we will evaluate the usefulness of the projection score for detection of sparsity in the leading principal components of a data matrix.
we generate a data set, with p =  <dig> variables and n =  <dig> samples, following the scheme given in  <cit> .
briefly, we form a matrix v =  from an orthonormal set of p principal components v <dig>  ..., vp ∈ ℝp, and a matrix c = diag containing the eigenvalues in decreasing order on the diagonal.
then, we form the covariance matrix Σ by
to generate data, we let z be a random draw from  and take x = vc1/2z.
the first two principal components  are constructed to have specific sparsity patterns.
in the first example, we let v <dig> and v <dig> contain  <dig> non-zero elements each  and choose c <dig> =  <dig>  c <dig> =  <dig>  we let ck =  <dig> for k =  <dig>  ...,  <dig> and sample the entries of the corresponding principal components uniformly between  <dig> and  <dig> before orthogonalizing and normalizing them.
in the second example, we let v <dig> and v <dig> instead contain  <dig> non-zero  elements.
we choose c <dig> =  <dig> and c <dig> =  <dig> and proceed as in the previous example to obtain the rest of the components.
the variable subsets are obtained by the shaving procedure, as outlined above and described in  <cit> .
this gives a nested sequence of variable subsets, which can be evaluated in terms of their projection scores.
we also compute the gap statistic as defined in  <cit>  as another quality measure of each variable subset.
the optimal variable subsets obtained from these methods are compared to those found by a sparse pca algorithm  <cit>  and the sparse svd proposed in  <cit> .
it should be noted that the aims of these methods are somewhat different.
the sparse pca attempts to find a good approximation of the original data set using only a subset of the variables, while the sparse svd was developed for biclustering, that is, finding small groups of variables which are related across possibly only a subset of the samples.
hence, also the sample representation can be sparse  in the results from the sparse svd.
in the examples studied here, however, the sparsity patterns of the principal components are designed to be identical across all samples.
the sparse svd and the sparse pca were applied through the r programs provided by the respective authors.
the sparse svd was applied with γu = γv =  <dig>  as suggested by the authors.
the sparsity parameter in the sparse pca was estimated via the cross-validation function provided with the r package, evaluating  <dig> different sparsity levels between  <dig> and  via 5-fold cross-validation.
estimating
to obtain , we repeatedly permute the values in each row of x and perform the variable filtering, which is computationally expensive .
a more efficient implementation can be obtained if we specify the distribution  for each m =  <dig>  ..., m in advance.
then, the values of  can be estimated in advance and stored, which leaves only the calculations for the observed data matrix and the subtraction of a known quantity for each variable subset.
for instance, we can fix  by assuming that each element in ϕm is drawn from a standard normal distribution, that is,
for i =  <dig>  ..., |rm| and j =  <dig>  ..., n. we can then calculate the expected value of  for a large collection of choices of variable subset cardinalities and sample sizes.
figure  <dig> shows an interpolated surface for  <dig> ≤ |rm| ≤  <dig>   <dig> and  <dig> ≤ n ≤  <dig>  this more computationally efficient approach may be used for example for variance filtering, if the observed data matrix is standardized before the principal components are extracted.
when the variable subsets are defined by response-related filtering, it is more difficult to specify and sample from the truncated distribution resulting from the filtering.
in particular, we note that even for a data matrix x containing only independent variables, we still expect to see an aggregation of the variance in the first principal components when we consider small submatrices ϕm.
in some cases, however, it may still be of interest to compare the singular values from the observed data to those from matrices with a given, known distribution such as the one described above.
we next give a small example to show the different conclusions that can result if different definitions of  are used.
example
we define x ∈ ℝ <dig>  ×  <dig> by letting  for i =  <dig>  ...,  <dig>  000; j =  <dig>  ...,  <dig>  we divide the samples into four consecutive groups a, b, c, d of  <dig> samples each, as in the response-related filtering above, and filter the variable set based on the absolute value of the t-statistic contrasting groups a∪c and b∪d, using s = {1}.
if  is defined by assuming that each element of ϕm is drawn from a standard normal distribution, the most informative variable subset contains  <dig> variables.
here, even though we study a completely random matrix and an artificial grouping of the samples, we obtain a good separation between the groups.
however, the projection score is not very high in this case .
figure  <dig> shows the projection score as a function of the significance threshold, and the optimal projection.
as can be seen, just looking at the visualization in the right panel we might be led to believe that there is actually some non-random structure in the data.
on the other hand, if we define  as we have done in the previous examples, by assuming independence among the samples and variables of the original matrix x and then filtering, no submatrix supports even s = {1}, and hence we get an indication that we are filtering a matrix without non-random relationships between the variables.
CONCLUSIONS
in this paper, we have introduced and shown the usefulness of the projection score, a measure of the informativeness of a subset of a given variable set, based on the variance contained in the corresponding sample configuration obtained by pca.
the definition of the projection score is straightforward and the interpretation in terms of captured variance is intuitively appealing in a pca visualization context.
moreover, the projection score allows a unified treatment of variable selection by filtering in the context of visualization, and we have shown that it indeed gives relevant results for three different filtering procedures, both for microarray data and for synthetic data.
by filtering with respect to a specific factor, we obtain sparse principal components where all variables receiving a non-zero weight are indeed strongly related to the chosen factor.
in this respect, the resulting components may be more easily interpretable than general sparse principal components, where the variables obtaining a non-zero weight can be related to many different factors.
the optimal number of variables included in the extracted principal components often increases with |s| and in many cases small variable subsets do not even support large subsets s.

 high-dimensional data sets, where the observed variables often by far outnumber the samples, are becoming increasingly prevalent in many scientific domains. as an example, microarrays are used extensively to measure a variety of genomic attributes such as gene expression and dna copy numbers. a peculiar feature of many high-dimensional data sets is that they are often collected with an exploratory aim, without a specific hypothesis in mind. this means, among other things, that a data set may contain many variables that are not really informative, while the informative structure is contained in a small subset of the variables. the presence of non-informative variables can have detrimental effects on the possibility to extract relevant biological knowledge from the observed data, and the performance of many machine learning algorithms may drastically improve if the number of variables is reduced before or in conjunction with the application of the algorithm. the exploratory perspective on high-dimensional data means that visualization methods, providing a graphical representation of the data, can be of great assistance. one of the most commonly used visualization methods for microarray data is principal component analysis which provides coupled, low-dimensional representations of the samples and variables in a data set. pca is applicable also to very high-dimensional data sets, but the resulting visualization can be severely confounded and the truly informative structure can be concealed by the presence of a large number of non-informative variables. hence, variable selection can be of great importance to improve the interpretability of pca visualizations. in the microarray context, one commonly used approach to reduce the dimensionality is to simply exclude the variables with the lowest variances before applying pca, implicitly assuming that they do not contain any useful information concerning the samples in the data set. such variable filtering often provides more informative and easily interpretable pca representations. other ways to reduce the dimension before pca are also common, such as excluding the variables with the lowest signal intensities or including only those variables that are highly related to a specific response. naturally, the resulting visualization may be highly dependent on the number of included variables and, hence, on the variance or significance threshold used as the inclusion criterion. to our knowledge, there exist no well-motivated, objective criteria that are useful for obtaining good stopping rules in such variable filterings, and therefore most studies apply some kind of ad-hoc criterion. in this paper, we present the projection score, which is a straightforward, intuitively appealing measure of the informativeness of a variable subset with respect to visualization. intuitively, the projection score compares the variance captured by a pre-defined set of principal components to the expected value under the assumption that the variables are independent, that is, that there is no non-random structure in the data. the projection score can be universally applied to provide a stopping rule for a variable filtering, and hence it can be used to systematically find the most interpretable pca visualizations in practical exploratory analysis applications, for example in microarray data analysis. moreover, as we will see in this paper, the projection score can be applied in more general contexts than those indicated above, to compare the informativeness of any variable subsets with respect to visualization by pca. related work many variable selection techniques have been devised for use with supervised learning algorithms, such as classifiers or predictors. in the supervised learning context, measuring the informativeness of a variable subset is fairly straightforward. probably the most common approach is to use some kind of cross-validation scheme to train the classifier on the samples in a training set and define the informativeness of the variable subset based on the performance of the obtained classifier in an independent test set. for unsupervised learning methods, such as visualization and clustering, which do not make use of any class labels or other response variables, this approach is not applicable. variable selection for model-based clustering, where the data are assumed to come from a mixture of several subpopulations, has been considered. in this case, the value of the objective function depends on how well the data comply with the assumed mixture model. for visualization purposes however, it is less clear what constitutes an optimal result. the projection score defined in this paper measures the informativeness of a variable subset based on the variance accounted for by a subset of the principal components of the variable subset, without using any side information such as the class memberships of samples. variable selection in the context of pca has been considered for many years, but often with the aim of approximating the original data set as well as possible with only a subset of the variables. moreover, the proposed methods do not in general address the problem of finding a suitable stopping criteria for variable filtering. the potential advantage of sparse components, in terms of interpretability, were recognized already in the 1970s. one common approach to variable selection for pca has been to fix the desired number of variables in advance and search for the optimal variable subset of this pre-selected cardinality. to comply with the original goals of pca, the optimal variable subset is often defined as the one containing the largest amount of variance, or providing the best approximation to the original data set. hence, one searches for a subset of the variables containing essentially the same information as the entire variable set. this goal is common also to many of the recently proposed sparse pca methods. these methods often take their starting point in one of the optimality properties of ordinary pca and introduce a penalty to limit the number of non-zero principal component weights. the algorithms are in general developed for a fixed penalty and the optimal penalty is determined by cross-validation, trying to approximate a test set as well as possible, or by ad-hoc methods. in contrast to these methods, the main objective of the approach proposed in this paper is to compare subsets of different sizes, such as, for example, subsets obtained by variance filtering with different inclusion thresholds. our aim is not explicitly to approximate the original data set as well as possible, but rather to find informative structures which may not be apparent by considering the entire data set. by filtering the variable set with respect to a specific factor we obtain sparse principal components where the included variables are all related to the same factor. these may be easier to interpret than general sparse principal components. with our approach, it is also possible to compare the informativeness contained in different factors, by filtering with respect to the association with each one of them and comparing the best projection scores obtained for each factor. another related approach for variable selection, or variable clustering, using pca is the gene shaving procedure. this procedure starts by computing the first principal component of the data. in each step, the variables with the smallest loadings are shaved off, and a new principal component is computed from the remaining variables. this yields a nested sequence of variable subsets. for each subset, a quality measure is computed as the ratio between the variance of the mean value of the expression levels of the genes in the subset and the total variance of the subset. then, this quality measure is compared to what would be expected from random data, using the gap statistic. the variable subset with the highest value of the gap statistic is considered to be the optimal variable subset. we will show that one way to use the projection score is to obtain another quality measure of the variable subsets that can be used in such a shaving approach, and that the optimal variable set is not necessarily the same as with the method described in. biclustering methods based on sparse matrix decompositions have been proposed by several authors. here, the aim is to find a subset of the variables which are correlated across a subset of the samples and hence sparsity is induced for both samples and variables. the biclustering methods use different measures to evaluate subsets of a given data set. for example, in the quality of a submatrix is defined based on the average expression value in the submatrix. biclustering methods have been shown to be useful for finding informative patterns which are not necessarily present across the entire data set. however, they are not explicitly optimized for visualization, and, as the sparse pca methods, they are not used to find stopping criteria for variable filtering. another variable selection criterion for pca was described in. for each variable, the authors compute the difference between the entropy of the entire data set and the entropy of the data set with the variable removed. this is used as a measure of the contribution of each of the variables, and the variables are ranked according to their contributions. the optimal variable subset is considered to consist of all variables whose contributions are more than one standard deviation higher than the mean value of all contributions. in contrast to this method, we compute the projection score for a large number of variable subsets and select the one with the highest value as the most informative in a visualization context.  in this section, we will first apply different filtering techniques, with varying inclusion criteria, to generate a collection of variable subsets from each of three microarray data sets. the projection score will be applied to find the most informative variable subset in each example. it is important to note that the informativeness is measured with respect to unsupervised, exploratory analysis by pca, where the aim often is generation of new hypotheses rather than validation of existing hypotheses. this means that we could use the projection score as a quality measure even in the absence of any side information about, for example, sample groups. it can also be used to quantify and compare the informativeness of variable subsets obtained by supervised methods, for example subsets consisting of the variables most highly related to a given response. regardless of how the variable subsets are obtained, the projection score evaluates their informativeness from an unsupervised perspective, based on the variance contained in a pre-defined subset of the principal components of the respective variable subsets. in all cases, when we vary the inclusion criterion, the projection score traces out a reasonably smooth curve, often with a clear maximum, which means that it is reasonable to say that there is indeed a maximally informative subset in the proposed sense. in this article, this curve will be referred to as the projection score curve for the filtering parameter. in the first example, we filter the variable set by variance and find an informative subset of variables with high variances, providing a graphical sample representation which is more easily interpretable than the one obtained from all variables. in the second example, we filter the variable set by the association with given responses, and show that the optimal projection score and the shape of the projection score curve obtained from filtering with respect to a certain response capture the overall evidence in the data for a significant association with that response. we also show the results from a combined variance and response-related filtering. in the third example, we apply a shaving procedure to generate variable subsets, and evaluate the obtained subsets by their projection scores. then, we validate the use of the projection score with synthetic examples, where we show that the variable subset with the highest projection score is often the one containing the non-random structure in the data. finally, we discuss some issues regarding the estimation of the projection score and warn against a potential pitfall. in all examples, we use the projection score to compare the informativeness of variable subsets with respect to visualization. assume that we are given a data set with n samples and p variables, represented by a rank-r matrix x ∈ ℝp×n, and a collection of m variable subsets rm ⊆ {..., p} for m =..., m. we define selection functions ϕm in such a way that consists only of the rows in x with indices in rm. to compute the projection score, we must also select an s ⊆ {..., r}, essentially representing the number of degrees of freedom we expect in the data. letting Λx = denote the vector of non-zero singular values of x in decreasing order, the fraction of the variance in x which is accounted for by the principal components with indices in s can be computed as the projection score of rm is then defined as here, denotes the inferred distribution of ϕm under the assumption of independence among the original samples and variables in x. we compute the projection score for each of the m variable subsets, and the subset with the largest projection score is considered the most informative variable subset for visualization. for further details, see the methods section. to obtain a visualization which reflects the correlation structure between the variables of the data set instead of the individual variances, we consistently extract the singular values from standardized data matrices, where each variable is mean-centered and scaled to unit variance. this standardization is indeed commonly used. if we had not standardized the data, the correlations between the variables would be less influential when computing the principal components, and the variances of the individual variables could potentially have a very large impact. since we define the projection score by comparing the observed data to data generated under a null model defined by assuming independence between the variables, we may argue that the non-random structure that is detected with the resulting score is that corresponding to correlations between variables and that therefore, the standardized data are better suited for our purposes. note that for the variance- and response-related filterings, the variable rankings are extracted from the original, unstandardized data. variance filtering of a lung cancer data set we first use the projection score to find particularly informative variable subsets among those obtained by applying variance filters to the lung cancer data set studied in. the data set contains gene expression measurements for genes in subjects. the subjects belong to four groups: normal, pulmonary carcinoid tumors, small cell carcinoma or colon metastases. for a given set of variance thresholds, we define rm as the collection of variables with variance exceeding θm. the variance thresholds are specified as fractions of the highest individual variable variance in the data set, meaning that the variables included in the variable subset rm are those with variances exceeding θm · varmax where varmax denotes the maximal variance among the variables in the data set. we choose s = { 3}, that is, we search for a variable subset providing an informative three-dimensional sample representation, which is reasonable in a visualization context. figure shows the three-dimensional sample representation obtained by applying pca only to the variables in the most informative subset. the representation based on the most informative subset is considerably more easily interpretable than the representation based on the entire set of variables, which is shown in figure in the representation based on the most informative variable subset, the pulmonary carcinoid tumors appear to cluster into several subgroups, an effect which was also noted in. figure shows the projection score curve for the filtering parameter θ. very small variable subsets, corresponding to high values of the variance threshold, do not support the chosen s. the projection score attains its maximal value of τ = for a variance threshold of % of the maximal variance, corresponding to a variable subset consisting of the variables with highest variances. in figure we show a heatmap for the expression matrix corresponding to the most informative variable subset. clearly, the variables with the highest variances contain much information about the four sample groups, which is not surprising. response-related filtering of the nci- data set in this section, we construct ϕm to consist of the variables which are most highly related to a given response variable. in the studied examples the response variable indicates the partition of the samples into different groups. given such a partition, we calculate the f-statistic, contrasting all these groups, for each variable. for a given set of significance thresholds, all variables which are significantly related to the response at the level αm are included in ϕm. for each randomized data set x* used to estimate we define the significance thresholds in such a way that the resulting variable subsets have the same cardinalities as the corresponding subsets from the original data set. the choice of s is guided by the underlying test statistic. when we contrast only two groups, we do not expect the optimal variable subset to support more than a one-dimensional sample configuration, and therefore we choose s = {1} in this case. when contrasting more than two groups, the choice of s must be guided by other criteria. this is because the variables with the highest f-score may in this case very well characterize many different sample groups, not all of which can simultaneously be accurately visualized in low dimension. the nci- data set contains expression measurements of genes in cell lines from nine different types of cancers. we first filter the variable set with respect to the association with the partition defined by all the nine cancer types, using s = { 3}. this gives a most informative subset consisting of variables, with a projection score τ =. the resulting sample representation, obtained by applying pca to the most informative variable subset, is shown in figure the representation based on all variables is shown in figure and the projection score is shown in figure as a function of log figure shows the p-value distribution, which indicates that there are indeed variables which are truly significantly associated to the response. figure shows a heatmap for the most informative variable subset ) where it can be seen that the samples are reasonably well clustered according to cancer type based on these variables only. next, we set out to study how informative the contrasts between one cancer type and all the others are. we filter the variable set using the association with the contrast of one cancer type vs the rest, using s = {1}. figure shows some of the projection score curves. first, we can note that the range of p-values, as well as the range of obtained projection scores, are highly different for the different contrasts. the highest projection scores in the respective cases are,, and. apparently, for each of the melanoma, leukemia and renal contrasts, a small subset of the variables related to the respective response contains a lot of non-random information. however, for the nsclc contrast the full variable set is the most informative. this can be understood from figure which shows a histogram over the p-values obtained from the f-test contrasting the nsclc group with the rest of the samples. the p-values are essentially uniformly distributed, indicating that there are no truly differentially expressed genes in this case. hence, in the filtering process we do not unravel any non-random structure, but only remove the variables which are informative in other respects. figure shows the p-value distribution for the melanoma contrast. in this case, there are indeed some differentially expressed genes, which means that in the filtering process, we purify this signal and are left with an informative set of variables. the projection scores obtained from the different contrasts are consistent with figure in the sense that the highest projection scores are obtained from the contrasts corresponding to the cancer types which form the most apparent clusters in this sample representation, that is, the melanoma samples and the leukemia samples. the nsclc samples do not form a tight cluster and are not particularly deviating from the rest of the samples in figure combined variance and response-related filtering of the nci- data set in some cases, the variable set is first filtered by variance before a statistical test is applied. removing supposedly non-informative variables in this way may be beneficial in terms of statistical power, since if the number of performed tests is high, we need to do a rather heavy correction for multiple comparisons. also with the approach proposed in this paper, we can combine different filtering procedures to find a variable subset which is informative from more than one perspective. here, we show how to combine variance filtering with response-related filtering for the nci- data set. in this way, we exclude all variables that obtain small p-values from the f-test mostly due to their low variances. as before, we define a collection of variance thresholds θm and a number of significance thresholds αm, for the f-test contrasting all nine cancer types. we choose s = { 3} as before. now, the projection score can be represented as a surface, shown in figure for varying values of the variance and p-value thresholds. the curves traced out for log = and θ = correspond to the projection score curves for the individual variance and response-related filterings, respectively. as can be seen in figure we get the maximal projection score for a variable subset obtained by a combination of variance filtering and response-related filtering. this subset includes all variables with a variance exceeding % of the maximal variance, and with a p-value below · 10- in total variables. the maximal projection score is τ =. this can be compared to the purely response-related filtering, which gave a maximal projection score of τ = by including the variables with p-values below · 10- figure shows the sample configuration obtained by applying pca to the most informative variable subset from the combined filtering. shaving of a leukemia data set here, we will use the projection score to evaluate the informativeness of variable subsets obtained by the shaving procedure described in. briefly, for a given fraction π ∈, we define a function ξ: 2{...,p} → 2{...,p} by letting ξ consist of the fraction - π of the variables in r having the highest loadings in the first principal component of the data set consisting of the variables in r. the selection functions ϕm are then defined by letting ϕm contain all variables in. we use π =, hence in each step shaving off 2% of the variables, continuing until only one variable remains. we compute the projection score for each of the variable subsets, as well as the gap statistic proposed in. we apply the described method to the leukemia data set studied in. the data set contains gene expression measurements from genes in samples with either aml or all. computing the projection score and the gap statistic for the variable subsets obtained by shaving gives optimal variable subsets containing variables and variables, respectively. figure shows the two informativeness measures as functions of the variable subset cardinality. both curves are smooth and have clear extreme points. figures and show heatmaps for the optimal variable subsets from the two methods. as can be seen in these figures, the two subsets contain much the same information about the samples in the data set. the variable subset selected by the projection score is larger than the one selected by the gap statistic. in some situations, a small number of included variables may be beneficial. however, as will be shown, the gap statistic tends to underestimate the number of variables in the optimal subset, which may potentially be the case also in this example. this would then lead to a number of "false negatives", that is, variables falsely excluded from the optimal combination. the median and range of the number of non-zero elements included in the optimal variable subset found by different methods. each underlying component contains non-zero elements. spc - sparse pca. ssvd - sparse svd. the gap statistic is defined as in. the median and range of the number of non-zero elements included in the optimal variable subset found by different methods. each underlying component contains non-zero elements. spc - sparse pca. ssvd - sparse svd. the gap statistic is defined as in. validation on synthetic data in this section, we will validate the projection score by using it to find informative variable subsets from different filtering processes applied to synthetic data sets. variance filtering we let and generate a synthetic data set with variables and samples by letting the only non-random structure in the data is contained in the first variables, discriminating between two groups of samples each. by varying σ we obtain data sets with difference variance properties. with σ =, the informative variables and the non-informative variables have comparable variances. with σ =, the informative variables obtain a lower variance than the non-informative variables and with σ = the informative variables are also those with the highest variances. we take s = {1}, since no other choice of s is supported by any variable subset. this is also consistent with the structure of the data matrix. across realizations, the mean number of variables in the subset giving the best projection score are for σ =, for σ = and for σ =. the projection score correctly indicates that when σ =, the informative structure in the data is indeed related to the variables with lowest variances, and hence all variables are included in the most informative subset. note that the association between the variables within each sample group is very strong when σ =. if the variables with lowest variance had been routinely filtered out in this example, we would lose the informativeness in the data. it can also be noted that when the number of variables is below a certain threshold in the σ = case, not even s = {1} is supported by the data since we have filtered out all the informative variables. for σ =, the optimal number of variables is highly dependent on the specific data set since the filtering removes both informative and non-informative variables uniformly. response-related filtering in this example, we simulate a data matrix containing two group structures. the data set consists of samples which are divided into four consecutive groups of samples each, denoted a, b, c, d. we define the data matrix is then generated by letting we perform a two-sided t-test contrasting a ∪ c and b ∪ d and order the variables by the absolute value of their t-statistic for this contrast. in this case, since we compare only two groups we are essentially searching for a one-dimensional separation, so we choose s = {1}. figure shows the structure of the data set. the data set contains one very strong factor, encoded by the first variables, and one weaker factor, the one we are interested in, which is related to the next variables. figures and show the projection score for different thresholds on the significance level α and for different variable subset cardinalities, respectively. the optimal projection score is obtained for variable subsets containing slightly less than variables. hence, even though there is indeed much information contained in the entire variable set, it is possible to obtain an even more informative variable subset by filtering. projecting onto the first principal component based only on the variables in the most informative subset clearly discriminates between a ∪ c and b ∪ d, as shown in figure as above, we can compare the maximal projection score corresponding to filtering by the association with different responses. filtering with respect to the association with the contrast between a ∪ b and c ∪ d, that is, the stronger factor in the data set, gives a maximal projection score around. hence, this correctly suggests that this factor is more informative than the previously studied. filtering with respect to the variance, still using s = {1}, gives a maximal projection score of, obtained for approximately variables. this implies that the variables with high variance in this case are even more informative than the variables closely associated with one of the responses, in the sense that the encoded information deviates more from what would be expected from the highly varying variables in a randomized data set. applying variance filtering with s = { 2} provides a most informative variable subset containing variables, capturing parts of both the two informative variable groups in the data. s = { 3} is not supported by any variable subset. detecting sparsity in principal components in this example, we will evaluate the usefulness of the projection score for detection of sparsity in the leading principal components of a data matrix. we generate a data set, with p = variables and n = samples, following the scheme given in. briefly, we form a matrix v = from an orthonormal set of p principal components v..., vp ∈ ℝp, and a matrix c = diag containing the eigenvalues in decreasing order on the diagonal. then, we form the covariance matrix Σ by to generate data, we let z be a random draw from and take x = vc1/2z. the first two principal components are constructed to have specific sparsity patterns. in the first example, we let v and v contain non-zero elements each and choose c = c = we let ck = for k =..., and sample the entries of the corresponding principal components uniformly between and before orthogonalizing and normalizing them. in the second example, we let v and v instead contain non-zero elements. we choose c = and c = and proceed as in the previous example to obtain the rest of the components. the variable subsets are obtained by the shaving procedure, as outlined above and described in. this gives a nested sequence of variable subsets, which can be evaluated in terms of their projection scores. we also compute the gap statistic as defined in as another quality measure of each variable subset. the optimal variable subsets obtained from these methods are compared to those found by a sparse pca algorithm and the sparse svd proposed in. it should be noted that the aims of these methods are somewhat different. the sparse pca attempts to find a good approximation of the original data set using only a subset of the variables, while the sparse svd was developed for biclustering, that is, finding small groups of variables which are related across possibly only a subset of the samples. hence, also the sample representation can be sparse in the results from the sparse svd. in the examples studied here, however, the sparsity patterns of the principal components are designed to be identical across all samples. the sparse svd and the sparse pca were applied through the r programs provided by the respective authors. the sparse svd was applied with γu = γv = as suggested by the authors. the sparsity parameter in the sparse pca was estimated via the cross-validation function provided with the r package, evaluating different sparsity levels between and via 5-fold cross-validation. estimating to obtain, we repeatedly permute the values in each row of x and perform the variable filtering, which is computationally expensive. a more efficient implementation can be obtained if we specify the distribution for each m =..., m in advance. then, the values of can be estimated in advance and stored, which leaves only the calculations for the observed data matrix and the subtraction of a known quantity for each variable subset. for instance, we can fix by assuming that each element in ϕm is drawn from a standard normal distribution, that is, for i =..., |rm| and j =..., n. we can then calculate the expected value of for a large collection of choices of variable subset cardinalities and sample sizes. figure shows an interpolated surface for ≤ |rm| ≤ and ≤ n ≤ this more computationally efficient approach may be used for example for variance filtering, if the observed data matrix is standardized before the principal components are extracted. when the variable subsets are defined by response-related filtering, it is more difficult to specify and sample from the truncated distribution resulting from the filtering. in particular, we note that even for a data matrix x containing only independent variables, we still expect to see an aggregation of the variance in the first principal components when we consider small submatrices ϕm. in some cases, however, it may still be of interest to compare the singular values from the observed data to those from matrices with a given, known distribution such as the one described above. we next give a small example to show the different conclusions that can result if different definitions of are used. example we define x ∈ ℝ × by letting for i =..., 000; j =..., we divide the samples into four consecutive groups a, b, c, d of samples each, as in the response-related filtering above, and filter the variable set based on the absolute value of the t-statistic contrasting groups a∪c and b∪d, using s = {1}. if is defined by assuming that each element of ϕm is drawn from a standard normal distribution, the most informative variable subset contains variables. here, even though we study a completely random matrix and an artificial grouping of the samples, we obtain a good separation between the groups. however, the projection score is not very high in this case. figure shows the projection score as a function of the significance threshold, and the optimal projection. as can be seen, just looking at the visualization in the right panel we might be led to believe that there is actually some non-random structure in the data. on the other hand, if we define as we have done in the previous examples, by assuming independence among the samples and variables of the original matrix x and then filtering, no submatrix supports even s = {1}, and hence we get an indication that we are filtering a matrix without non-random relationships between the variables.  in this paper, we have introduced and shown the usefulness of the projection score, a measure of the informativeness of a subset of a given variable set, based on the variance contained in the corresponding sample configuration obtained by pca. the definition of the projection score is straightforward and the interpretation in terms of captured variance is intuitively appealing in a pca visualization context. moreover, the projection score allows a unified treatment of variable selection by filtering in the context of visualization, and we have shown that it indeed gives relevant results for three different filtering procedures, both for microarray data and for synthetic data. by filtering with respect to a specific factor, we obtain sparse principal components where all variables receiving a non-zero weight are indeed strongly related to the chosen factor. in this respect, the resulting components may be more easily interpretable than general sparse principal components, where the variables obtaining a non-zero weight can be related to many different factors. the optimal number of variables included in the extracted principal components often increases with |s| and in many cases small variable subsets do not even support large subsets s. 
the exploratory perspective often makes statistically guided visualization methods, such as principal component analysis, the methods of choice. we apply the presented measure to find optimal variable subsets for different filtering methods in both microarray data sets and synthetic data sets. we note also that the projection score can be applied in general contexts, to compare the informativeness of any variable subsets with respect to visualization by pca. we present the projection score, which is a straightforward, intuitively appealing measure of the informativeness of a variable subset with respect to pca visualization. for microarray data, more easily interpretable visualizations are often obtained by filtering the variable set, for example by removing the variables with the smallest variances or by only including the variables most highly related to a specific response. in many scientific domains, it is becoming increasingly common to collect high-dimensional data sets, often with an exploratory aim, to generate new and relevant hypotheses. we conclude that the projection score provides an easily interpretable and universally applicable measure of the informativeness of a variable subset with respect to visualization by pca, that can be used to systematically find the most interpretable pca visualization in practical exploratory analysis. however, the clarity of the obtained visualizations, and thereby the potential to use them to formulate relevant hypotheses, may be confounded by the presence of the many non-informative variables. the resulting visualization may depend heavily on the inclusion criterion, that is, effectively the number of retained variables. to our knowledge, there exists no objective method for determining the optimal inclusion criterion in the context of visualization.
 in many scientific domains, it is becoming increasingly common to collect high-dimensional data sets, often with an exploratory aim, to generate new and relevant hypotheses. the exploratory perspective often makes statistically guided visualization methods, such as principal component analysis, the methods of choice. however, the clarity of the obtained visualizations, and thereby the potential to use them to formulate relevant hypotheses, may be confounded by the presence of the many non-informative variables. for microarray data, more easily interpretable visualizations are often obtained by filtering the variable set, for example by removing the variables with the smallest variances or by only including the variables most highly related to a specific response. the resulting visualization may depend heavily on the inclusion criterion, that is, effectively the number of retained variables. to our knowledge, there exists no objective method for determining the optimal inclusion criterion in the context of visualization.  we present the projection score, which is a straightforward, intuitively appealing measure of the informativeness of a variable subset with respect to pca visualization. this measure can be universally applied to find suitable inclusion criteria for any type of variable filtering. we apply the presented measure to find optimal variable subsets for different filtering methods in both microarray data sets and synthetic data sets. we note also that the projection score can be applied in general contexts, to compare the informativeness of any variable subsets with respect to visualization by pca.  we conclude that the projection score provides an easily interpretable and universally applicable measure of the informativeness of a variable subset with respect to visualization by pca, that can be used to systematically find the most interpretable pca visualization in practical exploratory analysis. 