BACKGROUND
the analysis of gene expression microarray data using clustering techniques plays an important role, for instance, in the discovery, validation, and understanding of various classes and subclasses of cancer  <cit> .
there are two ways of clustering a gene expression matrix  <cit> :  gene function may be inferred from clusters of genes similarly expressed across the samples and  samples can form groups which show similar expression across the genes.
moreover, genes and samples can be clustered simultaneously, with their inter-relationship represented by bi-clusters  <cit> .
the clustering of the genes on the basis of the samples is a standard cluster analysis problem that can be effected by a variety of algorithms  <cit> .
for a comprehensive review see  <cit> .
a more challenging problem is the clustering of the samples on the basis of the genes, where the standard clustering techniques, such as k-means or hierarchical clustering, fail to capture complex local structures, due to the high-dimenionality of the data  <cit> .
in recent years, computational improvement enabled new clustering techniques and contributed to the development of previously unfeasible methods.
in this context, mclachlan et al.
<cit>  propose a mixture model-based approach to cluster microarray expression data.
their scheme accounts for gene selection through mixtures of t distributions, and dimensionality reduction through a mixture of factor analyzers.
more precisely, they select a gene on the basis of a likelihood ratio statistic for testing one versus two components in the mixture model.
in the second step of their algorithm, they cluster the samples by fitting a two-component mixture of factor analyzers.
although their method sounds like a good approach to clustering samples in a high-dimensional space, there are three main limitations.
firstly, the parametric assumptions about clusters distributions can be restrictive  <cit> ; for example, two gaussian random variables can result in a single mode  or even a two component multivariate gaussian mixture can lead to more than two modes  <cit> .
moreover, it requires pre-specification of the number of the mixture components; this represents a serious limitation from an unsupervised perspective, which assumes that the true number of clusters is unknown.
finally, the number of parameters per component grows as the square of the dimension of the data  <cit> , this is a major shortcoming in high-dimensional data.
in this paper, we present a novel strategy, which consists in applying a clustering technique after gene filtering and dimensionality reduction, in order to exploit the most significant dimensions in the definition of the clusters.
our procedure can be thought of as a three-step algorithm:  gene filtering;  dimensionality reduction;  clustering in the reduced space.
several authors outlined the importance of a gene filtering step prior to inferential procedures  <cit>  or cluster analysis  <cit> .
tritchler et al.
<cit>  empirically showed that principal components and cluster analysis are strongly affected by gene selection, and that filtering out uninformative genes can reduce bias in the clustering of samples.
furthermore, johnstone and lu  <cit>  showed, from a theoretical point of view, that some initial reduction in dimensionality is desirable before applying a principal component analysis, when p is larger than n.
traditional approaches to gene filtering are based on thresholding the mean or the variance of genes across samples.
bourgon et al.
<cit>  found that gene-by-gene filtering by overall variance increased the power of the subsequent t-test.
tritchler et al.
<cit>  considered the covariance structure of the genes, defining filters that preserve the topology of the network.
nevertheless, from a clustering point of view, these approaches could be unsafe: a gene should be considered relevant if it is important in the definition of the clusters; therefore, it seems more appropriate to retain those genes whose univariate distribution highlights a clear grouping among the observations rather than the ones with higher variance.
to evaluate our general strategy, we implement an algorithm based on a nonparametric model-based clustering technique by azzalini and torelli  <cit> , which we will refer to as pdfcluster .
we compare it with a traditional partition algorithm , and with a similar strategy in which we use, instead of pdfcluster, its direct competitor, mclust  <cit> , a state-of-the-art mixture-model-based clustering tool.
by using the nonparametric approach based on pdfcluster, we achieve improvements in clustering of samples both in simulated and in real experiments.
to be consistent with microarray applications, we use here the typical microarray terminology: we denote by "genes" the p variables and by "samples" the n observations.
nonetheless, it should be clear that the proposed approach is not limited to microarray data, but, in principle, it could be applied to every set of continuous variables with "large p, small n".
RESULTS
a novel algorithm to clustering of expression data
as we said, clustering samples using expression data is a challenging statistical problem due to dimensionality issues.
therefore, in this context, it is unfeasible to directly apply a clustering technique to the whole data matrix.
here, we evaluate our strategy implementing an algorithm which exploits the self-detection of number of clusters feature of pdfcluster.
the algorithm can be summarized as follows:  cluster samples using the univariate distribution of each gene and select for the subsequent analyses the p' genes, in which pdfcluster identifies two or more clusters;  reduce dimensionality by selecting the first p" principal components;  apply pdfcluster in the p"-dimensional space.
it is straightforward to see that this algorithm falls within the general framework defined in the background section.
as for step , i.e., gene filtering, we consider a gene relevant if its values in one category  are different from the ones in the other category  or categories.
from another point of view, this means that the samples representing the healthy subjects are separated from the unhealthy ones, or, more simply, the samples are in different clusters.
in this way, it seems reasonable to apply a cluster method to each gene, and retain as relevant those genes for which the method identifies distinct clusters.
in a nonparametric framework, we can apply pdfcluster to each gene, taking advantage of the self-detection number of clusters feature.
we select only the genes for which the method detects two or more clusters .
we consider a clustering technique to define informative genes over the traditional variance-based approaches, because small overall variance does not necessarily imply a single cluster, and high overall variance is not always an indication of two of more clusters.
moreover, variance-based filters depend on the choice of an arbitrary threshold, which can be difficult to choose, since one typically does not know which portion of the genes is responsible for the clustering of the samples.
as for step , i.e., dimensionality reduction, considered if the selected genes are still too many, we propose to keep the first principal components, as in  <cit> .
the principal component analysis is a very simple procedure which reduces the dimension of a data set of a large number of interrelated variables, preserving as much as possible of the data set variation.
since it has no requirements about the data distribution, it is consistent with our nonparametric strategy.
in order to compare our approach to mclust, we carried out a procedure analogous to the one described here, but using the normal-mixture model both in step  and .
note that this procedure differs from the one in mclachlan et al.
<cit> , because they use a mixture of factor analyzers to select genes and reduce dimensionality.
computational issues
the further dimensionality reduction in our step  is necessary since pdfcluster, in order to compute the delaunay triangulation, exploits the quickhull algorithm  <cit> .
barber et al.
<cit>  state that the quickhull algorithm for finding the convex hull of a set of n points in ℝp requires at most o operations if p ≤  <dig>  and o where m = ⌊p/2⌋ for p >  <dig>  azzalini and torelli  <cit>  observed that the computing time increases less than quadratically in n for any fixed p, but it increases more than exponentially in p for fixed n. our experience is that for p =  <dig>  the algorithm is very fast  core <dig> quad cpu q <dig> @  <dig>  ghz).
therefore, there are no computationally related problems for step .
moreover, with p <  <dig> it takes a reasonable time  to complete the procedure.
number of principal components
in order to choose the number of principal components, we carried out a small simulation study : we found that, with n =  <dig>  pdfcluster performs at best, in terms of misclassification error, with 3- <dig> dimensions, while with p =  <dig> the misclassification error starts to grow.
this is probably due to the extreme dispersion of the observations in higher dimensional spaces .
the performances of pdfcluster are slightly better with  <dig> components, but the improvement does not justify the increased computational time .
thus, for the subsequent analyses, we will retain  <dig> principal components.
simulated data
in this section, we evaluate our proposal by means of simulated data.
for simulating data with structure similar to that of real microarray experiments, we use two schemes, i.e., the gamma-gamma  model  <cit>  and the normal-uniform  model  <cit> .
in gg model we simulate data with two clusters , such that the majority of genes are equally expressed between the groups and a small fraction of them  is differentially expressed.
this mimics a classical experiment in which some diseased subjects are compared to healthy controls.
in nu model we simulate data with three clusters: cluster  <dig> consists of  <dig> samples with  <dig> up-regulated and  <dig> down-regulated genes; cluster  <dig> consists of  <dig> samples with  <dig> down-regulated genes; cluster  <dig> consists of  <dig> samples with neither up- nor down-regulated genes.
note that cluster  <dig> and  <dig> are "closer" to each other than to cluster  <dig> and that cluster  <dig> has smaller sample size.
this mimics a more elaborate design, e.g.
two different types of a specific disease versus a normal control.
gg model
simulation results for pdfcluster and mclust in gg model: rate of correct identification of number of clusters , sensitivity , specificity  and error rate  in the classification of the samples, and error rate in the selection of relevant genes .
more interesting is the very different behaviour in the choice of the relevant genes: pdfcluster is very good in recognizing them, with a very low error rate , while mclust shows a very high error rate .
we simulated a relatively small number of "marker" genes; pdfcluster correctly discards the majority of genes as non-relevant in the determination of the clusters, while mclust seems to be too sensitive to outliers, erroneously capturing differences due to random noise.
nu model
as expected, table  <dig> shows that in this model both pdfcluster and mclust lead to higher classification errors than in gg model.
also in the gene filtering step, both methods have difficulties in finding the relevant genes.
simulation results for pdfcluster and mclust in nu model: rate of two clusters identification , rate of three clusters identification , error rate in the classification of samples  and error rate in the selection of relevant genes .
mclust is able to recognize three clusters in 39% and two clusters in 34% of the simulations; pdfcluster recognizes three clusters in 19% and two clusters in 47% of the simulations.
on the other hand, the mean error rate of the final classification is  <dig>  for pdfcluster while for mclust is  <dig> .
this is probably due to the fact that the cases in which pdfcluster correctly recognizes three clusters are those with the most separated clusters among the ones recognized by mclust: obviously, in the less separated clusters cases, it is more difficult to allocate the samples.
finally, it is worth noting that pdfcluster outperforms mclust according to the gene selection error rate : as in previous simulation study, pdfcluster works better in recognizing which genes are effectively responsible for the determination of the clusters.
sample size
one issue with microarray data is often the low sample size.
in order to evaluate its effect on the performance of our approach, we simulated data from the nu model, varying the sample size n. for different values of n, namely n =  <dig>   <dig>   <dig>   <dig>   <dig>  we simulated b =  <dig>  samples in a setting similar to the previous section, i.e., 40% of the observations forming cluster  <dig>  40% forming cluster  <dig> and 20% cluster  <dig>
misclassification error rate  for pdfcluster and mclust in nu model, varying the sample size.
real data
along with simulations, we consider two benchmarking real datasets, studied before by several authors  <cit> , which we will refer to as the colon data and the leukaemia data .
colon data
as described above, we analyze the dataset, following three steps:  gene filtering,  dimensionality reduction,  clustering of samples.
namely, the first step of the procedure consists in applying the cluster algorithm to the univariate distribution of each gene.
the genes that show two or more clusters are considered for the further steps.
in the first step, the pdfcluster algorithm is able to recognize  <dig> genes, which discriminate data into two or more groups.
we proceed by considering the first three principal components of this reduced data-matrix.
the procedure finds three clusters, summarized in table  <dig> which clearly correspond to biologically meaningful groups.
the first cluster consists of tumor tissues , while clusters  <dig> and  <dig> comprise normal tissues .
it is worth noting that six out of the eight misallocated samples  are found to be misclassified in several previous analyses, including  <cit> .
as stated, for instance, in  <cit> , these six samples are likely to be wrongly labeled.
furthermore, getz et al.
<cit>  reported that there was a change in the protocol during the experiments: tumor samples 1- <dig> and normal samples 41- <dig> were collected within the first protocol, while tumor samples 12- <dig> and normal samples 52- <dig> were collected within the second.
although for the tumor samples our approach did not recognize any difference between the protocols, cluster  <dig> and cluster  <dig> split normal tissues in two groups according to the protocols.
clusters found after pdfcluster procedure in colon data; tumor samples are labeled 1- <dig>  normal samples 41-62; misallocated samples are shown in bold.
the star represents a wrongly labeled samples.
in the first step, mclust is able to find  <dig> discriminant genes.
we consider the first three principal components of this sub-space for clustering.
the procedure finds two clusters, with a rather high misclassification error .
we also apply the k-means algorithm to the entire dataset.
the results of the three approaches are shown in table  <dig>  it can be seen that k-means, exploited in the original p-dimensional space, does not perform well.
moreover, pdfcluster outperforms  mclust, if one considers cluster  <dig> and  <dig> together as the normal samples.
confusion matrices for pdfcluster, mclust and k-means with error rates  for colon data.
as stated before, mclachlan et al.
<cit>  studied the same microarray dataset.
they selected  <dig> relevant genes, achieving clusters that seem to recognize the change of protocol in the data structure, but fail to recognize the normal/tumor differences  <cit> .
nevertheless, they achieved results slightly better than ours  considering a particular subspace: they clustered genes in  <dig> groups and considered only the second group  to cluster data  <cit> .
although this approach leads to good results in this example, it seems difficult to reproduce the procedure in an unsupervised setting.
leukaemia data
as stated in  <cit> , the leukaemia dataset presents two different problems: an easier one, consisting of separating all from aml  and a harder one, consisting also of recognizing the differences in b-cell and t-cell subclasses .
again, we consider the strategy previously described.
in the filtering step, pdfcluster recognizes  <dig> discriminant genes.
note that the higher number of genes selected with respect to colon data is consistent with the higher difficulty of the problem.
we proceed by considering the first three principal components of this subspace.
the pdfcluster algorithm finds two clusters, which clearly represent all and aml samples, with  <dig> aml samples classified as all and  <dig> all samples classified as aml, leading to a misclassification error rate of  <dig>  : pdfcluster is able to solve the two-class problem, but it misses the three-class problem.
confusion matrices for pdfcluster, mclust and k-means with error rates  for leukaemia data.
in the first step, mclust fails to select relevant genes, recognizing  <dig>  out of  <dig>  genes as discriminant among the groups.
based on the first three principal components of the subspace spanned by these genes, mclust clusters samples in four groups.
we could interpret the merged clusters 1- <dig> as the all b-cell class, and cluster  <dig> as the aml class, while cluster  <dig> interpretation is less clear .
although mclust is able to find more than two clusters, it fails to distinguish between b-cell and t-cell classes, leading to hardly interpretable clusters.
the leukaemia dataset has been studied by mclachlan et al.
<cit>  as well.
the authors found  <dig>  relevant genes after the variable selection step.
for the two-class problem, their results were very good , but they failed to solve the three-class problem.
it should be noted that, unlike our algorithm, the procedure used in  <cit>  needs prior specification of the number of clusters, which is not desirable in an unsupervised learning, especially in cancer tissue classification, where one of the main goals is to find new subclasses of tumors.
CONCLUSIONS
model-based approaches to clustering of data have received increasing attention in the last few years, as they provide a sound mathematical-based method.
unfortunately, in microarray applications, the high dimensionality of the space makes the clustering of samples in the whole space unfeasible within a model-based framework.
here, we have discussed a general strategy for the clustering of microarray expression data, based on gene filtering and dimensionality reduction as preliminary steps in the cluster analysis.
we have discussed a nonparametric density estimation-based algorithm within this framework, showing promising results both in simulated data and in two real applications, with surprisingly good computational performances.
in our simulation experiments, we have found that pdfcluster leads to slightly better performances than mclust.
moreover, the gene filtering step is much more effective using pdfcluster than using mclust both in simulated and in real datasets.
here, "effective" means good results in terms of both dimension reduction  and of correct selection .
here we have used pdfcluster in order to select relevant genes.
we underlined the assets of this chioce, but it is clear that any unsupervised technique able to discard the irrelevant genes can be used.
similarly, the choice of principal component analysis in the dimensionality reduction step is only one among several possible choices.
since there are no guarantees that the first principal components preserve the cluster structure in the reduction of original dimension of data  <cit> , future efforts could be made in trying different approaches, such as the projection pursuit  <cit>  or the principal curves  <cit> .
nevertheless, in our case the principal component analysis gives good results and provides a low dimensional dataset on which it is feasible to apply a model-based technique such as pdfcluster.
all the statistical analyses and simulations have been performed with r  <cit>  and with a public domain implementation of the "quickhull" algorithm  <cit>  available at http://www.qhull.org/.
the datasets used are both freely available as bioconductor  <cit>  packages .

 the analysis of gene expression microarray data using clustering techniques plays an important role, for instance, in the discovery, validation, and understanding of various classes and subclasses of cancer. there are two ways of clustering a gene expression matrix: gene function may be inferred from clusters of genes similarly expressed across the samples and samples can form groups which show similar expression across the genes. moreover, genes and samples can be clustered simultaneously, with their inter-relationship represented by bi-clusters. the clustering of the genes on the basis of the samples is a standard cluster analysis problem that can be effected by a variety of algorithms. for a comprehensive review see. a more challenging problem is the clustering of the samples on the basis of the genes, where the standard clustering techniques, such as k-means or hierarchical clustering, fail to capture complex local structures, due to the high-dimenionality of the data. in recent years, computational improvement enabled new clustering techniques and contributed to the development of previously unfeasible methods. in this context, mclachlan et al. propose a mixture model-based approach to cluster microarray expression data. their scheme accounts for gene selection through mixtures of t distributions, and dimensionality reduction through a mixture of factor analyzers. more precisely, they select a gene on the basis of a likelihood ratio statistic for testing one versus two components in the mixture model. in the second step of their algorithm, they cluster the samples by fitting a two-component mixture of factor analyzers. although their method sounds like a good approach to clustering samples in a high-dimensional space, there are three main limitations. firstly, the parametric assumptions about clusters distributions can be restrictive; for example, two gaussian random variables can result in a single mode or even a two component multivariate gaussian mixture can lead to more than two modes. moreover, it requires pre-specification of the number of the mixture components; this represents a serious limitation from an unsupervised perspective, which assumes that the true number of clusters is unknown. finally, the number of parameters per component grows as the square of the dimension of the data, this is a major shortcoming in high-dimensional data. in this paper, we present a novel strategy, which consists in applying a clustering technique after gene filtering and dimensionality reduction, in order to exploit the most significant dimensions in the definition of the clusters. our procedure can be thought of as a three-step algorithm: gene filtering; dimensionality reduction; clustering in the reduced space. several authors outlined the importance of a gene filtering step prior to inferential procedures or cluster analysis. tritchler et al. empirically showed that principal components and cluster analysis are strongly affected by gene selection, and that filtering out uninformative genes can reduce bias in the clustering of samples. furthermore, johnstone and lu showed, from a theoretical point of view, that some initial reduction in dimensionality is desirable before applying a principal component analysis, when p is larger than n. traditional approaches to gene filtering are based on thresholding the mean or the variance of genes across samples. bourgon et al. found that gene-by-gene filtering by overall variance increased the power of the subsequent t-test. tritchler et al. considered the covariance structure of the genes, defining filters that preserve the topology of the network. nevertheless, from a clustering point of view, these approaches could be unsafe: a gene should be considered relevant if it is important in the definition of the clusters; therefore, it seems more appropriate to retain those genes whose univariate distribution highlights a clear grouping among the observations rather than the ones with higher variance. to evaluate our general strategy, we implement an algorithm based on a nonparametric model-based clustering technique by azzalini and torelli, which we will refer to as pdfcluster. we compare it with a traditional partition algorithm, and with a similar strategy in which we use, instead of pdfcluster, its direct competitor, mclust, a state-of-the-art mixture-model-based clustering tool. by using the nonparametric approach based on pdfcluster, we achieve improvements in clustering of samples both in simulated and in real experiments. to be consistent with microarray applications, we use here the typical microarray terminology: we denote by "genes" the p variables and by "samples" the n observations. nonetheless, it should be clear that the proposed approach is not limited to microarray data, but, in principle, it could be applied to every set of continuous variables with "large p, small n".  a novel algorithm to clustering of expression data as we said, clustering samples using expression data is a challenging statistical problem due to dimensionality issues. therefore, in this context, it is unfeasible to directly apply a clustering technique to the whole data matrix. here, we evaluate our strategy implementing an algorithm which exploits the self-detection of number of clusters feature of pdfcluster. the algorithm can be summarized as follows: cluster samples using the univariate distribution of each gene and select for the subsequent analyses the p' genes, in which pdfcluster identifies two or more clusters; reduce dimensionality by selecting the first p" principal components; apply pdfcluster in the p"-dimensional space. it is straightforward to see that this algorithm falls within the general framework defined in the background section. as for step, i.e., gene filtering, we consider a gene relevant if its values in one category are different from the ones in the other category or categories. from another point of view, this means that the samples representing the healthy subjects are separated from the unhealthy ones, or, more simply, the samples are in different clusters. in this way, it seems reasonable to apply a cluster method to each gene, and retain as relevant those genes for which the method identifies distinct clusters. in a nonparametric framework, we can apply pdfcluster to each gene, taking advantage of the self-detection number of clusters feature. we select only the genes for which the method detects two or more clusters. we consider a clustering technique to define informative genes over the traditional variance-based approaches, because small overall variance does not necessarily imply a single cluster, and high overall variance is not always an indication of two of more clusters. moreover, variance-based filters depend on the choice of an arbitrary threshold, which can be difficult to choose, since one typically does not know which portion of the genes is responsible for the clustering of the samples. as for step, i.e., dimensionality reduction, considered if the selected genes are still too many, we propose to keep the first principal components, as in. the principal component analysis is a very simple procedure which reduces the dimension of a data set of a large number of interrelated variables, preserving as much as possible of the data set variation. since it has no requirements about the data distribution, it is consistent with our nonparametric strategy. in order to compare our approach to mclust, we carried out a procedure analogous to the one described here, but using the normal-mixture model both in step and. note that this procedure differs from the one in mclachlan et al., because they use a mixture of factor analyzers to select genes and reduce dimensionality. computational issues the further dimensionality reduction in our step is necessary since pdfcluster, in order to compute the delaunay triangulation, exploits the quickhull algorithm. barber et al. state that the quickhull algorithm for finding the convex hull of a set of n points in ℝp requires at most o operations if p ≤ and o where m = ⌊p/2⌋ for p > azzalini and torelli observed that the computing time increases less than quadratically in n for any fixed p, but it increases more than exponentially in p for fixed n. our experience is that for p = the algorithm is very fast core quad cpu q @ ghz). therefore, there are no computationally related problems for step. moreover, with p < it takes a reasonable time to complete the procedure. number of principal components in order to choose the number of principal components, we carried out a small simulation study: we found that, with n = pdfcluster performs at best, in terms of misclassification error, with 3- dimensions, while with p = the misclassification error starts to grow. this is probably due to the extreme dispersion of the observations in higher dimensional spaces. the performances of pdfcluster are slightly better with components, but the improvement does not justify the increased computational time. thus, for the subsequent analyses, we will retain principal components. simulated data in this section, we evaluate our proposal by means of simulated data. for simulating data with structure similar to that of real microarray experiments, we use two schemes, i.e., the gamma-gamma model and the normal-uniform model. in gg model we simulate data with two clusters, such that the majority of genes are equally expressed between the groups and a small fraction of them is differentially expressed. this mimics a classical experiment in which some diseased subjects are compared to healthy controls. in nu model we simulate data with three clusters: cluster consists of samples with up-regulated and down-regulated genes; cluster consists of samples with down-regulated genes; cluster consists of samples with neither up- nor down-regulated genes. note that cluster and are "closer" to each other than to cluster and that cluster has smaller sample size. this mimics a more elaborate design, e.g. two different types of a specific disease versus a normal control. gg model simulation results for pdfcluster and mclust in gg model: rate of correct identification of number of clusters, sensitivity, specificity and error rate in the classification of the samples, and error rate in the selection of relevant genes. more interesting is the very different behaviour in the choice of the relevant genes: pdfcluster is very good in recognizing them, with a very low error rate, while mclust shows a very high error rate. we simulated a relatively small number of "marker" genes; pdfcluster correctly discards the majority of genes as non-relevant in the determination of the clusters, while mclust seems to be too sensitive to outliers, erroneously capturing differences due to random noise. nu model as expected, table shows that in this model both pdfcluster and mclust lead to higher classification errors than in gg model. also in the gene filtering step, both methods have difficulties in finding the relevant genes. simulation results for pdfcluster and mclust in nu model: rate of two clusters identification, rate of three clusters identification, error rate in the classification of samples and error rate in the selection of relevant genes. mclust is able to recognize three clusters in 39% and two clusters in 34% of the simulations; pdfcluster recognizes three clusters in 19% and two clusters in 47% of the simulations. on the other hand, the mean error rate of the final classification is for pdfcluster while for mclust is. this is probably due to the fact that the cases in which pdfcluster correctly recognizes three clusters are those with the most separated clusters among the ones recognized by mclust: obviously, in the less separated clusters cases, it is more difficult to allocate the samples. finally, it is worth noting that pdfcluster outperforms mclust according to the gene selection error rate: as in previous simulation study, pdfcluster works better in recognizing which genes are effectively responsible for the determination of the clusters. sample size one issue with microarray data is often the low sample size. in order to evaluate its effect on the performance of our approach, we simulated data from the nu model, varying the sample size n. for different values of n, namely n = we simulated b = samples in a setting similar to the previous section, i.e., 40% of the observations forming cluster 40% forming cluster and 20% cluster misclassification error rate for pdfcluster and mclust in nu model, varying the sample size. real data along with simulations, we consider two benchmarking real datasets, studied before by several authors, which we will refer to as the colon data and the leukaemia data. colon data as described above, we analyze the dataset, following three steps: gene filtering, dimensionality reduction, clustering of samples. namely, the first step of the procedure consists in applying the cluster algorithm to the univariate distribution of each gene. the genes that show two or more clusters are considered for the further steps. in the first step, the pdfcluster algorithm is able to recognize genes, which discriminate data into two or more groups. we proceed by considering the first three principal components of this reduced data-matrix. the procedure finds three clusters, summarized in table which clearly correspond to biologically meaningful groups. the first cluster consists of tumor tissues, while clusters and comprise normal tissues. it is worth noting that six out of the eight misallocated samples are found to be misclassified in several previous analyses, including. as stated, for instance, in, these six samples are likely to be wrongly labeled. furthermore, getz et al. reported that there was a change in the protocol during the experiments: tumor samples 1- and normal samples 41- were collected within the first protocol, while tumor samples 12- and normal samples 52- were collected within the second. although for the tumor samples our approach did not recognize any difference between the protocols, cluster and cluster split normal tissues in two groups according to the protocols. clusters found after pdfcluster procedure in colon data; tumor samples are labeled 1- normal samples 41-62; misallocated samples are shown in bold. the star represents a wrongly labeled samples. in the first step, mclust is able to find discriminant genes. we consider the first three principal components of this sub-space for clustering. the procedure finds two clusters, with a rather high misclassification error. we also apply the k-means algorithm to the entire dataset. the results of the three approaches are shown in table it can be seen that k-means, exploited in the original p-dimensional space, does not perform well. moreover, pdfcluster outperforms mclust, if one considers cluster and together as the normal samples. confusion matrices for pdfcluster, mclust and k-means with error rates for colon data. as stated before, mclachlan et al. studied the same microarray dataset. they selected relevant genes, achieving clusters that seem to recognize the change of protocol in the data structure, but fail to recognize the normal/tumor differences. nevertheless, they achieved results slightly better than ours considering a particular subspace: they clustered genes in groups and considered only the second group to cluster data. although this approach leads to good results in this example, it seems difficult to reproduce the procedure in an unsupervised setting. leukaemia data as stated in, the leukaemia dataset presents two different problems: an easier one, consisting of separating all from aml and a harder one, consisting also of recognizing the differences in b-cell and t-cell subclasses. again, we consider the strategy previously described. in the filtering step, pdfcluster recognizes discriminant genes. note that the higher number of genes selected with respect to colon data is consistent with the higher difficulty of the problem. we proceed by considering the first three principal components of this subspace. the pdfcluster algorithm finds two clusters, which clearly represent all and aml samples, with aml samples classified as all and all samples classified as aml, leading to a misclassification error rate of: pdfcluster is able to solve the two-class problem, but it misses the three-class problem. confusion matrices for pdfcluster, mclust and k-means with error rates for leukaemia data. in the first step, mclust fails to select relevant genes, recognizing out of genes as discriminant among the groups. based on the first three principal components of the subspace spanned by these genes, mclust clusters samples in four groups. we could interpret the merged clusters 1- as the all b-cell class, and cluster as the aml class, while cluster interpretation is less clear. although mclust is able to find more than two clusters, it fails to distinguish between b-cell and t-cell classes, leading to hardly interpretable clusters. the leukaemia dataset has been studied by mclachlan et al. as well. the authors found relevant genes after the variable selection step. for the two-class problem, their results were very good, but they failed to solve the three-class problem. it should be noted that, unlike our algorithm, the procedure used in needs prior specification of the number of clusters, which is not desirable in an unsupervised learning, especially in cancer tissue classification, where one of the main goals is to find new subclasses of tumors.  model-based approaches to clustering of data have received increasing attention in the last few years, as they provide a sound mathematical-based method. unfortunately, in microarray applications, the high dimensionality of the space makes the clustering of samples in the whole space unfeasible within a model-based framework. here, we have discussed a general strategy for the clustering of microarray expression data, based on gene filtering and dimensionality reduction as preliminary steps in the cluster analysis. we have discussed a nonparametric density estimation-based algorithm within this framework, showing promising results both in simulated data and in two real applications, with surprisingly good computational performances. in our simulation experiments, we have found that pdfcluster leads to slightly better performances than mclust. moreover, the gene filtering step is much more effective using pdfcluster than using mclust both in simulated and in real datasets. here, "effective" means good results in terms of both dimension reduction and of correct selection. here we have used pdfcluster in order to select relevant genes. we underlined the assets of this chioce, but it is clear that any unsupervised technique able to discard the irrelevant genes can be used. similarly, the choice of principal component analysis in the dimensionality reduction step is only one among several possible choices. since there are no guarantees that the first principal components preserve the cluster structure in the reduction of original dimension of data, future efforts could be made in trying different approaches, such as the projection pursuit or the principal curves. nevertheless, in our case the principal component analysis gives good results and provides a low dimensional dataset on which it is feasible to apply a model-based technique such as pdfcluster. all the statistical analyses and simulations have been performed with r and with a public domain implementation of the "quickhull" algorithm available at http://www.qhull.org/. the datasets used are both freely available as bioconductor packages. 
here, we present a general framework to deal with the clustering of microarray data, based on a three-step procedure: gene filtering; dimensionality reduction; clustering of observations in the reduced space. such studies pose challenging statistical problems due to dimensionality issues, since the number of variables can be much higher than the number of observations. cluster analysis is a crucial tool in several biological and medical studies dealing with microarray data. the proposed algorithm is a simple and effective tool for the clustering of microarray data, in an unsupervised setting. via a nonparametric model-based clustering approach we obtain promising results both in simulated and real data.
 cluster analysis is a crucial tool in several biological and medical studies dealing with microarray data. such studies pose challenging statistical problems due to dimensionality issues, since the number of variables can be much higher than the number of observations.  here, we present a general framework to deal with the clustering of microarray data, based on a three-step procedure: gene filtering; dimensionality reduction; clustering of observations in the reduced space. via a nonparametric model-based clustering approach we obtain promising results both in simulated and real data.  the proposed algorithm is a simple and effective tool for the clustering of microarray data, in an unsupervised setting. 