BACKGROUND
the analysis of gene expression data generated by microarrays, such as the high-density oligonucleotide microarays produced by affymetrix , is an often laborious process in which a basic understanding of molecular biology, computer science and statistics is required.
in a typical microarray experiment, rna obtained under various conditions  is hybridised to microarrays.
by tagging the rna with a fluorescent marker, intensity values can be obtained that correspond to the amount of labeled rna bound to the array.
on the widely used affymetrix platform, gene expression is measured using probe sets consisting of  <dig> to  <dig> perfect match  probes of  <dig> nucleotides, which are complementary to a target sequence, and a similar number of mismatch  probes in which the 13th nucleotide has been changed.
the mm probe measurements are thought to comprise most of the background cross-hybridization and stray signal affecting the pm probes.
normalization of probe intensity values is performed to remove any non-biological variation.
the individual probe measurements are then summarized as probe set expression levels, as estimates of the amount of specific mrna present in the biological sample.
normalization and probe set summarization are statistical procedures for which several methods have been developed.
microarray suite , a software package provided by affymetrix, normalizes intensities using a global scaling procedure and measures expression using a one-step tukey biweight algorithm, which is defined as the anti-log of a robust average of differences between log and log  <cit> .
the same algorithms are implemented in the software package currently provided by affymetrix, gcos.
one of the first alternatives to this approach was provided by li and wong with the dchip-method, which scales the intensity data towards the median intensity in a group of arrays and then uses model-based index estimates, giving variable weight to pm-mm probe pairs of a probe set based on variance between arrays, to measure expression  <cit> .
irizarry et al.
introduced rma , later followed by gcrma .
rma, often preceded by quantile normalization  <cit> , applies a median polish procedure to pm intensities only in summarization.
gcrma is based on a similar model as rma but takes into account the effect of stronger bonding of g/c pairs  <cit> .
an overview of these methods is shown in table  <dig>  other normalization methods, such as the variance stabilizing normalization  and summarization methods, such as plier  <cit> , have been developed, but are less frequently applied.
various studies have been published which assess the differences in outcome of these different data pre-processing methods  <cit> .
to validate and test pre-processing methods, two publicly available datasets are commonly used.
the latin square dataset provided by affymetrix  <cit>  contains spiked-in crna's at several concentrations facilitating the assessment of the relation between mrna concentration and expression value.
the genelogic dilution series  gives an estimate of the relation between actual and measured differential expression.
based on these datasets, an online benchmark tool has been developed to encourage authors to test their method  <cit> .
this tool assesses quality of pre-processing using several parameters in five different groups:  variability of expression across replicate arrays,  response of expression measure to changes in abundance of rna,  sensitivity of fold-change measures to the amount of actual rna sample,  accuracy of fold-change as a measure of relative expression and  usefulness of raw fold-change score for the detection of differential expression.
pronounced differences between different procedures have been shown to occur  <cit> .
the studies on the latin square and dilution data were performed using data generated specifically for this purpose, allowing comparisons of specific analyses, showing accurately which methods perform best.
in effect, statistical properties of the various estimators are tested.
several authors noted that the use of two special-purpose datasets for calibration of statistical procedures creates a risk on overfitting of the available data and therefore focused on using experimental data to compare methods with respect to the sets of differentially expressed genes found  <cit> .
this sometimes lead to contradictory results, where for instance a study using the latin square dataset showed the mas <dig>  method to outperform the dchip method on detecting differentially expressed genes  <cit> , while a study on experimental data showed the opposite  <cit> .
therefore, more work is needed to reliably establish how important the effect of choice of pre-processing method is in every-day practice, especially when analyses such as clustering and classification are applied.
in this paper, we focus on one practical application of microarrays: patient-cohort studies  <cit> .
in such studies, researchers typically select sets of genes that are differentially expressed between certain known conditions, a supervised analysis.
moreover, unsupervised techniques  such as clustering are applied to detect biological relations between samples or genes by grouping them according to their expression profiles.
often the goal is to obtain a predictor  for, for instance, prognostically relevant categories, using supervised analysis.
given that different pre-processing procedures will influence the outcome of these analyses, several questions can be asked, such as: how well is expression measured using a number of different pre-processing methods?
what is their effect on the detection of differentially expressed genes, clusters found and classification results?
by focusing on practical applications of microarray studies, we hope to give insight into the relevance of different pre-processing procedures to biology-oriented researchers.
RESULTS
the aim of our work was to evaluate the effect of several microarray data pre-processing methods on the outcome of analyses commonly applied in patient-cohort studies.
four types of analysis were performed:  expression level measurement,  detection of differential expression ,  cluster analysis  and  classification of samples .
these analyses were applied to two publicly available datasets, one of  <dig> acute myeloid leukemia  samples profiled on affymetrix hg-u133a genechips  <cit>  and one consisting of  <dig> affymetrix hugenefl genechips hybridized with central nervous system  tumor tissue  <cit> .
comparing expression levels to rt-pcr
pearson correlation coefficients between expression levels of evi <dig>  cebpa, meis <dig>  hoxa <dig>  hoxa <dig>  trka, prdm <dig>  prdm <dig>  p <dig> and gmcsf found in the aml dataset, measured by rt-pcr and microarray after pre-processing using different methods, are listed in table  <dig>  the actual expression levels measured by rt-pcr and the affymetrix probe sets  are listed in supp.
table  <dig> .
average correlation to rt-pcr expression is  <dig> – <dig>  for the different methods with rma  and gcrma  showing the highest correlation on average, dchip  scoring lowest and mas  scoring intermediate.
no significant differences have been found between correlations of different pre-processing methods and rt-pcr data and  no pre-processing method unequivocally performed best in measuring expression level.
correlation overall is moderate, but this result is likely to be influenced by different genomic location of rt-pcr primers and affymetrix probes, resulting in different expression values when alternative splicing occurs; by incorrect annotation of individual probe sets ; and by suboptimal rt-pcr primer and affymetrix probe design.
comparing expression levels between pre-processing methods
correlations are depicted in figure 1a for the aml dataset and in figure 1b for the cns dataset for each probe set present on the microarray, ordered by average expression level over the four differently pre-processed datasets.
overall, a clear trend of increasing correlation at increasing expression levels is apparent, which has been noticed before  <cit> .
aside from a dense area of highly correlated genes with intermediate to high expression, in several comparisons, for instance that of rma to mas, a second more densely populated area is visible in the range of extremely low expression levels.
these expression levels correspond to non-expressed genes .
at these levels, variability is relatively higher, resulting in moderate correlations.
as the normalization method of rma and gcrma is the same  and their summarization methods are very similar, it is not surprising that these methods show the highest resemblance in measured expression.
however, they show more agreement with mas than with dchip .
perhaps this has to do with the fact that dchip calculates expression on the original probe intensity values rather than the log-transformed ones used by the other methods.
the cns dataset shows similar trends, but the much higher level of variation suggests that sample size and/or quality of the platform and biological sample have a much more profound effect on estimated expression levels, than has the pre-processing method.
in conclusion, the aml dataset shows that variation in estimated expression levels exists between different pre-processing methods and that this variation is higher at lower mrna concentrations.
the clear trend of increasing correlation with increasing expression level suggests that pre-processing has an influence, but this concerns only a minority of probe sets.
using the affymetrix latin square dataset, rajagopalan noted that mas and dchip perform equally well on estimating expression levels, with a small non-significant advantage for mas  <cit> .
although trends towards mas seem visible in our study as well, mas and dchip behave rather differently in the experimental dataset used here.
testing not only the latin square dataset but also the genelogic dataset, irizarry et al.
conclude that rma shows highest sensitivity and specificity when compared to dchip and the avdiff algorithm  <cit> .
as no method performs significantly different in our study, these results are not confirmed
differential expression
significance of differences in expression when comparing two conditions was calculated using three standard methods: the t-test; the wilcoxon rank sum test, controlling the family-wise error rate ; and significance analysis of microarrays , a test controlling the false discovery rate  using a statistic resembling that of the t-test  <cit> .
different pre-processing methods were compared by assessing the overlap in the number of probe sets marked as differentially expressed by two pre-processing methods.
we call a probe set differentially expressed below an fwer or fdr of 5%.
in the aml-dataset, p-values  and q-values  were computed for samples with recurrent flt <dig> itd mutations vs. the rest, inv vs. the rest, t vs. the rest and t vs. the rest.
in the cns-dataset, p-values and q-values were computed for pnet-, rhab-, glio- and med-samples vs. the rest, respectively.
although different subdivisions into conditions were thus compared, the outcomes are remarkably similar.
considering the aml dataset, the overlap between the probe sets selected on rma and gcrma pre-processed data is most striking, with a minimum r of  <dig>  .
overall, the overlap between different pre-processing methods is considerable: a minimum r of  <dig>   is found, independent of the statistical test used.
the combination mas-dchip comes up as least comparable ).
mas shows higher concordance with rma and gcrma than does dchip.
no indications were found that r will increase for smaller fwer or fdr .
the overlap between probe set lists detected as differentially expressed is considerably less in the cns dataset than in the aml dataset and there is more variation, which could be due to the higher amount of noise in this dataset and/or its smaller sample size.
+the rma-gcrma comparison results in an average r of  <dig>  ).
again, pre-processing with mas will result in less differences with rma pre-processing than with dchip ).
overall, average r is  <dig>  for this dataset.
when using q-values, again rma and/or mas often detect larger numbers of differentially expressed probe sets than dchip and gcrma.
note also that the difference between the number of probe sets selected using the t-statistic and the wilcoxon statistic is larger than for the aml dataset.
this may be caused by outlier data on the hugenefl microarrays, to which the t-test is more susceptible.
in a study evaluating experimental datasets of  <dig> ovary tumors and  <dig> colon tumors profiled on the affymetrix hg-u133a platform, shedden et al.
<cit>  show that dchip results are closer to those obtained using rma than to those obtained using mas, an observation not confirmed by our results.
statistical tests on rma and mas pre-processed data detect the largest number of differentially expressed probe sets in most cases, where gcrma and dchip select less, with a maximum difference in number of selected probe sets of  <dig> % in the aml dataset , q-values).
this does not confirm the observations of shedden et al., who found that dchip outperformed mas and gcrma in terms of sensitivity.
irizarry et al.
<cit>  report that rma performs better than dchip and the avdiff algorithm in finding truly differentially expressed genes.
no statement on the true nature of probe sets measured as differentially expressed here can be made.
however, mas and rma score roughly equal numbers of probe sets as differentially expressed and both methods find more probe sets to be differentially expressed than dchip and gcrma, as in  <cit> .
recently, hoffmann et al.
<cit>  stated that normalization will have a larger influence on the number of differentially expressed genes than the actual statistical test used.
although a direct comparison of  <cit>  and our work is not possible due to differences in multiple testing correction, in our  larger datasets we observe a larger difference between the number of probe sets selected as a result of the multiple testing correction used  than as a result of the choice of pre-processing method.
overall, the overlap between sets of genes selected as differentially expressed is considerable when pre-processing the data using different methods and overlap increases when non-biological variation decreases.
using the current datasets, it is not possible to give indications of the quality of probe sets selected, due to the lack of ground truth.
cluster analysis
data resulting from different pre-processing methods was clustered by k-means  and hierarchical clustering with single, average and complete linkage .
clusterings of both the aml and cns datasets were compared using the jaccard index; results are shown in figure  <dig> and supp.
figure  <dig>   <cit> .
rma and gcrma results are often similar, which is to be expected.
dchip results frequently differ from results obtained using other pre-processing methods.
in general, km, hc/a and hc/c on both datasets show jaccard indices of  <dig> – <dig> .
hc/s shows higher indices, but the actual resulting clusterings are very poor due to the well-known high susceptibility of this method to outliers : almost all samples end up in a single cluster, the remaining samples form individual clusters.
as an example, the confusion matrix in table  <dig> shows that many clusters found using the mas pre-processed dataset are also found reasonably well using the rma pre-processed dataset .
however, as there are  <dig> sample pairs co-occurring in a cluster in both clustering results,  <dig> sample pairs co-occurring in a cluster in the mas clustering result only and  <dig> sample pairs co-occurring in a cluster in the rma result only, this leads to a jaccard-index j of only 2716/ =  <dig> .
in an attempt to quantify the sensitivity of clusterings found to small perturbations, stability-normalized jaccard indices jsn were therefore calculated, indicating to what extent the jaccard indices j found are out of the ordinary.
figure 3a illustrates that for km and the pair of pre-processing methods used , j =  <dig>  is actually better than the jaccard index obtained on average on a slightly changed version of the mas pre-processed dataset , but worse than that obtained on average on a slightly changed version of the rma pre-processed dataset .
the cns dataset  largely tells the same story, although the jsn are somewhat larger, especially for k-means clustering.
this is due to the smaller sample size: leaving out 10% of the samples relatively has more impact on the jaccard indices.
supp.
figures  <dig> and  <dig>  illustrate the influence of the choice of the number of clusters  and the distance measure .
both datasets show the same effects.
for lower k , both jaccard indices and stability-normalized jaccard indices are much higher, as clusterings of data pre-processed by the various methods agree on structure clearly present in the data.
for higher k , jaccard indices and stability-normalized jaccard indices are similar to or even lower than those for the k chosen originally.
using euclidean distance leads to slightly lower jaccard indices, with an increase in difference between note however the more pronounced differences between how dchip and other methods.
this may be the result of the negative values it produces rma), which are thresholded at  <dig>  in the data transformation steps.
due to the centering by the geometric mean this can lead to larger extreme probe set values over arrays.
in conclusion, clustering results are sensitive to the choice of pre-processing method.
this sensitivity is smallest for small numbers of clusters k  and when using correlation distance.
additionally, using rma seems to result in more stable clusterings than using mas or dchip.
classification
a number of different classification problems defined on the datasets have been approached using several classifiers trained on data of all pre-processing methods.
resulting performances are listed in tables  <dig>   <dig>   <dig>   <dig> and supp.
table  <dig> .
results are reported only for the number of probe sets giving lowest average test set error over the four methods.
although this makes the performance estimates biased, it does not influence comparison between methods.
in the aml dataset, inversion of chromosome  <dig> is well predictable, with error rates smaller than 5% .
differences in error rate between classification algorithms are very small: although the nearest centroid classifier often performs worst, no algorithm performs significantly better than others.
more importantly, no pre-processing method scores significantly better or worse than others .
although predicted with a higher error rate, these observations are confirmed on the flt <dig>  and ccr  aml problems.
interestingly, this also holds for the cns dataset ), although performances show much more variation and mas no longer comes out best.
ofcourse, the cns dataset is rather small, so obtaining good classifiers is harder.
no classifier or pre-processing method scores significantly better than others.
this can be explained by the fact that the probe sets on which classification is based are already selected to give good classification results: on differently pre-processed datasets, different probe sets may be selected .
the six classifiers used seem to be equally susceptible to different pre-processing methods; that is, for each of them performance varies with pre-processing method used in at least some of the problems.
for classification, the choice of pre-processing method  seems to be irrelevant.
CONCLUSIONS
patient-cohort studies using microarrays are often performed to find pathobiologically relevant relations between genes and patient classes.
the affymetrix platform has become increasingly popular for this type of study.
processing intensity values obtained using affymetrix genechips remains a challenging task for many microarray researchers.
apart from the affymetrix mas procedure, several statistical procedures have been proposed to assess expression, such as dchip, rma and gcrma.
our study has tried to estimate the effects of the choice of pre-processing method from a practical viewpoint.
to this end, we have applied a number of analyses to two datasets, which we believe to represent two extremes in recent patient-cohort studies both in terms of sample size and of platform used.
the experimental results indicate that the normalization step in rma has a larger effect on the data than the one in mas and dchip, but this cannot be separated from the effect of applying different models for summarization.
and, although the dchip and rma summarization models are more related to each other than the mas and rma ones, mas pre-processed data shows more similarity to rma than does dchip.
in practical terms, the question of which method will give expression value estimates closest to the actual data is still to be answered; this study has not attempted to answer it, because we have not used data with accompanying ground truth.
we showed that results of various analyses are not always dependent on the choice of pre-processing method.
analyses such as calculating expression levels or assessing differential expression are reasonably susceptible to differences between pre-processing methods; clustering as well, except when looking for clearly present structure ; but classification far less so.
the message is that while care should be taken in assigning biological meaning to individual probe set measurements, this holds less for global statements about the data.
several other studies have been performed to assess the level of concordance in differential gene sets between pre-processing methods and noted that the choice of the method was of major influence, with different studies favoring different pre-processing methods  <cit> .
our results do not conclusively confirm one or more studies, although results partially overlap.
one major difference with other studies is the size of the used datasets, where one of the datasets used in this study is considerably larger.
it is to be expected that with the evolution of the array technology, the number of profiled samples in any single patient-cohort study is likely to increase.
the effects of the choice of pre-processing method are far more profound in the cns dataset than in the aml dataset.
several possible explanations can be given for this, but it is not possible to single any of them out based only on the two datasets used in this study.
the aml dataset contains more samples, which allows for better parameter estimates in the analysis methods presented in this work.
furthermore, affymetrix technology has evolved over time, resulting in a more stable platform for the aml dataset  than the cns dataset .
biological differences also play a role in the two datasets.
the amount of viable cells obtained from bone marrow is also likely to be higher compared to solid tumors, which often show necrotic areas, leading to difference in rna-quality and -degradation.
also, tumor cells can be purified from bone marrow samples using ficoll-centrifugation, a technique which is not available for the solid tumors which were hybridized in the cns dataset, resulting in less contamination with other cell types in hybridized samples, which is known to be an important factor  <cit> .
we recommend that the emphasis in setting up a large microarray-based study should therefore be on the quality of the biological sample and the quality of rna rather than on the choice of the pre-processing procedure.
however, we do believe that an inverse relation exists, with the importance of the method of normalization and expression summarization increasing when the quality of the biological sample and the number of studied samples decrease.
although we base this on a limited number of pre-processing methods and data sets, we think that taking into account more available methods will have no effect on our conclusion.

 the analysis of gene expression data generated by microarrays, such as the high-density oligonucleotide microarays produced by affymetrix, is an often laborious process in which a basic understanding of molecular biology, computer science and statistics is required. in a typical microarray experiment, rna obtained under various conditions is hybridised to microarrays. by tagging the rna with a fluorescent marker, intensity values can be obtained that correspond to the amount of labeled rna bound to the array. on the widely used affymetrix platform, gene expression is measured using probe sets consisting of to perfect match probes of nucleotides, which are complementary to a target sequence, and a similar number of mismatch probes in which the 13th nucleotide has been changed. the mm probe measurements are thought to comprise most of the background cross-hybridization and stray signal affecting the pm probes. normalization of probe intensity values is performed to remove any non-biological variation. the individual probe measurements are then summarized as probe set expression levels, as estimates of the amount of specific mrna present in the biological sample. normalization and probe set summarization are statistical procedures for which several methods have been developed. microarray suite, a software package provided by affymetrix, normalizes intensities using a global scaling procedure and measures expression using a one-step tukey biweight algorithm, which is defined as the anti-log of a robust average of differences between log and log. the same algorithms are implemented in the software package currently provided by affymetrix, gcos. one of the first alternatives to this approach was provided by li and wong with the dchip-method, which scales the intensity data towards the median intensity in a group of arrays and then uses model-based index estimates, giving variable weight to pm-mm probe pairs of a probe set based on variance between arrays, to measure expression. irizarry et al. introduced rma, later followed by gcrma. rma, often preceded by quantile normalization, applies a median polish procedure to pm intensities only in summarization. gcrma is based on a similar model as rma but takes into account the effect of stronger bonding of g/c pairs. an overview of these methods is shown in table other normalization methods, such as the variance stabilizing normalization and summarization methods, such as plier, have been developed, but are less frequently applied. various studies have been published which assess the differences in outcome of these different data pre-processing methods. to validate and test pre-processing methods, two publicly available datasets are commonly used. the latin square dataset provided by affymetrix contains spiked-in crna's at several concentrations facilitating the assessment of the relation between mrna concentration and expression value. the genelogic dilution series gives an estimate of the relation between actual and measured differential expression. based on these datasets, an online benchmark tool has been developed to encourage authors to test their method. this tool assesses quality of pre-processing using several parameters in five different groups: variability of expression across replicate arrays, response of expression measure to changes in abundance of rna, sensitivity of fold-change measures to the amount of actual rna sample, accuracy of fold-change as a measure of relative expression and usefulness of raw fold-change score for the detection of differential expression. pronounced differences between different procedures have been shown to occur. the studies on the latin square and dilution data were performed using data generated specifically for this purpose, allowing comparisons of specific analyses, showing accurately which methods perform best. in effect, statistical properties of the various estimators are tested. several authors noted that the use of two special-purpose datasets for calibration of statistical procedures creates a risk on overfitting of the available data and therefore focused on using experimental data to compare methods with respect to the sets of differentially expressed genes found. this sometimes lead to contradictory results, where for instance a study using the latin square dataset showed the mas method to outperform the dchip method on detecting differentially expressed genes, while a study on experimental data showed the opposite. therefore, more work is needed to reliably establish how important the effect of choice of pre-processing method is in every-day practice, especially when analyses such as clustering and classification are applied. in this paper, we focus on one practical application of microarrays: patient-cohort studies. in such studies, researchers typically select sets of genes that are differentially expressed between certain known conditions, a supervised analysis. moreover, unsupervised techniques such as clustering are applied to detect biological relations between samples or genes by grouping them according to their expression profiles. often the goal is to obtain a predictor for, for instance, prognostically relevant categories, using supervised analysis. given that different pre-processing procedures will influence the outcome of these analyses, several questions can be asked, such as: how well is expression measured using a number of different pre-processing methods? what is their effect on the detection of differentially expressed genes, clusters found and classification results? by focusing on practical applications of microarray studies, we hope to give insight into the relevance of different pre-processing procedures to biology-oriented researchers.  the aim of our work was to evaluate the effect of several microarray data pre-processing methods on the outcome of analyses commonly applied in patient-cohort studies. four types of analysis were performed: expression level measurement, detection of differential expression, cluster analysis and classification of samples. these analyses were applied to two publicly available datasets, one of acute myeloid leukemia samples profiled on affymetrix hg-u133a genechips and one consisting of affymetrix hugenefl genechips hybridized with central nervous system tumor tissue. comparing expression levels to rt-pcr pearson correlation coefficients between expression levels of evi cebpa, meis hoxa hoxa trka, prdm prdm p and gmcsf found in the aml dataset, measured by rt-pcr and microarray after pre-processing using different methods, are listed in table the actual expression levels measured by rt-pcr and the affymetrix probe sets are listed in supp. table. average correlation to rt-pcr expression is – for the different methods with rma and gcrma showing the highest correlation on average, dchip scoring lowest and mas scoring intermediate. no significant differences have been found between correlations of different pre-processing methods and rt-pcr data and no pre-processing method unequivocally performed best in measuring expression level. correlation overall is moderate, but this result is likely to be influenced by different genomic location of rt-pcr primers and affymetrix probes, resulting in different expression values when alternative splicing occurs; by incorrect annotation of individual probe sets; and by suboptimal rt-pcr primer and affymetrix probe design. comparing expression levels between pre-processing methods correlations are depicted in figure 1a for the aml dataset and in figure 1b for the cns dataset for each probe set present on the microarray, ordered by average expression level over the four differently pre-processed datasets. overall, a clear trend of increasing correlation at increasing expression levels is apparent, which has been noticed before. aside from a dense area of highly correlated genes with intermediate to high expression, in several comparisons, for instance that of rma to mas, a second more densely populated area is visible in the range of extremely low expression levels. these expression levels correspond to non-expressed genes. at these levels, variability is relatively higher, resulting in moderate correlations. as the normalization method of rma and gcrma is the same and their summarization methods are very similar, it is not surprising that these methods show the highest resemblance in measured expression. however, they show more agreement with mas than with dchip. perhaps this has to do with the fact that dchip calculates expression on the original probe intensity values rather than the log-transformed ones used by the other methods. the cns dataset shows similar trends, but the much higher level of variation suggests that sample size and/or quality of the platform and biological sample have a much more profound effect on estimated expression levels, than has the pre-processing method. in conclusion, the aml dataset shows that variation in estimated expression levels exists between different pre-processing methods and that this variation is higher at lower mrna concentrations. the clear trend of increasing correlation with increasing expression level suggests that pre-processing has an influence, but this concerns only a minority of probe sets. using the affymetrix latin square dataset, rajagopalan noted that mas and dchip perform equally well on estimating expression levels, with a small non-significant advantage for mas. although trends towards mas seem visible in our study as well, mas and dchip behave rather differently in the experimental dataset used here. testing not only the latin square dataset but also the genelogic dataset, irizarry et al. conclude that rma shows highest sensitivity and specificity when compared to dchip and the avdiff algorithm. as no method performs significantly different in our study, these results are not confirmed differential expression significance of differences in expression when comparing two conditions was calculated using three standard methods: the t-test; the wilcoxon rank sum test, controlling the family-wise error rate; and significance analysis of microarrays, a test controlling the false discovery rate using a statistic resembling that of the t-test. different pre-processing methods were compared by assessing the overlap in the number of probe sets marked as differentially expressed by two pre-processing methods. we call a probe set differentially expressed below an fwer or fdr of 5%. in the aml-dataset, p-values and q-values were computed for samples with recurrent flt itd mutations vs. the rest, inv vs. the rest, t vs. the rest and t vs. the rest. in the cns-dataset, p-values and q-values were computed for pnet-, rhab-, glio- and med-samples vs. the rest, respectively. although different subdivisions into conditions were thus compared, the outcomes are remarkably similar. considering the aml dataset, the overlap between the probe sets selected on rma and gcrma pre-processed data is most striking, with a minimum r of. overall, the overlap between different pre-processing methods is considerable: a minimum r of is found, independent of the statistical test used. the combination mas-dchip comes up as least comparable ). mas shows higher concordance with rma and gcrma than does dchip. no indications were found that r will increase for smaller fwer or fdr. the overlap between probe set lists detected as differentially expressed is considerably less in the cns dataset than in the aml dataset and there is more variation, which could be due to the higher amount of noise in this dataset and/or its smaller sample size. +the rma-gcrma comparison results in an average r of ). again, pre-processing with mas will result in less differences with rma pre-processing than with dchip ). overall, average r is for this dataset. when using q-values, again rma and/or mas often detect larger numbers of differentially expressed probe sets than dchip and gcrma. note also that the difference between the number of probe sets selected using the t-statistic and the wilcoxon statistic is larger than for the aml dataset. this may be caused by outlier data on the hugenefl microarrays, to which the t-test is more susceptible. in a study evaluating experimental datasets of ovary tumors and colon tumors profiled on the affymetrix hg-u133a platform, shedden et al. show that dchip results are closer to those obtained using rma than to those obtained using mas, an observation not confirmed by our results. statistical tests on rma and mas pre-processed data detect the largest number of differentially expressed probe sets in most cases, where gcrma and dchip select less, with a maximum difference in number of selected probe sets of % in the aml dataset, q-values). this does not confirm the observations of shedden et al., who found that dchip outperformed mas and gcrma in terms of sensitivity. irizarry et al. report that rma performs better than dchip and the avdiff algorithm in finding truly differentially expressed genes. no statement on the true nature of probe sets measured as differentially expressed here can be made. however, mas and rma score roughly equal numbers of probe sets as differentially expressed and both methods find more probe sets to be differentially expressed than dchip and gcrma, as in. recently, hoffmann et al. stated that normalization will have a larger influence on the number of differentially expressed genes than the actual statistical test used. although a direct comparison of and our work is not possible due to differences in multiple testing correction, in our larger datasets we observe a larger difference between the number of probe sets selected as a result of the multiple testing correction used than as a result of the choice of pre-processing method. overall, the overlap between sets of genes selected as differentially expressed is considerable when pre-processing the data using different methods and overlap increases when non-biological variation decreases. using the current datasets, it is not possible to give indications of the quality of probe sets selected, due to the lack of ground truth. cluster analysis data resulting from different pre-processing methods was clustered by k-means and hierarchical clustering with single, average and complete linkage. clusterings of both the aml and cns datasets were compared using the jaccard index; results are shown in figure and supp. figure. rma and gcrma results are often similar, which is to be expected. dchip results frequently differ from results obtained using other pre-processing methods. in general, km, hc/a and hc/c on both datasets show jaccard indices of –. hc/s shows higher indices, but the actual resulting clusterings are very poor due to the well-known high susceptibility of this method to outliers: almost all samples end up in a single cluster, the remaining samples form individual clusters. as an example, the confusion matrix in table shows that many clusters found using the mas pre-processed dataset are also found reasonably well using the rma pre-processed dataset. however, as there are sample pairs co-occurring in a cluster in both clustering results, sample pairs co-occurring in a cluster in the mas clustering result only and sample pairs co-occurring in a cluster in the rma result only, this leads to a jaccard-index j of only 2716/ =. in an attempt to quantify the sensitivity of clusterings found to small perturbations, stability-normalized jaccard indices jsn were therefore calculated, indicating to what extent the jaccard indices j found are out of the ordinary. figure 3a illustrates that for km and the pair of pre-processing methods used, j = is actually better than the jaccard index obtained on average on a slightly changed version of the mas pre-processed dataset, but worse than that obtained on average on a slightly changed version of the rma pre-processed dataset. the cns dataset largely tells the same story, although the jsn are somewhat larger, especially for k-means clustering. this is due to the smaller sample size: leaving out 10% of the samples relatively has more impact on the jaccard indices. supp. figures and illustrate the influence of the choice of the number of clusters and the distance measure. both datasets show the same effects. for lower k, both jaccard indices and stability-normalized jaccard indices are much higher, as clusterings of data pre-processed by the various methods agree on structure clearly present in the data. for higher k, jaccard indices and stability-normalized jaccard indices are similar to or even lower than those for the k chosen originally. using euclidean distance leads to slightly lower jaccard indices, with an increase in difference between note however the more pronounced differences between how dchip and other methods. this may be the result of the negative values it produces rma), which are thresholded at in the data transformation steps. due to the centering by the geometric mean this can lead to larger extreme probe set values over arrays. in conclusion, clustering results are sensitive to the choice of pre-processing method. this sensitivity is smallest for small numbers of clusters k and when using correlation distance. additionally, using rma seems to result in more stable clusterings than using mas or dchip. classification a number of different classification problems defined on the datasets have been approached using several classifiers trained on data of all pre-processing methods. resulting performances are listed in tables and supp. table. results are reported only for the number of probe sets giving lowest average test set error over the four methods. although this makes the performance estimates biased, it does not influence comparison between methods. in the aml dataset, inversion of chromosome is well predictable, with error rates smaller than 5%. differences in error rate between classification algorithms are very small: although the nearest centroid classifier often performs worst, no algorithm performs significantly better than others. more importantly, no pre-processing method scores significantly better or worse than others. although predicted with a higher error rate, these observations are confirmed on the flt and ccr aml problems. interestingly, this also holds for the cns dataset ), although performances show much more variation and mas no longer comes out best. ofcourse, the cns dataset is rather small, so obtaining good classifiers is harder. no classifier or pre-processing method scores significantly better than others. this can be explained by the fact that the probe sets on which classification is based are already selected to give good classification results: on differently pre-processed datasets, different probe sets may be selected. the six classifiers used seem to be equally susceptible to different pre-processing methods; that is, for each of them performance varies with pre-processing method used in at least some of the problems. for classification, the choice of pre-processing method seems to be irrelevant.  patient-cohort studies using microarrays are often performed to find pathobiologically relevant relations between genes and patient classes. the affymetrix platform has become increasingly popular for this type of study. processing intensity values obtained using affymetrix genechips remains a challenging task for many microarray researchers. apart from the affymetrix mas procedure, several statistical procedures have been proposed to assess expression, such as dchip, rma and gcrma. our study has tried to estimate the effects of the choice of pre-processing method from a practical viewpoint. to this end, we have applied a number of analyses to two datasets, which we believe to represent two extremes in recent patient-cohort studies both in terms of sample size and of platform used. the experimental results indicate that the normalization step in rma has a larger effect on the data than the one in mas and dchip, but this cannot be separated from the effect of applying different models for summarization. and, although the dchip and rma summarization models are more related to each other than the mas and rma ones, mas pre-processed data shows more similarity to rma than does dchip. in practical terms, the question of which method will give expression value estimates closest to the actual data is still to be answered; this study has not attempted to answer it, because we have not used data with accompanying ground truth. we showed that results of various analyses are not always dependent on the choice of pre-processing method. analyses such as calculating expression levels or assessing differential expression are reasonably susceptible to differences between pre-processing methods; clustering as well, except when looking for clearly present structure; but classification far less so. the message is that while care should be taken in assigning biological meaning to individual probe set measurements, this holds less for global statements about the data. several other studies have been performed to assess the level of concordance in differential gene sets between pre-processing methods and noted that the choice of the method was of major influence, with different studies favoring different pre-processing methods. our results do not conclusively confirm one or more studies, although results partially overlap. one major difference with other studies is the size of the used datasets, where one of the datasets used in this study is considerably larger. it is to be expected that with the evolution of the array technology, the number of profiled samples in any single patient-cohort study is likely to increase. the effects of the choice of pre-processing method are far more profound in the cns dataset than in the aml dataset. several possible explanations can be given for this, but it is not possible to single any of them out based only on the two datasets used in this study. the aml dataset contains more samples, which allows for better parameter estimates in the analysis methods presented in this work. furthermore, affymetrix technology has evolved over time, resulting in a more stable platform for the aml dataset than the cns dataset. biological differences also play a role in the two datasets. the amount of viable cells obtained from bone marrow is also likely to be higher compared to solid tumors, which often show necrotic areas, leading to difference in rna-quality and -degradation. also, tumor cells can be purified from bone marrow samples using ficoll-centrifugation, a technique which is not available for the solid tumors which were hybridized in the cns dataset, resulting in less contamination with other cell types in hybridized samples, which is known to be an important factor. we recommend that the emphasis in setting up a large microarray-based study should therefore be on the quality of the biological sample and the quality of rna rather than on the choice of the pre-processing procedure. however, we do believe that an inverse relation exists, with the importance of the method of normalization and expression summarization increasing when the quality of the biological sample and the number of studied samples decrease. although we base this on a limited number of pre-processing methods and data sets, we think that taking into account more available methods will have no effect on our conclusion. 
using two publicly available datasets, i.e., gene-expression data of patients with acute myeloid leukemia and samples of tumor tissue of the embryonal central nervous system, we tested the effect of the four pre-processing strategies mentioned above, on expression level measurements, detection of differential expression, cluster analysis and classification of samples. analyses on individual probe sets, such as testing for differential expression, are affected most; supervised, multivariate analyses such as classification are far less sensitive to pre-processing. in most cases, the effect of pre-processing is relatively small compared to other choices made in an analysis for the aml dataset, but has a more profound effect on the outcome of the cns dataset. intensity values measured by affymetrix microarrays have to be both normalized, to be able to compare different microarrays by removing non-biological variation, and summarized, generating the final probe set expression values. this study assesses the effect of applying different pre-processing methods on the results of analyses of large affymetrix datasets. using two experimental datasets, we show that the choice of pre-processing method is of relatively minor influence on the final analysis outcome of large microarray studies whereas it can have important effects on the results of a smaller study. by focusing on practical applications of microarray-based research, this study provides insight into the relevance of pre-processing procedures to biology-oriented researchers.
 intensity values measured by affymetrix microarrays have to be both normalized, to be able to compare different microarrays by removing non-biological variation, and summarized, generating the final probe set expression values. various pre-processing techniques, such as dchip, gcrma, rma and mas have been developed for this purpose. this study assesses the effect of applying different pre-processing methods on the results of analyses of large affymetrix datasets. by focusing on practical applications of microarray-based research, this study provides insight into the relevance of pre-processing procedures to biology-oriented researchers.  using two publicly available datasets, i.e., gene-expression data of patients with acute myeloid leukemia and samples of tumor tissue of the embryonal central nervous system, we tested the effect of the four pre-processing strategies mentioned above, on expression level measurements, detection of differential expression, cluster analysis and classification of samples. in most cases, the effect of pre-processing is relatively small compared to other choices made in an analysis for the aml dataset, but has a more profound effect on the outcome of the cns dataset. analyses on individual probe sets, such as testing for differential expression, are affected most; supervised, multivariate analyses such as classification are far less sensitive to pre-processing.  using two experimental datasets, we show that the choice of pre-processing method is of relatively minor influence on the final analysis outcome of large microarray studies whereas it can have important effects on the results of a smaller study. the data source is potentially of bigger importance than the choice of pre-processing method. 