BACKGROUND
a fundamental step in most microarray experiments is determining one or more short lists of differentially expressed genes  that distinguish biological conditions, such as disease from health.
challenges regarding the reliability of microarray results have largely been founded on the inability of researchers to replicate deg lists across highly similar experiments.
for example, tan et al.
<cit>  found only four common degs using an identical set of rna samples across three popular commercial platforms.
independent studies by the groups of ramalho-santos  <cit>  and ivanova  <cit>  of stem cell-specific genes using the same affymetrix platform and similar study design found a disappointing six common degs among about  <dig> identified in each study  <cit> .
a comparative neurotoxicological study by miller et al.
<cit>  using the same set of rna samples found only  <dig> common degs among  <dig> and  <dig>  respectively, from affymetrix and codelink platforms.
all these studies ranked genes by p-value from simple t-tests, used a p threshold to identify deg lists, and applied the concept of the percentage of overlapping genes , or the venn diagram, between deg lists as the measure of reproducibility.
criticism of and concerns about microarrays continue to appear in some of the most prestigious scientific journals  <cit> , leading to a growing negative perception regarding microarray reproducibility, and hence reliability.
however, in reanalyzing the data set of tan et al.
<cit> , shi et al.
<cit>  found that cross-platform concordance was markedly improved when either simple fold change  or significance analysis of microarrays   <cit>  methods were used to rank order genes before determining deg lists.
the awareness that microarray reproducibility is sensitive to how degs are identified was, in fact, a major motivator for the microarray quality control  project  <cit> .
several plausible explanations and solutions have been proposed to interpret and address the apparent lack of reproducibility and stability of deg lists from microarray studies.
larger sample sizes  <cit> ; novel, microarray-specific statistical methods  <cit> ; more accurate array annotation information by mapping probe sequences across platforms  <cit> ; eliminating absent call genes from data analysis  <cit> ; improving probe design to minimize cross-hybridization  <cit> ; standardizing manufacturing processes  <cit> ; and improving data quality by fully standardizing sample preparation and hybridization procedures are among the suggestions for improvement  <cit> .
the maqc study  <cit>  was specifically designed to address these previously identified sources of variability in deg lists.
two distinct rna samples, stratagene universal human reference rna  and ambion human brain reference rna , with thousands of differentially expressed genes, were prepared in sufficient quantities and distributed to three different laboratories for each of the five different commercial whole genome microarray platforms participating in the study.
for each platform, each sample was analyzed using five technical replicates with standardized procedures for sample processing, hybridization, scanning, data acquisition, data preprocessing, and data normalization at each site.
the probe sequence information was used to generate a stringent mapping of genes across the different platforms and  <dig> genes were further analyzed with taqman® assays using the same rna samples.
in addition to assessing the technical performance of different microarray platforms, the maqc study also discussed the idea of using fold-change ranking along with a non-stringent p-value cutoff for selecting degs  <cit> .
however, a lot of detailed results have not been formally published to support the idea  <cit> .
the maqc project, while positively received by the community  <cit> , also stimulated criticism from the statistical community about appropriate ways of identifying degs  <cit> .
to help the microarray community better understand the issue at debate and move forward, in this study, we conducted a careful analysis of these maqc data sets, along with numerical simulations and mathematical arguments.
we demonstrate that the reported lack of reproducibility of deg lists can be attributed in large part to identifying degs from simple t-tests without consideration of fc.
the finding holds for intra-laboratory, inter-laboratory, and cross-platform comparisons independent of sample pairs and normalization methods, and is increasingly apparent with decreasing number of genes selected.
as a basic procedure for improving reproducibility while balancing specificity and sensitivity, choosing genes using a combination of fc-ranking and p threshold was investigated.
this joint criterion results in deg lists with much higher pog, commensurate with better reproducibility, than lists generated by t-test p alone, even when selecting a relatively small numbers of genes.
an fc criterion explicitly incorporates the measured quantity to enhance reproducibility, whereas a p criterion incorporates control of sensitivity and specificity.
the results increase our confidence in the reproducibility of microarray studies while supporting a need for caution in the use of inferential statistics when selecting degs.
while numerous more advanced statistical modeling techniques have been proposed and compared for selecting degs  <cit> , the primary objectives here are to explain that the primary reason for microarray reproducibility concerns is failure to include an fc criterion during gene selection, and to recommend a simple and straightforward approach concurrently satisfying statistical and reproducibility requirements.
it should be stressed that robust methods are needed to meet stringent clinical requirements for reproducibility, sensitivity and specificity of microarray applications in, for example, clinical diagnostics and prognostics.
RESULTS
the pog for a number of gene selection scenarios employing p and/or fc are compared and a numerical example  is provided that shows how the simple t-test, when sample size is small, results in selection of different genes purely by chance.
while the data generate from the maqc samples a and b lack biological variability, the results are supported by the toxicogenomic data of guo et al.
<cit>  while p could be computed from many different statistical methods, for simplicity and consistency, throughout this article p is calculated with the two-tailed t-test that is widely employed in microarray data analysis.
inter-site concordance for the same platform
the pog is higher when the analyses are limited to the genes commonly detected  by both test sites under comparison .
in addition to a slight increase , the inter-site pog lines after noise filtering are more stable than those before noise filtering, particularly for abi, ag <dig>  and geh.
furthermore, differences between the three ilm test sites are further decreased after noise filtering, as seen from the convergence of the pog for s1–s <dig>  s1–s <dig>  and s2–s <dig> comparisons.
importantly, noise filtering does not change either the trend or magnitude of the higher pog graphs for fc-ranking compared with p-ranking.
inter-site concordance for different fc- and p-ranking criteria were also calculated for other maqc sample pairs having much smaller differences than for sample a versus sample b, and correspondingly lower fc.
in general, pog is much lower for other sample pairs regardless of ranking method and ranking order varies more greatly, though fc-ranking methods still consistently give a higher pog than p-ranking methods.
figure  <dig> gives the plots of pog for sample c versus sample d, a 3: <dig> and 1: <dig>  mixture, respectively <cit> , for all inter-site comparisons.
the substantial difference in inter-site pog shown in figures  <dig> and  <dig> is a direct result of applying different gene selection methods to the same data sets, and clearly depicts how perceptions of inter-site reproducibility can be affected for any microarray platform.
while the emphasis here is on reproducibility in terms of pog, in practice, this criterion must be balanced against other desirable characteristics of gene lists, such as specificity and sensitivity  or mean squared error , considerations that that are discussed further in later sections.
cross-platform concordance
concordance between microarray and taqman® assays
taqman® real-time pcr assays are widely used to validate microarray results  <cit> .
in the maqc project, the expression levels of  <dig> genes randomly selected from available taqman® assays have been quantified in the four maqc samples  <cit> .
nine hundred and six  of the  <dig> genes are among the " <dig> " set of genes found on all of the six genome-wide microarray platforms  <cit> .
there are four technical replicates for each sample and the degs for taqman® assays were identified using the same six gene selection procedures as those used for microarray data.
the degs calculated from the microarray data are compared with degs calculated from taqman® assay data.
with noise filtering , 80–85% concordance was observed .
consistent with inter-site and cross-platform comparisons, pogs comparing microarray with taqman® assays also show that ranking genes by fc results in markedly higher pog than ranking by p alone, especially for short gene lists.
pog results without noise filtering  are some 5% lower but the notable differences in pog between the fc- and p-ranking are unchanged.
reproducibility of fc and t-statistic: different metrics for identifying differentially expressed genes
joint fc and p rule illustrated with a volcano plot: ranking by fc, not by p
concordance using other statistical tests
numerous different statistical tests including rank tests  and shrunken t-tests  have been used for the identification of degs.
although this work is not intended to serve as a comprehensive performance survey of different statistical procedures, we set out to briefly examine a few examples due to their popularity.
figure  <dig> shows the pog results of several commonly used approaches including fc-ranking, t-test statistic, wilcoxon rank-sum test, and sam using afx site-site comparison as an example <cit> .
the pog by sam , although greatly improved over that of simple t-test statistic , approached, but did not exceed, the level of pog based on fc-ranking .
in addition, the small numbers of replicates in this study rendered many ties in the wilcoxon rank statistic, resulting in poor inter-site concordance in terms of rank-order of the degs between the two afx test sites.
similar findings  were observed using the toxicogenomics data set of guo et al.
<cit> .
gene selection in simulated datasets
the maqc data, like data from actual experiments, allows evaluation of deg list reproducibility, but not of truth.
statistics are used to estimate truth, often in terms of sensitivity and specificity, but the estimates are based on assumptions about data variance and error structure that are also unknown.
simulations where truth can be specified a priori are useful to conduct parametric evaluations of gene selection methods, and true false positives and negatives are then known.
however, results are sensitive to assumptions regarding data structure and error that for microarrays remains poorly characterized.
for the maqc-simulated data, either fc-ranking or fc-ranking combined with any of the p threshold resulted in markedly higher pog than any p-ranking method, regardless of gene list length and coefficient of variation  of replicates.
the pog is ~100%, ~95%, and ~75%, for replicate cv values of 2%, 10%, and 35% cv, respectively, decreasing to about 20–30% with an exceedingly high  cv.
in contrast, pog from p-ranking alone varies from a few percent to only ~10% when  <dig> genes are selected.
for the medium- and small-delta simulated data sets, differences start to emerge between using fc alone and fc with p cutoff.
from figure  <dig>  when variances in replicates become larger , the reproducibility is greatly enhanced using fc-ranking with a suitable p cutoff versus fc or p by themselves.
in addition, when variances are small , the reproducibility is essentially the same for fc with p or without.
what is clear is that p by itself did not produce the most reproducible deg list under any simulated condition.
although p-ranking generally resulted in very low pog, a false positive was rarely produced, even for a list size of  <dig> .
thus, the p criterion performed as expected, and identified mostly true positives.
unfortunately, the probability of selection of the same true positives with a fixed p cutoff in a replicated experiment appears small due to variation in the p statistic itself .
fc-ranking by itself resulted in a large number of false positives with a large number of genes for the medium and small-delta sets when genes with small fcs are selected as differentially expressed.
these false positives were greatly reduced to the same level as for the p-ranking alone when fc-ranking was combined with a p-cutoff.
variability of the two-sample t-statistic
in a two-sample t-test comparing the mean of sample a to the mean of sample b, the t-statistic is given as follows:
t=x¯b−x¯asp2nb+sp2na
where x¯a is the average of the log <dig> expression levels of sample a with na replicates, and x¯b is the average of the log <dig> expression levels of sample b with nb replicates, and sp <dig> = / is the pooled variance of samples a and b, and ss denotes the sum of squared errors.
the numerator of the t-statistic is the fold-change  in log <dig> scale and represents the signal level of the measurements .
the denominator represents the noise components from the measurements of samples a and b. thus, the t-statistic represents a measure of the signal-to-noise ratio.
therefore, the fc and the t-statistic  are two measures for the differences between the means of sample a and sample b. the t-statistic is intrinsically less reproducible than fc when the variance is small.
assume that the data are normally distributed, the variances of samples a and b are equal , the numbers of replicates in samples a and b are equal , and that there is a real difference in the mean values between samples a and b, d .
then the t-statistic has a non-central t-distribution with non-central parameter
δ=,
and the mean and variance of the t-statistic  are
e=12Γ)Γδ,var=νν−2+12Γ)Γ]2)δ <dig>
where v =  and is the degrees of freedom of the non-central t-distribution.
when d =  <dig> , then the t-statistic has a t-distribution with mean e =  <dig> and var = v/.
the variance of the t-statistic depends on the sample size n, the magnitude of the difference between the two means d, and the variance σ <dig>  on the other hand, the variance of the mean difference for the fc is σ <dig>  that is, the variance of the fc depends only on the sample size n and the variance σ <dig>  regardless of the magnitude of the difference d between the two sample means.
in an maqc data set, a typical sample variance for the log <dig> expression levels is approximately σ <dig> =  <dig> .
with n =  <dig>  the standard deviation of the fc  is  <dig> .
for a differentially expressed gene with a 4-fold change between  <dig> replicates of sample a and  <dig> replicates of sample b, d =  <dig> and the t-values have a non-central t-distribution with  =  <dig> degrees of freedom and δ =  <dig> .
from the equations above, the mean and the variance of the t-values are e =  <dig>  and var =  <dig> .
within two standard deviations the expected value of the t-value ranges from  <dig>   to  <dig>  , corresponding to ps from  <dig> × 10- <dig> to  <dig> × 10- <dig>  based on the student's two-sided t-test with  <dig> degrees of freedom.
in contrast, when n =  <dig> the standard deviation of the fc  is  <dig> .
the expected value of the fc ranges only from  <dig>   to  <dig>   within two standard deviations.
in this case, this gene would be selected as differentially expressed using either a fc cutoff of  <dig>  or a p cutoff of  <dig> × 10- <dig>  on the other hand, for a gene with a 2-fold change , the t-statistic has a non-central t-distribution with δ =  <dig> .
the mean and the variance of the t-statistic are e =  <dig>  and var =  <dig>  with a corresponding p of  <dig> × 10- <dig> at t =  <dig> .
using the same p cutoff,  <dig> × 10- <dig>  this gene is likely to be selected with the probability greater than  <dig> .
for the fc criterion, the expected value of the fc ranges from  <dig>   to  <dig>  .
using the same fc cutoff of  <dig> , this gene is very unlikely to be selected.
thus, the top ranked gene list based on the fc is more reproducible than the top ranked gene list based on the p. the top ranked genes selected by a p cutoff may not be reproducible between experiments although both lists may contain mostly differentially expressed genes.
reference: n. johnson and s. kotz .
continuous univariate distributions –  <dig>  houghton mifflin, boston.
discussion
a fundamental requirement in microarray experiments is that the identification of deg lists must be reproducible if the data and scientific conclusion from them are to be credible.
deg lists are normally developed by rank-ordering genes in accordance with a suitable surrogate value to represent biological relevance, such as the magnitude of the differential expression  or the measure of statistical significance  of the expression change, or both.
the results show that concurrent use of both fc-ranking and p-cutoff as criteria to identify biological relevant genes can be essential to attain reproducible deg lists across laboratories and platforms.
a decade since the microarray-generated differential gene expression results of schena et al <cit>  and lockhart et al <cit>  were published, microarray usage has become ubiquitous.
over this time, many analytical techniques for identifying degs have been introduced and used.
early studies predominantly relied on the magnitude of differential expression change in experiments done with few if any replicates, with an fc cutoff typically of two used to reduce false positives.
mutch et al <cit>  recommended using intensity-dependent fc cutoffs to reduce biased selection of genes with low expression.
gene selection using statistical significance estimates became more prevalent during the last few years as studies with replicates became possible.
incorporation of a t-statistic in gene selection was intended to compensate for the heterogeneity of variances of genes  <cit> .
haslett et al.
<cit>  employed stringent values of both fc and p to determine degs.
in recent years, there has been an increasing tendency to use p-ranking for gene selection.
kittleson et al.
<cit>  selected genes with a fc cutoff of two and a very restrictive bonferroni corrected p of  <dig>  in a quest for a short list of true positive genes.
tan et al.
<cit>  used p to rank genes.
correlation coefficient, which behaves similarly to the t-statistic, has also been widely used as a gene selection method in the identification of signature genes for classification purposes  <cit> .
new and widely employed methods have appeared in recent years and implicitly correct for the large variance in the t-statistic that results when gene variance is estimated with a small number of samples.
allison et al.
<cit>  collectively described these methods as "variance shrinkage" approaches.
they include the popular permutation-based "sam" procedure  <cit> , bayesian-based approaches  <cit>  and others  <cit> .
qin et al.
<cit>  compared several variance shrinkage methods with a simple t-statistic and fc for spike-in gene identification on a two-color platform, concluding that all methods except p performed well.
all these methods have the effect of reducing a gene's variance to be between the average for the samples, and the average over the arrays.
in some cases, however, the use of fc for gene selection was criticized and entirely abandoned.
for example, callow et al.
<cit> , using p alone for identifying degs, concluded that p alone eliminated the need for filtering low intensity spots because the t-statistic is uniformly distributed across the entire intensity range.
reliance on p alone to represent a gene's fc and variability in gene selection has become commonplace.
norris and kahn  <cit>  describe how false discovery rate  has become so widely used as to constitute a standard to which microarray analyses are held.
however, fdr usually employs a shrunken t-statistic and genes are ranked and selected similar to p .
prior to maqc, irizarry et al.
<cit>  compared data from five laboratories and three platforms using the cat plots that are essentially the same as the pog graphs used in our study.
lists of less than  <dig> genes derived from fc-ranking showed  <dig> to 80% intra-site, inter-site, and inter-platform concordance.
interestingly, important disagreements were attributable to a small number of genes with large fc that they posit resulted from a laboratory effect due to inexperienced technicians and sequence-specific effects where some genes are not correctly measured.
exactly how to best employ fc with p to identify genes is a function of both the nature of the data, and the inevitable tradeoff between sensitivity and specificity that is familiar across research, clinical screening and diagnostics, and even drug discovery.
but how the tradeoff is made depends on the application.
fewer false negatives at the cost of more false positives may be desirable when the application is identifying a few hundred genes for further study, and fc-ranking with a non-stringent p value cutoff from a simple t-test could be used to eliminate some noise.
the gene list can be further evaluated in terms of gene function and biological pathway data, as illustrated in guo et al.
<cit>  for toxicogenomic data.
even for relatively short gene lists, fc-ranking together with a non-stringent p cutoff should result in reproducible lists.
in addition, deg lists identified by the ranking of fc is much less susceptible to the impact of normalization methods.
in fact, global scaling methods  do not change the relative ranking of genes based on fc; they do, however, impact gene ranking by p-value  <cit> .
the tradeoffs between reproducibility, sensitivity, and specificity become pronounced when genes are selected by p alone without consideration of fc, especially when a stringent p cutoff is used to reduce false positives.
when sample numbers are small, any gene's t-statistic can change considerably in repeated studies within or across laboratories or across platforms.
each study can select different significant genes, purely by chance.
it is entirely possible that separately determined lists will have a small proportion of common genes even while each list comprises mostly true positives.
this apparent lack of reproducibility of the gene lists is an expected outcome of statistical variation in the t-statistic for small numbers of sample replicates.
in other words, each study fails to produce some, but not all, of the correct results.
the side box provides a numerical example of how gene list discordance can result from variation in the t-statistic across studies.
decreasing the p cutoff will increase the proportion of true positives, but also diminish the number of selected genes, diminish genes common across lists, and increase false negatives.
importantly, selecting genes based on a small p cutoff derived from a simple t-test without consideration of fc renders the gene list non-reproducible.
additional insight is gained by viewing gene selection from the perspective of the biologist ultimately responsible for interpreting microarray results.
statistically speaking, a microarray experiment tests  <dig>  or more null hypotheses where essentially all genes have non-zero differential expression.
statistical tests attempt to account for an unknowable error structure, in order to eliminate the genes with low probability of biological relevance.
to the biologist, however, the variance of a gene with a large fc in one microarray study may be irrelevant if a similar experiment again finds the gene to have a large fc; the second experiment would probably be considered a validating reproduction.
this conclusion would be reasonable since the gene's p depends on a poor estimate of variance across few samples, whereas a repeated fc measurement is tangible reproducibility which tends to increase demonstrably with increasing fc.
the biological interpreter can also consider knowledge of gene function and biological pathways before finally assigning biological relevance, and will be well aware that either p or fc is only another indicator regarding biological significance.
this study shows that genes with smaller expression fold changes generated from one platform or laboratory are, in general, less reproducible in another laboratory with the same or different platforms.
however, it should be noted that genes with small fold changes may be biologically important  <cit> .
when a fixed fc cutoff is chosen, sensitivity could be sacrificed for reproducibility.
alternatively, when a high p cutoff  is used, specificity could be sacrificed for reproducibility.
ultimately, the acceptable trade-off is based on the specific question being asked or the need being addressed.
when searching for a few reliable biomarkers, high fc and low p cutoffs can be used to produce a highly specific and reproducible gene list.
when identifying the components of genetic networks involved in biological processes, a lower fc and higher p cutoff can be used to identify larger, more sensitive but less specific, gene lists.
in this case, additional biological information about putative gene functions can be incorporated to identify reliable gene lists that are specific to the biological process of interest.
truly differentially expressed genes should be more likely identified as differentially expressed by different platforms and from different laboratories than those genes with no differential expression between sample groups.
in the microarray field, we usually do not have the luxury of knowing the "truth" in a given study.
therefore, it is not surprising that most microarray studies and data analysis protocols have not been adequately evaluated against the "truth".
a reasonable surrogate of such "truth" could be the consensus of results from different microarray platforms, from different laboratories using the same platform, or from independent methods such as taqman® assays, as we have extensively explored in this study.
the fundamental scientific requirement of reproducibility is a critical dimension to consider along with balancing specificity and sensitivity when defining a gene list.
irreproducibility would render microarray technology generally, and any research result, specifically, vulnerable to criticism.
new methods for the identification of degs continue to appear in the scientific literature.
these methods are typically promoted in terms of improved sensitivity  while retaining nominal rates of specificity.
however, reproducibility is seldom emphasized.
CONCLUSIONS
the results show that selecting degs based solely on p from a simple t-test most often predestines a poor concordance in deg lists, particularly for small numbers of genes.
in contrast, using fc-ranking in conjunction with a non-stringent p-cutoff results in more concordant gene lists concomitant with needed reproducibility, even for fairly small numbers of genes.
moreover, enhanced reproducibility holds for inter-site, cross-platform, and between microarray and taqman® assay comparisons, and is independent of platforms, sample pairs, and normalization methods.
the results should increase confidence in the reproducibility of data produced by microarray technology and should also expand awareness that gene lists identified solely based on p will tend to be discordant.
this work demonstrates the need for a shift from the common practice of selecting differentially expressed genes solely on the ranking of a statistical significance measure  to an approach that emphasizes fold-change, a quantity actually measured by microarray technology.
conclusions and recommendations
<dig>  a fundamental step of microarray studies is the identification of a small subset of degs from among tens of thousands of genes probed on the microarray.
deg lists must be concordant to satisfy the scientific requirement of reproducibility, and must also be specific and sensitive for scientific relevance.
a baseline practice is needed for properly assessing reproducibility/concordance alongside specificity and sensitivity.
<dig>  reports of deg list instability in the literature are often a direct consequence of comparing deg lists derived from a simple t-statistic when the sample size is small and variability in variance estimation is large.
therefore, the practice of using p alone for gene selection should be discouraged.
<dig>  a deg list should be chosen in a manner that concurrently satisfies scientific objectives in terms of inherent tradeoffs between reproducibility, specificity, and sensitivity.
<dig>  using fc and p together balances reproducibility, specificity, and sensitivity.
control of specificity and sensitivity can be accomplished with a p criterion, while reproducibility is enhanced with an fc criterion.
sensitivity can also be improved by better platforms with greater dynamic range and lower variability or by increased sample sizes.
<dig>  fc-ranking should be used in combination with a non-stringent p threshold to select a deg list that is reproducible, specific, and sensitive, and a joint rule is recommended as a baseline practice.

 a fundamental step in most microarray experiments is determining one or more short lists of differentially expressed genes that distinguish biological conditions, such as disease from health. challenges regarding the reliability of microarray results have largely been founded on the inability of researchers to replicate deg lists across highly similar experiments. for example, tan et al. found only four common degs using an identical set of rna samples across three popular commercial platforms. independent studies by the groups of ramalho-santos and ivanova of stem cell-specific genes using the same affymetrix platform and similar study design found a disappointing six common degs among about identified in each study. a comparative neurotoxicological study by miller et al. using the same set of rna samples found only common degs among and respectively, from affymetrix and codelink platforms. all these studies ranked genes by p-value from simple t-tests, used a p threshold to identify deg lists, and applied the concept of the percentage of overlapping genes, or the venn diagram, between deg lists as the measure of reproducibility. criticism of and concerns about microarrays continue to appear in some of the most prestigious scientific journals, leading to a growing negative perception regarding microarray reproducibility, and hence reliability. however, in reanalyzing the data set of tan et al., shi et al. found that cross-platform concordance was markedly improved when either simple fold change or significance analysis of microarrays methods were used to rank order genes before determining deg lists. the awareness that microarray reproducibility is sensitive to how degs are identified was, in fact, a major motivator for the microarray quality control project. several plausible explanations and solutions have been proposed to interpret and address the apparent lack of reproducibility and stability of deg lists from microarray studies. larger sample sizes; novel, microarray-specific statistical methods; more accurate array annotation information by mapping probe sequences across platforms; eliminating absent call genes from data analysis; improving probe design to minimize cross-hybridization; standardizing manufacturing processes; and improving data quality by fully standardizing sample preparation and hybridization procedures are among the suggestions for improvement. the maqc study was specifically designed to address these previously identified sources of variability in deg lists. two distinct rna samples, stratagene universal human reference rna and ambion human brain reference rna, with thousands of differentially expressed genes, were prepared in sufficient quantities and distributed to three different laboratories for each of the five different commercial whole genome microarray platforms participating in the study. for each platform, each sample was analyzed using five technical replicates with standardized procedures for sample processing, hybridization, scanning, data acquisition, data preprocessing, and data normalization at each site. the probe sequence information was used to generate a stringent mapping of genes across the different platforms and genes were further analyzed with taqman® assays using the same rna samples. in addition to assessing the technical performance of different microarray platforms, the maqc study also discussed the idea of using fold-change ranking along with a non-stringent p-value cutoff for selecting degs. however, a lot of detailed results have not been formally published to support the idea. the maqc project, while positively received by the community, also stimulated criticism from the statistical community about appropriate ways of identifying degs. to help the microarray community better understand the issue at debate and move forward, in this study, we conducted a careful analysis of these maqc data sets, along with numerical simulations and mathematical arguments. we demonstrate that the reported lack of reproducibility of deg lists can be attributed in large part to identifying degs from simple t-tests without consideration of fc. the finding holds for intra-laboratory, inter-laboratory, and cross-platform comparisons independent of sample pairs and normalization methods, and is increasingly apparent with decreasing number of genes selected. as a basic procedure for improving reproducibility while balancing specificity and sensitivity, choosing genes using a combination of fc-ranking and p threshold was investigated. this joint criterion results in deg lists with much higher pog, commensurate with better reproducibility, than lists generated by t-test p alone, even when selecting a relatively small numbers of genes. an fc criterion explicitly incorporates the measured quantity to enhance reproducibility, whereas a p criterion incorporates control of sensitivity and specificity. the results increase our confidence in the reproducibility of microarray studies while supporting a need for caution in the use of inferential statistics when selecting degs. while numerous more advanced statistical modeling techniques have been proposed and compared for selecting degs, the primary objectives here are to explain that the primary reason for microarray reproducibility concerns is failure to include an fc criterion during gene selection, and to recommend a simple and straightforward approach concurrently satisfying statistical and reproducibility requirements. it should be stressed that robust methods are needed to meet stringent clinical requirements for reproducibility, sensitivity and specificity of microarray applications in, for example, clinical diagnostics and prognostics.  the pog for a number of gene selection scenarios employing p and/or fc are compared and a numerical example is provided that shows how the simple t-test, when sample size is small, results in selection of different genes purely by chance. while the data generate from the maqc samples a and b lack biological variability, the results are supported by the toxicogenomic data of guo et al. while p could be computed from many different statistical methods, for simplicity and consistency, throughout this article p is calculated with the two-tailed t-test that is widely employed in microarray data analysis. inter-site concordance for the same platform the pog is higher when the analyses are limited to the genes commonly detected by both test sites under comparison. in addition to a slight increase, the inter-site pog lines after noise filtering are more stable than those before noise filtering, particularly for abi, ag and geh. furthermore, differences between the three ilm test sites are further decreased after noise filtering, as seen from the convergence of the pog for s1–s s1–s and s2–s comparisons. importantly, noise filtering does not change either the trend or magnitude of the higher pog graphs for fc-ranking compared with p-ranking. inter-site concordance for different fc- and p-ranking criteria were also calculated for other maqc sample pairs having much smaller differences than for sample a versus sample b, and correspondingly lower fc. in general, pog is much lower for other sample pairs regardless of ranking method and ranking order varies more greatly, though fc-ranking methods still consistently give a higher pog than p-ranking methods. figure gives the plots of pog for sample c versus sample d, a 3: and 1: mixture, respectively, for all inter-site comparisons. the substantial difference in inter-site pog shown in figures and is a direct result of applying different gene selection methods to the same data sets, and clearly depicts how perceptions of inter-site reproducibility can be affected for any microarray platform. while the emphasis here is on reproducibility in terms of pog, in practice, this criterion must be balanced against other desirable characteristics of gene lists, such as specificity and sensitivity or mean squared error, considerations that that are discussed further in later sections. cross-platform concordance concordance between microarray and taqman® assays taqman® real-time pcr assays are widely used to validate microarray results. in the maqc project, the expression levels of genes randomly selected from available taqman® assays have been quantified in the four maqc samples. nine hundred and six of the genes are among the " " set of genes found on all of the six genome-wide microarray platforms. there are four technical replicates for each sample and the degs for taqman® assays were identified using the same six gene selection procedures as those used for microarray data. the degs calculated from the microarray data are compared with degs calculated from taqman® assay data. with noise filtering, 80–85% concordance was observed. consistent with inter-site and cross-platform comparisons, pogs comparing microarray with taqman® assays also show that ranking genes by fc results in markedly higher pog than ranking by p alone, especially for short gene lists. pog results without noise filtering are some 5% lower but the notable differences in pog between the fc- and p-ranking are unchanged. reproducibility of fc and t-statistic: different metrics for identifying differentially expressed genes joint fc and p rule illustrated with a volcano plot: ranking by fc, not by p concordance using other statistical tests numerous different statistical tests including rank tests and shrunken t-tests have been used for the identification of degs. although this work is not intended to serve as a comprehensive performance survey of different statistical procedures, we set out to briefly examine a few examples due to their popularity. figure shows the pog results of several commonly used approaches including fc-ranking, t-test statistic, wilcoxon rank-sum test, and sam using afx site-site comparison as an example. the pog by sam, although greatly improved over that of simple t-test statistic, approached, but did not exceed, the level of pog based on fc-ranking. in addition, the small numbers of replicates in this study rendered many ties in the wilcoxon rank statistic, resulting in poor inter-site concordance in terms of rank-order of the degs between the two afx test sites. similar findings were observed using the toxicogenomics data set of guo et al.. gene selection in simulated datasets the maqc data, like data from actual experiments, allows evaluation of deg list reproducibility, but not of truth. statistics are used to estimate truth, often in terms of sensitivity and specificity, but the estimates are based on assumptions about data variance and error structure that are also unknown. simulations where truth can be specified a priori are useful to conduct parametric evaluations of gene selection methods, and true false positives and negatives are then known. however, results are sensitive to assumptions regarding data structure and error that for microarrays remains poorly characterized. for the maqc-simulated data, either fc-ranking or fc-ranking combined with any of the p threshold resulted in markedly higher pog than any p-ranking method, regardless of gene list length and coefficient of variation of replicates. the pog is ~100%, ~95%, and ~75%, for replicate cv values of 2%, 10%, and 35% cv, respectively, decreasing to about 20–30% with an exceedingly high cv. in contrast, pog from p-ranking alone varies from a few percent to only ~10% when genes are selected. for the medium- and small-delta simulated data sets, differences start to emerge between using fc alone and fc with p cutoff. from figure when variances in replicates become larger, the reproducibility is greatly enhanced using fc-ranking with a suitable p cutoff versus fc or p by themselves. in addition, when variances are small, the reproducibility is essentially the same for fc with p or without. what is clear is that p by itself did not produce the most reproducible deg list under any simulated condition. although p-ranking generally resulted in very low pog, a false positive was rarely produced, even for a list size of. thus, the p criterion performed as expected, and identified mostly true positives. unfortunately, the probability of selection of the same true positives with a fixed p cutoff in a replicated experiment appears small due to variation in the p statistic itself. fc-ranking by itself resulted in a large number of false positives with a large number of genes for the medium and small-delta sets when genes with small fcs are selected as differentially expressed. these false positives were greatly reduced to the same level as for the p-ranking alone when fc-ranking was combined with a p-cutoff. variability of the two-sample t-statistic in a two-sample t-test comparing the mean of sample a to the mean of sample b, the t-statistic is given as follows: t=x¯b−x¯asp2nb+sp2na where x¯a is the average of the log expression levels of sample a with na replicates, and x¯b is the average of the log expression levels of sample b with nb replicates, and sp = / is the pooled variance of samples a and b, and ss denotes the sum of squared errors. the numerator of the t-statistic is the fold-change in log scale and represents the signal level of the measurements. the denominator represents the noise components from the measurements of samples a and b. thus, the t-statistic represents a measure of the signal-to-noise ratio. therefore, the fc and the t-statistic are two measures for the differences between the means of sample a and sample b. the t-statistic is intrinsically less reproducible than fc when the variance is small. assume that the data are normally distributed, the variances of samples a and b are equal, the numbers of replicates in samples a and b are equal, and that there is a real difference in the mean values between samples a and b, d. then the t-statistic has a non-central t-distribution with non-central parameter δ=, and the mean and variance of the t-statistic are e=12Γ)Γδ,var=νν−2+12Γ)Γ]2)δ where v = and is the degrees of freedom of the non-central t-distribution. when d =, then the t-statistic has a t-distribution with mean e = and var = v/. the variance of the t-statistic depends on the sample size n, the magnitude of the difference between the two means d, and the variance σ on the other hand, the variance of the mean difference for the fc is σ that is, the variance of the fc depends only on the sample size n and the variance σ regardless of the magnitude of the difference d between the two sample means. in an maqc data set, a typical sample variance for the log expression levels is approximately σ =. with n = the standard deviation of the fc is. for a differentially expressed gene with a 4-fold change between replicates of sample a and replicates of sample b, d = and the t-values have a non-central t-distribution with = degrees of freedom and δ =. from the equations above, the mean and the variance of the t-values are e = and var =. within two standard deviations the expected value of the t-value ranges from to, corresponding to ps from × 10- to × 10- based on the student's two-sided t-test with degrees of freedom. in contrast, when n = the standard deviation of the fc is. the expected value of the fc ranges only from to within two standard deviations. in this case, this gene would be selected as differentially expressed using either a fc cutoff of or a p cutoff of × 10- on the other hand, for a gene with a 2-fold change, the t-statistic has a non-central t-distribution with δ =. the mean and the variance of the t-statistic are e = and var = with a corresponding p of × 10- at t =. using the same p cutoff, × 10- this gene is likely to be selected with the probability greater than. for the fc criterion, the expected value of the fc ranges from to. using the same fc cutoff of, this gene is very unlikely to be selected. thus, the top ranked gene list based on the fc is more reproducible than the top ranked gene list based on the p. the top ranked genes selected by a p cutoff may not be reproducible between experiments although both lists may contain mostly differentially expressed genes. reference: n. johnson and s. kotz. continuous univariate distributions – houghton mifflin, boston. discussion a fundamental requirement in microarray experiments is that the identification of deg lists must be reproducible if the data and scientific conclusion from them are to be credible. deg lists are normally developed by rank-ordering genes in accordance with a suitable surrogate value to represent biological relevance, such as the magnitude of the differential expression or the measure of statistical significance of the expression change, or both. the results show that concurrent use of both fc-ranking and p-cutoff as criteria to identify biological relevant genes can be essential to attain reproducible deg lists across laboratories and platforms. a decade since the microarray-generated differential gene expression results of schena et al and lockhart et al were published, microarray usage has become ubiquitous. over this time, many analytical techniques for identifying degs have been introduced and used. early studies predominantly relied on the magnitude of differential expression change in experiments done with few if any replicates, with an fc cutoff typically of two used to reduce false positives. mutch et al recommended using intensity-dependent fc cutoffs to reduce biased selection of genes with low expression. gene selection using statistical significance estimates became more prevalent during the last few years as studies with replicates became possible. incorporation of a t-statistic in gene selection was intended to compensate for the heterogeneity of variances of genes. haslett et al. employed stringent values of both fc and p to determine degs. in recent years, there has been an increasing tendency to use p-ranking for gene selection. kittleson et al. selected genes with a fc cutoff of two and a very restrictive bonferroni corrected p of in a quest for a short list of true positive genes. tan et al. used p to rank genes. correlation coefficient, which behaves similarly to the t-statistic, has also been widely used as a gene selection method in the identification of signature genes for classification purposes. new and widely employed methods have appeared in recent years and implicitly correct for the large variance in the t-statistic that results when gene variance is estimated with a small number of samples. allison et al. collectively described these methods as "variance shrinkage" approaches. they include the popular permutation-based "sam" procedure, bayesian-based approaches and others. qin et al. compared several variance shrinkage methods with a simple t-statistic and fc for spike-in gene identification on a two-color platform, concluding that all methods except p performed well. all these methods have the effect of reducing a gene's variance to be between the average for the samples, and the average over the arrays. in some cases, however, the use of fc for gene selection was criticized and entirely abandoned. for example, callow et al., using p alone for identifying degs, concluded that p alone eliminated the need for filtering low intensity spots because the t-statistic is uniformly distributed across the entire intensity range. reliance on p alone to represent a gene's fc and variability in gene selection has become commonplace. norris and kahn describe how false discovery rate has become so widely used as to constitute a standard to which microarray analyses are held. however, fdr usually employs a shrunken t-statistic and genes are ranked and selected similar to p. prior to maqc, irizarry et al. compared data from five laboratories and three platforms using the cat plots that are essentially the same as the pog graphs used in our study. lists of less than genes derived from fc-ranking showed to 80% intra-site, inter-site, and inter-platform concordance. interestingly, important disagreements were attributable to a small number of genes with large fc that they posit resulted from a laboratory effect due to inexperienced technicians and sequence-specific effects where some genes are not correctly measured. exactly how to best employ fc with p to identify genes is a function of both the nature of the data, and the inevitable tradeoff between sensitivity and specificity that is familiar across research, clinical screening and diagnostics, and even drug discovery. but how the tradeoff is made depends on the application. fewer false negatives at the cost of more false positives may be desirable when the application is identifying a few hundred genes for further study, and fc-ranking with a non-stringent p value cutoff from a simple t-test could be used to eliminate some noise. the gene list can be further evaluated in terms of gene function and biological pathway data, as illustrated in guo et al. for toxicogenomic data. even for relatively short gene lists, fc-ranking together with a non-stringent p cutoff should result in reproducible lists. in addition, deg lists identified by the ranking of fc is much less susceptible to the impact of normalization methods. in fact, global scaling methods do not change the relative ranking of genes based on fc; they do, however, impact gene ranking by p-value. the tradeoffs between reproducibility, sensitivity, and specificity become pronounced when genes are selected by p alone without consideration of fc, especially when a stringent p cutoff is used to reduce false positives. when sample numbers are small, any gene's t-statistic can change considerably in repeated studies within or across laboratories or across platforms. each study can select different significant genes, purely by chance. it is entirely possible that separately determined lists will have a small proportion of common genes even while each list comprises mostly true positives. this apparent lack of reproducibility of the gene lists is an expected outcome of statistical variation in the t-statistic for small numbers of sample replicates. in other words, each study fails to produce some, but not all, of the correct results. the side box provides a numerical example of how gene list discordance can result from variation in the t-statistic across studies. decreasing the p cutoff will increase the proportion of true positives, but also diminish the number of selected genes, diminish genes common across lists, and increase false negatives. importantly, selecting genes based on a small p cutoff derived from a simple t-test without consideration of fc renders the gene list non-reproducible. additional insight is gained by viewing gene selection from the perspective of the biologist ultimately responsible for interpreting microarray results. statistically speaking, a microarray experiment tests or more null hypotheses where essentially all genes have non-zero differential expression. statistical tests attempt to account for an unknowable error structure, in order to eliminate the genes with low probability of biological relevance. to the biologist, however, the variance of a gene with a large fc in one microarray study may be irrelevant if a similar experiment again finds the gene to have a large fc; the second experiment would probably be considered a validating reproduction. this conclusion would be reasonable since the gene's p depends on a poor estimate of variance across few samples, whereas a repeated fc measurement is tangible reproducibility which tends to increase demonstrably with increasing fc. the biological interpreter can also consider knowledge of gene function and biological pathways before finally assigning biological relevance, and will be well aware that either p or fc is only another indicator regarding biological significance. this study shows that genes with smaller expression fold changes generated from one platform or laboratory are, in general, less reproducible in another laboratory with the same or different platforms. however, it should be noted that genes with small fold changes may be biologically important. when a fixed fc cutoff is chosen, sensitivity could be sacrificed for reproducibility. alternatively, when a high p cutoff is used, specificity could be sacrificed for reproducibility. ultimately, the acceptable trade-off is based on the specific question being asked or the need being addressed. when searching for a few reliable biomarkers, high fc and low p cutoffs can be used to produce a highly specific and reproducible gene list. when identifying the components of genetic networks involved in biological processes, a lower fc and higher p cutoff can be used to identify larger, more sensitive but less specific, gene lists. in this case, additional biological information about putative gene functions can be incorporated to identify reliable gene lists that are specific to the biological process of interest. truly differentially expressed genes should be more likely identified as differentially expressed by different platforms and from different laboratories than those genes with no differential expression between sample groups. in the microarray field, we usually do not have the luxury of knowing the "truth" in a given study. therefore, it is not surprising that most microarray studies and data analysis protocols have not been adequately evaluated against the "truth". a reasonable surrogate of such "truth" could be the consensus of results from different microarray platforms, from different laboratories using the same platform, or from independent methods such as taqman® assays, as we have extensively explored in this study. the fundamental scientific requirement of reproducibility is a critical dimension to consider along with balancing specificity and sensitivity when defining a gene list. irreproducibility would render microarray technology generally, and any research result, specifically, vulnerable to criticism. new methods for the identification of degs continue to appear in the scientific literature. these methods are typically promoted in terms of improved sensitivity while retaining nominal rates of specificity. however, reproducibility is seldom emphasized.  the results show that selecting degs based solely on p from a simple t-test most often predestines a poor concordance in deg lists, particularly for small numbers of genes. in contrast, using fc-ranking in conjunction with a non-stringent p-cutoff results in more concordant gene lists concomitant with needed reproducibility, even for fairly small numbers of genes. moreover, enhanced reproducibility holds for inter-site, cross-platform, and between microarray and taqman® assay comparisons, and is independent of platforms, sample pairs, and normalization methods. the results should increase confidence in the reproducibility of data produced by microarray technology and should also expand awareness that gene lists identified solely based on p will tend to be discordant. this work demonstrates the need for a shift from the common practice of selecting differentially expressed genes solely on the ranking of a statistical significance measure to an approach that emphasizes fold-change, a quantity actually measured by microarray technology. conclusions and recommendations a fundamental step of microarray studies is the identification of a small subset of degs from among tens of thousands of genes probed on the microarray. deg lists must be concordant to satisfy the scientific requirement of reproducibility, and must also be specific and sensitive for scientific relevance. a baseline practice is needed for properly assessing reproducibility/concordance alongside specificity and sensitivity. reports of deg list instability in the literature are often a direct consequence of comparing deg lists derived from a simple t-statistic when the sample size is small and variability in variance estimation is large. therefore, the practice of using p alone for gene selection should be discouraged. a deg list should be chosen in a manner that concurrently satisfies scientific objectives in terms of inherent tradeoffs between reproducibility, specificity, and sensitivity. using fc and p together balances reproducibility, specificity, and sensitivity. control of specificity and sensitivity can be accomplished with a p criterion, while reproducibility is enhanced with an fc criterion. sensitivity can also be improved by better platforms with greater dynamic range and lower variability or by increased sample sizes. fc-ranking should be used in combination with a non-stringent p threshold to select a deg list that is reproducible, specific, and sensitive, and a joint rule is recommended as a baseline practice. 
some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes are not reproducible in similar experiments. we recommend the use of fc-ranking plus a non-stringent p cutoff as a straightforward and baseline practice in order to generate more reproducible deg lists. meanwhile, new statistical methods for identifying degs continue to appear in the scientific literature. the resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable deg lists. we present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from taqman – the widely regarded "standard" gene expression platform. our results demonstrate that previously reported discordance between deg lists could simply result from ranking and selecting degs solely by statistical significance derived from widely used simple t-tests; when fold change is used as the ranking criterion with a non-stringent p-value cutoff filtering, the deg lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and the instability of short deg lists solely based on p-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the p-value threshold, the less reproducible the deg list is. these observations are also consistent with results from extensive simulation calculations. our results provide practical guidance to choose the appropriate fc and p-value cutoffs when selecting a given number of degs. using the data sets generated by the microarray quality control project, we investigated the impact on the reproducibility of deg lists of a few widely used gene selection procedures. reproducibility is a fundamental requirement in scientific experiments.
 reproducibility is a fundamental requirement in scientific experiments. some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes are not reproducible in similar experiments. meanwhile, new statistical methods for identifying degs continue to appear in the scientific literature. the resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable deg lists.  using the data sets generated by the microarray quality control project, we investigated the impact on the reproducibility of deg lists of a few widely used gene selection procedures. we present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from taqman – the widely regarded "standard" gene expression platform. our results demonstrate that previously reported discordance between deg lists could simply result from ranking and selecting degs solely by statistical significance derived from widely used simple t-tests; when fold change is used as the ranking criterion with a non-stringent p-value cutoff filtering, the deg lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and the instability of short deg lists solely based on p-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the p-value threshold, the less reproducible the deg list is. these observations are also consistent with results from extensive simulation calculations.  we recommend the use of fc-ranking plus a non-stringent p cutoff as a straightforward and baseline practice in order to generate more reproducible deg lists. specifically, the p-value cutoff should not be stringent and fc should be as large as possible. our results provide practical guidance to choose the appropriate fc and p-value cutoffs when selecting a given number of degs. the fc criterion enhances reproducibility, whereas the p criterion balances sensitivity and specificity. 23– february fifth annual mcbios conference. systems biology: bridging the omics oklahoma city, ok, usa 