BACKGROUND
continual invention of new microarray data analysis algorithms for the identification of which genes express differently across two groups calls for objectively comparing the performance of existing algorithms  <cit> .
while there have been thorough empirical comparisons between supervised learning methods of classifying microarrays  <cit> , comparisons between methods of detecting differential gene expression have tended to depend heavily on either theory or simulation and thus on strong underlying assumptions  <cit> .
more empirical alternatives include the use of biologically-derived prior information regarding which experiments are more likely to contain differentially expressed genes  <cit>  and the use of spike-in data sets  <cit> .
the latter can represent equivalently expressed genes better than simulations, but the artificial spike-in levels do not necessarily correspond to levels of differentially expression across conditions of biological interest.
an early report of the microarray quality control  project  <cit>  may mark a turning point in the methodology of comparing of statistical methods designed to identify differential gene expression on the basis of microarray observations.
the critical advantage of this "concordance"  method is its validation entirely on the basis of the microarray data without resorting to spiking in known quantities of analytes or to prior information from other types of experiments; it is thus applicable to any microarray data set with sufficient replication.
validation by non-microarray information such as rt-pcr measurements of gene expression or public pathway/functional information on genes does have great value in overcoming shortcomings in microarray platforms  <cit> .
for that very reason, however, such validation has markedly less value in judging the performance of statistical methods of detecting differential gene expression.
for example, the inability of rt-pcr to validate a microarray prediction of differential gene expression might indicate a problem with the statistical assumptions used to make the prediction, but it may instead refect a problem with cross hybridization due to the microarray platform.
participants in the maqc project avoided such confounding between microarray platform effects and statistical method effects by quantifying the degree of overlap between gene lists produced by an algorithm on the basis of two independent data sets  <cit> .
although a significant step forward, this way of comparing algorithms, like that of  <cit> , requires examining gene lists of given sizes, which is why chen et al.
<cit>  consider the concordance to be too unstable for use as an algorithm performance criterion.
without depending on arbitrarily selected numbers of genes, the platform-algorithm confounding may be overcome by cross validation, which instead uses a test set of microarrays to validate predictions made on the basis of a separate training set of microarrays, while maintaining the empirical nature of the concordance method.
like concordance, cross validation does not incorporate knowledge that only a small portion of genes are differentially expressed.
encoding this information when appropriate into a hierarchical model enables a more reliable assessment of the performance of differential expression detection algorithms if only a few biological replicates are available.
these methods are explained in section  <dig> and illustrated in section 3; implications are discussed in section  <dig>
methods
if a gene is known to be differentially expressed at a certain level on average, then that level would predict future measurements of gene expression better than would making such predictions on the assumption that there is on average no differential expression.
likewise, if a gene is known to be equivalently expressed, then using an expression level of  <dig> or an expression ratio of  <dig> would predict future measurements better than making such predictions on the assumption that there is some differential expression.
thus, a method of selecting genes as differentially expressed may be judged by estimating its ability to predict future measurements of gene expression.
this estimation may be carried out by a process of cross validation: the microarrays are divided between a training set used to determine which genes the method considers differentially expressed and a test set used to estimate how well such results would agree with future measurements.
the strategy of assessing gene selection algorithms by estimated prediction error may be more precisely specified in mathematical notation.
let xi,j denote the logarithm of the measured expression intensity or ratio of intensities of the ith of m genes in the jth of n biological replicates of the control or reference group; each value of xi,j may represent an average over technically replicated microarrays; xi = ; x = t. likewise,  denotes the logarithm of the measured expression intensity or ratio of intensities of the ith gene in the jth of n' biological replicates of the treatment or perturbation group; .
the observations xi,j and  are realizations of the random variables xi and , respectively.
the ith gene is called equivalently expressed if e =  <dig> or differentially expressed if e ≠  <dig>  in hypothesis testing parlance, the null hypothesis associated with the ith gene is hi: e =  <dig>
gene selection algorithms
a gene selection algorithm α returns πα , an estimate of the posterior probability that the ith gene is equivalently expressed; it follows that  <dig> - πα  is the algorithm's probability that the gene is differentially expressed across the perturbation and reference groups.
many algorithms  <cit>  give πα  directly as a local false discovery rate estimate  <cit> , whereas traditional false discovery rate estimates and other non-bayesian algorithms in effect assign πα  a value of either  <dig> or  <dig>  depending on whether or not a gene is considered differentially expressed at a given threshold.
for example, the practice of considering a gene differentially expressed if exp , its estimated fold change, is at least φ may be expressed as
with φ >  <dig>  , and .
the discontinuity can be removed by introducing smooth functions on an ad hoc basis; here we use
as an example of such a smooth function.
the trivial algorithms
which completely ignore the data, will serve as informative points of reference.
some of the empirical bayes algorithms implemented in two r packages  <cit>  are considered here  <cit> .
from calculations based on a moderated  t-statistic that are performed by the r package limma  <cit> , one may readily obtain pi (), a one-sided p-value of the ith null hypothesis;
p () = , p <dig> (), ..., pm ()).
given the moderated t-statistics and π , the proportion of genes expected to be equivalently expressed, limma also computes logωi ), the estimated logarithm of the posterior odds that gene i is differentially expressed rather than equivalently expressed, from which the local false discovery rate may be readily obtained as ))- <dig> .
since, for use with the log-odds, the author of the algorithm does not recommend computing π  using limma's convest function , we instead iterated the log-odds function until convergence by adapting a method  <cit>  originally proposed for another empirical bayes algorithm  <cit> :
<dig>  let π <dig>  = 90% and initialize k to  <dig>
<dig>  increment k by  <dig>
<dig>  let .
<dig>  repeat steps 2- <dig> until the absolute value of the proportion difference is sufficiently small, i.e., |πk  - πk- <dig> | < 1/ <dig>  or until the sign of the proportion difference changes, i.e.,
- πk- <dig> )  - πk- <dig> ) <  <dig>  the number of iterations performed until such convergence is denoted by k.
<dig>  let π  = πk .
based on that value of π , the estimated probability of equivalent expression is derived by solving for it in the definition of the odds of differential expression , yielding
also using standard distributions of test statistics under the null hypothesis, the r package locfdr  <cit>  maps p, a vector of single-tailed p-values for all genes, to estimates of a local false discovery rate , πlocfdr .
the use of moderated t-statistics is incorporated by
more commonly, p , a vector of standard  t-test p-values, which also assume the normality of  - xi, or p , a vector of  wilcoxon test p-values, which do not assume normality, yield local false discovery rate estimates
alternatively, the locfdr package can employ an empirical maximum-likelihood estimate of the null distribution  <cit>  for computation of the local-false-discovery-rate estimate πemp.null :
whereas the empirical bayes algorithms provide approximations to a posterior probability of a hierarchical bayesian class of models, we included comparisons to the posterior probability πbayes factor  under a non-hierarchical set of models.
the data densities under the non-hierarchical models are based on the same assumptions as those of standard linear regression: unconstrained data means under the alternative hypothesis  and, for each gene, normal iid noise and equal variance within each group in the unpaired case.
let  represent the hypothesis of differential expression .
the posterior odds of differential expression under these models are
where p  is the prior predictive density or integrated likelihood under hypothesis h. the left-hand side of equation  is the posterior odds of equivalent expression to differential expression; on the right-hand side, the first factor is the prior odds of equivalent expression to differential expression, and the second factor is known as the bayes factor.
since we take p  = p() = 1/ <dig>  our posterior odds is equal to the bayes factor; thus putting equal prior mass on each hypothesis does not share the conservatism of the above empirical bayes algorithms.
additional file  <dig> gives the analytical derivation of the resulting posterior probability, which may be expressed in terms of some additional notation.
define
if n = n' and  is paired with xi,j, or
if  and xi are independent.
then the posterior probability is given by
we also applied two "information criteria" used in model selection to estimate the posterior probability; the information criteria were applied to the same linear regression framework used in the above bayes factor computation.
in model selection terminology, each criterion selects either model hi or model   for the ith gene, but we instead averaged the estimates corresponding to the two models for each gene as follows.
we first applied the bayesian information criterion   <cit> .
up to a factor of - 1/ <dig> and a constant term, the bic approximates the logarithm of the prior predictive probability density given a statistical model and a sufficiently diffuse proper prior distribution under the given model without requiring specification of such a prior.
with a prior mass on each model considered, the bic leads to an approximation of a posterior probability that is less conservative than that of the above bayes factor.
the general formula for the bic under a model with normal errors is
where n is the number of data points and k is the number of parameters in the model.
for paired data, n = n; under hi the only parameter is the data variance, giving k =  <dig>  while under  the model includes both the data mean and data variance, giving k =  <dig>  therefore the bic for each hypothesis is
with ssrh as defined in .
for independent data, n = n+ n'; under hi the model includes a single mean log-expression level and the data variance, giving k =  <dig>  while under  the model includes two distinct mean log-expression levels  and the data variance, giving k =  <dig>  therefore the bic for each hypothesis is
with ssrh as defined in .
since we again use p  = p (), the bic approximation of the posterior odds  is equal to its approximation of bayes factors corresponding to a wide class of priors on the model parameters.
transformed from the logarithmic scale to the probability scale  <cit> , the result is an equation of the same form as ,
the second information criterion we assessed was the akaike information criterion corrected for small samples .
while - aicc/ <dig> plus a constant term is in general only an approximately unbiased estimator of the expected kullback-leibler distance between the model/hypothesis and the unknown true data generating distribution  <cit> , it is exactly unbiased for linear regression models with normal errors  <cit> , a class that includes the present non-hierarchical models.
under the name of akaike weights, it and other alc-like criteria have been used to generate predictions that take model uncertainty into account in a manner exactly analogous to bayesian model averaging  <cit> , giving rise to an equation of the same form as.
the general formula for the aicc under a model with normal errors is
the particular values of n and k for paired and independent data under  and hi are the same as those given above for the bic.
for paired data, the aicc values of the hypotheses or models are
with ssrh as defined in ; for independent data, the aicc values are
with ssrh as defined in .
transforming from the logarithmic scale yields the effective probability
where
is the ratio of akaike weights.
these algorithms were chosen as representatives of various classes of possible approaches.
whereas the fold-change-dependent algorithms represent algorithms that take no account of the data variance, the information criterion algorithms and the non-hierarchical bayesian algorithm represent algorithms that do take data variance into account but do not share information across genes.
the local fdr algorithms based on classical p-values share information across genes for the purpose of determining false discovery rates, thus accounting for multiple comparisons, but do not share information for estimating data variance.
algorithms employing the moderated t-statistic share information across genes to account for multiple comparisons and also to estimate data variance.
methods of assessing gene selection algorithms
each of the next subsections describes a different method of quantifying the performance of gene selection algorithms.
the first, cross validation, has the advantage that it is an unbiased estimator of squared prediction error  without assuming any parametric model.
the second, the computation of posterior predictive loss, takes advantage of the knowledge that gene expression is approximately lognormal and that relatively few genes will have substantial differential expression, the vast majority being equivalently expressed for all practical purposes.
the two methods will differ in results; if nearly all genes have only negligible differential expression, the latter is deemed more reliable except in the case of extensive biological replication since the former achieves low bias by admitting a high variance of performance estimates.
cross validation
algorithm α's best prediction of future values of  - xi is the posterior expected degree of expression,
the term  is the best estimator of the degree of expression conditional on definite knowledge that gene i is differentially expressed; it is multiplied by ), the posterior probability of differential expression.
the posterior expected degree of expression has been compared to a method of correcting estimates for gene selection bias  <cit> .
for a new observation of gene i, the squared prediction error is,
the squared prediction error does not directly target the question of which genes are differentially expressed; instead, it addresses the question of what the value of the next observation will be.
however, good performance of one algorithm relative to another on either of these questions implies good performance on the other, as can be seen by considering that in general the mean squared prediction error is the sum of an algorithm's squared predictive bias and the data variance.
the squared predictive bias term summarizes the ability of an algorithm to correctly distinguish differentially expressed genes from equivalently expressed genes.
it is more fexible than the 0/ <dig> loss in that it penalizes algorithms not just for being wrong, but for how wrong they are.
the data variance sets the scale for "wrongness", in that for one algorithm to appear significantly worse than another, its squared predictive bias must dominate the data variance term.
under the "all nulls false" reference algorithm, the best prediction of future values of  - xi for all genes is the maximum likelihood estimator .
other algorithms make gains over this reference by correctly assigning equivalently expressed genes, thereby avoiding the contribution of the variance of the mle to the squared prediction error.
under the "all nulls true" reference algorithm, the best prediction of future values of  - xi for all genes is  <dig>  other algorithms make gains over this reference by correctly assigning differentially expressed genes, thereby avoiding the contribution of the squared bias ]2) to the squared prediction error.
the squared prediction error criterion therefore quantifies the relative costs of false positives and false negatives in terms of the bias-variance trade-off.
to estimate the squared prediction error, we used leave-one-out cross validation,
if n = n' and  is paired with xi,j or
if  and xi are independent, where  means the jth replicate is omitted:
, and .
for example, suppose that  is paired with xi,,j and the data for gene i were  =  and xi = .
for j from  <dig> to  <dig>   = , and using the fold change shrinkage calculation of equation  <dig>   <dig> - πα ) = .
).)
the individual terms in the sum in equation  <dig> are  <dig>   <dig>  and ) <dig>  and their mean is  <dig> .
if the given data were independent instead of paired, the calculation would involve each of the  <dig> subsets obtained by leaving out one perturbation data point and one control data point.
we considered measuring error relative to always predicting that  - xi =  <dig> on a gene-wise basis using the ratio
with two measures of central tendency,
<cit>  is a fast, robust estimator of the mode that is suitable as for asymmetric distributions.
it is implemented as the hsm function in the modeest package of r.) we also considered an absolute error criterion,
this measure is relative to a base model such as the "all nulls true" model or the "all nulls false" model because we expect only the relative performances of the estimators to be meaningful.
we found that the relative error mean essentially reproduced the absolute error relative to the "all nulls true" model, and the relative error mode often evaluated estimators as not practically different from the "all nulls true" benchmark.
therefore, we show only the results for the absolute error measure.
the use of cross-validation for estimation of classification error, appropriate for the problem of categorizing samples or microarrays given known classifications for use in the training and test sets, differs from the use cross-validation for estimation of squared prediction error, appropriate for the distinct problem of determining which genes are differentially expressed without knowledge of which genes are differentially expressed for use in the training and test sets.
jeffery et al.
<cit>  used a cross-validation approach to estimate the predictive error of a variety of gene selection algorithms, but with microarray classification error rather than equations - as the performance criterion.
such classification error depends not only on the gene selection algorithm, but also on the particular classifier for which that algorithm selects features.
since our interest lies strictly in identifying differentially expressed genes, our methods instead quantify performance in terms of predicting new measurements.
we have also addressed the problem using estimation error in place of prediction error  <cit> .
posterior predictive expected squared error
the local fdr shrinkage algorithm can be used to define an estimator's posterior predictive expected squared error.
in general, the posterior predictive expected squared error is
where  and xi,new are random variables for new observations,  is algorithm α's point prediction for  - xi,new, and eposterior and varposterior are the expectation and variance with respect to the posterior distribution.
the effective posterior distribution that leads to estimators of the form  is
which has variance,
we use the local fdr estimator with t-statistics and theoretical null distribution as our gold standard model for the computation of πα ; this model will be accurate under the reasonable assumption that few genes are differentially expressed at appreciable levels.
to fully express the posterior predictive loss, we must define the posterior predictive distribution for  - xnew under both the null and alternative hypotheses for both paired and non-paired data.
conditional on each hypothesis, we use improper prior distributions for convenience.
strictly speaking, this is inconsistent with our choice of πα , an empirical bayes approximation to a posterior probability; under a full bayesian analysis, posterior probabilities of hypotheses can only be computed under proper priors for the parameters conditional on each hypothesis, as in the bayes factor algorithm of equation .
our choice of πα  enables sharing information across genes to give a sensible empirical bayes posterior probability for the hypotheses but otherwise relies on the same assumptions as our conditional prior distributions.
for paired data under the null hypotheses,  - xnew has a normal sampling distribution with zero mean and sampling variance estimated from the data.
under the usual improper prior for the sampling variance ∝ σ-2), the posterior distribution for the sampling variance is a scaled-inverse-χ <dig> distribution with degrees of freedom n and scale .
the posterior predictive density is the expectation of the sampling density with respect to the posterior distribution of the sampling variance,
where n  is the normal distribution parameterized in terms of mean and variance, and tv  is a shifted, scaled version of the t distribution with v degrees of freedom, center c, and scale factor s. , then /s is distributed as the usual tv distribution.)
for paired data under the alternative hypothesis,  - xnew has a normal sampling distribution with both mean and sampling variance estimated from the data.
it can be shown that under the usual improper joint prior for mean μ ) and the sampling variance  ∝ σ-2), the posterior predictive distribution for  - xnew is,
where , i.e., s <dig> is the usual unbiased variance estimator.
for non-paired data under the null hypothesis, if the treatment and control data are modeled as having distinct sampling variances ) then the posterior predictive distribution is
where  <dig> and σ <dig> are the sampling variance for treatment and control data respectively.
this integral is intractable because πposterior 2)has a non-standard form .
we estimated it by drawing samples from πposterior 2) using markov chain monte carlo   <cit>  and then calculating the mcmc average,
where the subscript k indicates the kth mcmc draw of parameter values  and k is the total number of draws.
in the present case, the mcmc algorithm we use is an inherently multi-chain procedure; we used  <dig> chains.
we used a burn-in of  <dig> samples per chain, followed by  <dig> samples per chain, for a total of k =  <dig> samples.
for each gene in a randomly chosen subset of genes from the complete data set, a contour plot of the posterior density was superimposed on a scatter plot of the mcmc draws of parameter values.
the scatter plots visually conformed to the contours of the posterior densities, verifying that the mcmc draws of parameter values provided a good approximation to the posterior distributions.
for non-paired data under the alternative hypothesis,  and xnew each have a normal sampling distribution with both mean and sampling variance estimated from the data.
it can be shown that under the usual improper joint prior for the individual means and sampling variances, the posterior predictive distributions for  and xnew are
and therefore
to summarize gene-wise posterior predictive expected squared error over all genes in a data set, we considered quantities analogous to the relative errors and absolute errors of equations -, with gene-wise posterior predictive expected squared errors replacing cross-validation-derived prediction errors.
again, we found that the relative error mean essentially replicated the results of the absolute error relative to the "all nulls true" benchmark; relative error mode evaluated the performance of all estimators as identical to the "all nulls true" benchmark.
therefore, we show only the results for the absolute error measure for posterior predictive expected squared error.
RESULTS
to illustrate the proposed methods of quantifying the performance of gene selection algorithms, we applied them to two example data sets, one relevant to agriculture and the other to medicine.
since this study is limited to the evaluation of high-level algorithms of detecting differentially expression, we did not consider multiple pre-processing schemes.
the agricultural data sets were processed as described in  <cit> ; the medical data sets were pre-processed according to the specifications of the chip manufacturer  <cit> .
agricultural data
dual-channel microarrays were used to measure in tomatoes the expression ratios  of m =  <dig>  genes at the breaker stage of ripening and at  <dig> and  <dig> days thereafter  <cit> .
each of the later two stages has six biological replicates , but one of the biological replicates is missing at the breaker stage of ripening .
the next subsection compares algorithms of determining which genes are differentially expressed between mutant and wild type at each point in time, whereas subsection  <dig>  uses the same data to instead compare algorithms of determining which genes are differentially expressed between one point in time and another point in time.
pairing across microarray channels
in order to determine the genes for which expected values of logarithms of mutant-to-wild-type ratios differ from  <dig>  let  be the expression level of the mutant sample with mrna hybridized to the same microarray as that of a wild type sample with expression level xi,j at  <dig>   <dig>  or  <dig> days after the breaker stage.
then  - xi,j is the logarithm of the observed ratio for the ith gene and jth microarray.
due to this dependence structure, paired  t-tests and wilcoxon signed-rank tests were used to obtain p-values, and equation  was used to estimate prediction error.
we measured absolute error relative to the local fdr using the t-statistic and the theoretical null  because this model had the best or near-best performance in seven out of the nine data sets considered in this paper.
thus, its use as the base model facilitated the plotting of multiple data sets in a single figure; this same model was used as the base model for all subsequent figure.
the estimated prediction errors for all algorithms mentioned above are displayed as figure
two independent groups
in order to determine which genes differ in mutant-to-wild-type ratios between different periods of time after the breaker stage, let  and xi,j denote the logarithms of ratios observed at two different points in time for gene i and for microarrays j' and j. since the measurement errors of observations made at one time point are independent of those made at the other time point, 2-sample t-tests and wilcoxon rank-sum tests were used to obtain p-values, and equation  was used to average estimated prediction error .
figure  <dig> shows, for each non-paired tomato data set, the total posterior predictive expected squared error ) for each estimator relative to that of the gold standard model.
biomedical data
maqc researchers  <cit>  measured gene expression responses to a rat liver treatment on four different platforms: applied biosystems, affymetrix, agilent, and ge healthcare.
each data set has six treatment biological replicates and six control biological replicates.
as in subsection  <dig> .
<dig>  observations in the treatment group are not paired with those of the control group.
the applied biosystems data set  and the two affymetrix data sets  were used to assess gene selection criteria on the basis of prediction error .
figure  <dig> shows, for each maqc data set, the total posterior predictive expected squared error ) for each estimator relative to that of the gold standard model.
discussion
fold change versus testing
fold change performs about as well as simple  model selection criteria except when it is penalized by the imposition of a hard threshold.
algorithms based on hard thresholds for fold change are outperformed by shrinkage fold-change and by all other non-trivial algorithms that are not restricted by arbitrary thresholds: tables  <dig> and  <dig> show that hard-threshold algorithms are never ranked in the top four by either cross validation or posterior predictive expected loss.
while the best local-fdr-based methods outperform shrinkage fold-change, as can be seen in figures  <dig>   <dig>   <dig>   <dig>   <dig> and  <dig>  shrinkage fold change has performance comparable to simple model selection criteria as represented by the bayes factor, bic, and aicc methods.
herein we examined only algorithms that fall into one of two distinct categories:
<dig>  the shrinkage and hard-threshold fold-change algorithms are based on estimated fold change without regard for statistical significance or estimates of variance.
<dig>  all other algorithms of the present study compute levels of significance without regard for fold change estimates.
since these categories represent opposite extremes, their algorithms might be outperformed by those that instead employ both fold-change information and variance/significance information.
our observation that fold change performs as well as simple model-selection criteria suggests consideration of less extreme algorithms that combine the advantages of the ones studied herein.
investigators reported that the estimation of fold-change following a non-stringent significance filter performs better than does either type of algorithm alone  <cit> , and  <cit>  have recently demonstrated that further improvement is possible by smoothly shrinking estimates of fold change according to statistical significance levels.
adjusting fold-change estimates according to significance levels is not the only way to combine the two types of information.
a complementary strategy instead adjusts significance levels according to fold-change thresholds.
in fact, the seemingly inferior performance of statistical methods that do not make use of fold-change estimates has been explained in terms of a distinction between statistical and biological significance  <cit> , which would call for the incorporation of the lowest fold change considered biologically relevant into the statistical hypotheses under consideration.
recent statistical methods designed to find genes expressed at biologically important levels include those utilizing false discovery rates  <cit> , empirical and full bayesian analyses  <cit> , and the likelihood paradigm of measuring the strength of statistical evidence  <cit> .
corroboration of cross validation by posterior predictive expected loss
in general, cross-validation is subject to high variance when sample sizes are small.
if each of the features had independent data of finite variance, the central limit theorem would nonetheless guarantee a small variance in the overall measure of performance .
in the present case, however, due to gene-gene interactions, the numerator and denominator of the overall measure of performance are sums of positively correlated quantities.
to address this concern, we performed an additional assessment of the differential-expression-detection algorithms using posterior predictive expected squared error methodology.
the posterior predictive expected squared error requires the choice of a particular gold standard model, a bayesian model consisting of a family of sampling distributions refecting knowledge about the biological system and a prior distribution.
here, we based our posterior predictive expected squared error on the implicit bayesian model approximated by the local-fdr mean expression estimator calculated using t-statistics and the theoretical null distribution.
the key assumption of the model is that few genes are differentially expressed at any notable level; the model also assumes that gene expression ratios are lognormally distributed.
the model accommodates unequal variances for non-paired data using conventional improper priors under each hypothesis since we have little prior information about the specific parameter values.
naturally, the model's corresponding estimator had the lowest posterior predictive expected squared error, but provided the assumptions encoded in the model hold, the posterior predictive expected squared error will nonetheless be a good way to rank the performance of the estimators.
the fitting of the gold-standard model generated estimates for the proportions of equivalently expressed genes, allowing the verification of the assumption that most genes were equivalently expressed.
for the  <dig> days,  <dig> days, and  <dig> days data sets, the estimated proportion of equivalently expressed genes were  <dig> ,  <dig> , and  <dig> , respectively; for the  <dig> days vs.  <dig> days,  <dig> days vs.  <dig> days, and  <dig> days vs.  <dig> days data sets, they were  <dig> ,  <dig> , and  <dig> , respectively; and for the applied biosystems, affymetrix  <dig>  and affymetrix  <dig> data sets, they were  <dig> ,  <dig> , and  <dig> , respectively.
the six tomato data sets have relatively high proportions, showing that these data sets more closely satisfy the assumption of a proportion close to  <dig>  therefore, the local-fdr-based rankings for the estimators in these data sets should be accurate.
the maqc data sets have lower proportions, indicating that the model assumption is a poor approximation.
it is not surprising that the maqc data sets have many differentially expressed genes, as they are derived from liver tissue treated with a potent toxin.
as noted before, the cross-validation performance measure ranks the gold standard model highly for the tomato data sets, that is, for the data sets that we expect good estimation from the gold standard model.
furthermore, a careful inspection of figures  <dig>  k <dig>  k <dig> and k <dig> revealed that the rankings of the estimators according to the posterior predictive assessment and the cross-validation assessment largely agreed.
, and the moderated t-statistic with limma, which was rated highly by cross-validation but poorly by posterior predictive expected loss for the  <dig> days,  <dig> days,  <dig> days vs.  <dig> days, and  <dig> days vs.  <dig> days data sets .)
in addition, the cross-validation performance measure does not rank the gold standard model as highly for the affymetrix data sets; the gold standard model itself has determined that its assumption of a high proportion of equivalently expressed genes fails for precisely those data sets.
these observations suggest that the cross-validation methodology was able to accurately rank estimators even though the number of biological replicates was small.
CONCLUSIONS
the posterior predictive methodology helped to confirm that the cross-validation methodology was effective for measuring estimators' relative performances.
the results support the use of local-fdr-based statistical algorithms over both conventional model-select ion criteria and over algorithms based only on fold change.
in particular, the estimator based on the local fdr calculated using t-statistics and the theoretical null had the overall best performance when the proportion of equivalently expressed genes was high.
as a second choice, the estimator based on the local fdr calculated using moderated t-statistics also performed quite well.
tables  <dig> and  <dig> show that it was ranked in the top four for eight data sets out of nine, including all three maqc data sets.
the fact that fold-change shrinkage performed as well as conventional model selection criteria calls for investigating algorithms that combine the strengths of significance testing and fold-change estimation.
authors' contributions
cmy selected the bayes factor algorithm, implemented the bayes factor, bic, and aicc algorithms, and implemented the posterior predictive expected squared loss assessment.
drb conceived the study, selected the data sets, and applied the fold change and empirical bayes algorithms.
each author made substantial contributions to writing the paper.
both authors read and approved the final manuscript.
supplementary material
additional file 1
this file contains a heuristic overview and detailed derivation of our bayes factor approach to calculating probabilities of differential expression.
click here for file
acknowledgements
we thank pei-chun hsieh for preparing the biomedical data for analysis, xuemei tang for providing the fruit development microarray data, and the reviewers for many helpful suggestions.
the biobase package of bioconductor  facilitated management of the expression data.
this work was partially supported by the canada foundation for innovation, the ministry of research and innovation of ontario, and the faculty of medicine of the university of ottawa.

 continual invention of new microarray data analysis algorithms for the identification of which genes express differently across two groups calls for objectively comparing the performance of existing algorithms. while there have been thorough empirical comparisons between supervised learning methods of classifying microarrays, comparisons between methods of detecting differential gene expression have tended to depend heavily on either theory or simulation and thus on strong underlying assumptions. more empirical alternatives include the use of biologically-derived prior information regarding which experiments are more likely to contain differentially expressed genes and the use of spike-in data sets. the latter can represent equivalently expressed genes better than simulations, but the artificial spike-in levels do not necessarily correspond to levels of differentially expression across conditions of biological interest. an early report of the microarray quality control project may mark a turning point in the methodology of comparing of statistical methods designed to identify differential gene expression on the basis of microarray observations. the critical advantage of this "concordance" method is its validation entirely on the basis of the microarray data without resorting to spiking in known quantities of analytes or to prior information from other types of experiments; it is thus applicable to any microarray data set with sufficient replication. validation by non-microarray information such as rt-pcr measurements of gene expression or public pathway/functional information on genes does have great value in overcoming shortcomings in microarray platforms. for that very reason, however, such validation has markedly less value in judging the performance of statistical methods of detecting differential gene expression. for example, the inability of rt-pcr to validate a microarray prediction of differential gene expression might indicate a problem with the statistical assumptions used to make the prediction, but it may instead refect a problem with cross hybridization due to the microarray platform. participants in the maqc project avoided such confounding between microarray platform effects and statistical method effects by quantifying the degree of overlap between gene lists produced by an algorithm on the basis of two independent data sets. although a significant step forward, this way of comparing algorithms, like that of, requires examining gene lists of given sizes, which is why chen et al. consider the concordance to be too unstable for use as an algorithm performance criterion. without depending on arbitrarily selected numbers of genes, the platform-algorithm confounding may be overcome by cross validation, which instead uses a test set of microarrays to validate predictions made on the basis of a separate training set of microarrays, while maintaining the empirical nature of the concordance method. like concordance, cross validation does not incorporate knowledge that only a small portion of genes are differentially expressed. encoding this information when appropriate into a hierarchical model enables a more reliable assessment of the performance of differential expression detection algorithms if only a few biological replicates are available. these methods are explained in section and illustrated in section 3; implications are discussed in section methods if a gene is known to be differentially expressed at a certain level on average, then that level would predict future measurements of gene expression better than would making such predictions on the assumption that there is on average no differential expression. likewise, if a gene is known to be equivalently expressed, then using an expression level of or an expression ratio of would predict future measurements better than making such predictions on the assumption that there is some differential expression. thus, a method of selecting genes as differentially expressed may be judged by estimating its ability to predict future measurements of gene expression. this estimation may be carried out by a process of cross validation: the microarrays are divided between a training set used to determine which genes the method considers differentially expressed and a test set used to estimate how well such results would agree with future measurements. the strategy of assessing gene selection algorithms by estimated prediction error may be more precisely specified in mathematical notation. let xi,j denote the logarithm of the measured expression intensity or ratio of intensities of the ith of m genes in the jth of n biological replicates of the control or reference group; each value of xi,j may represent an average over technically replicated microarrays; xi =; x = t. likewise, denotes the logarithm of the measured expression intensity or ratio of intensities of the ith gene in the jth of n' biological replicates of the treatment or perturbation group;. the observations xi,j and are realizations of the random variables xi and, respectively. the ith gene is called equivalently expressed if e = or differentially expressed if e ≠ in hypothesis testing parlance, the null hypothesis associated with the ith gene is hi: e = gene selection algorithms a gene selection algorithm α returns πα, an estimate of the posterior probability that the ith gene is equivalently expressed; it follows that - πα is the algorithm's probability that the gene is differentially expressed across the perturbation and reference groups. many algorithms give πα directly as a local false discovery rate estimate, whereas traditional false discovery rate estimates and other non-bayesian algorithms in effect assign πα a value of either or depending on whether or not a gene is considered differentially expressed at a given threshold. for example, the practice of considering a gene differentially expressed if exp, its estimated fold change, is at least φ may be expressed as with φ >, and. the discontinuity can be removed by introducing smooth functions on an ad hoc basis; here we use as an example of such a smooth function. the trivial algorithms which completely ignore the data, will serve as informative points of reference. some of the empirical bayes algorithms implemented in two r packages are considered here. from calculations based on a moderated t-statistic that are performed by the r package limma, one may readily obtain pi (), a one-sided p-value of the ith null hypothesis; p () =, p (),..., pm ()). given the moderated t-statistics and π, the proportion of genes expected to be equivalently expressed, limma also computes logωi ), the estimated logarithm of the posterior odds that gene i is differentially expressed rather than equivalently expressed, from which the local false discovery rate may be readily obtained as ))-. since, for use with the log-odds, the author of the algorithm does not recommend computing π using limma's convest function, we instead iterated the log-odds function until convergence by adapting a method originally proposed for another empirical bayes algorithm: let π = 90% and initialize k to increment k by let. repeat steps 2- until the absolute value of the proportion difference is sufficiently small, i.e., |πk - πk- | < 1/ or until the sign of the proportion difference changes, i.e., - πk- ) - πk- ) < the number of iterations performed until such convergence is denoted by k. let π = πk. based on that value of π, the estimated probability of equivalent expression is derived by solving for it in the definition of the odds of differential expression, yielding also using standard distributions of test statistics under the null hypothesis, the r package locfdr maps p, a vector of single-tailed p-values for all genes, to estimates of a local false discovery rate, πlocfdr. the use of moderated t-statistics is incorporated by more commonly, p, a vector of standard t-test p-values, which also assume the normality of - xi, or p, a vector of wilcoxon test p-values, which do not assume normality, yield local false discovery rate estimates alternatively, the locfdr package can employ an empirical maximum-likelihood estimate of the null distribution for computation of the local-false-discovery-rate estimate πemp.null: whereas the empirical bayes algorithms provide approximations to a posterior probability of a hierarchical bayesian class of models, we included comparisons to the posterior probability πbayes factor under a non-hierarchical set of models. the data densities under the non-hierarchical models are based on the same assumptions as those of standard linear regression: unconstrained data means under the alternative hypothesis and, for each gene, normal iid noise and equal variance within each group in the unpaired case. let represent the hypothesis of differential expression. the posterior odds of differential expression under these models are where p is the prior predictive density or integrated likelihood under hypothesis h. the left-hand side of equation is the posterior odds of equivalent expression to differential expression; on the right-hand side, the first factor is the prior odds of equivalent expression to differential expression, and the second factor is known as the bayes factor. since we take p = p() = 1/ our posterior odds is equal to the bayes factor; thus putting equal prior mass on each hypothesis does not share the conservatism of the above empirical bayes algorithms. additional file gives the analytical derivation of the resulting posterior probability, which may be expressed in terms of some additional notation. define if n = n' and is paired with xi,j, or if and xi are independent. then the posterior probability is given by we also applied two "information criteria" used in model selection to estimate the posterior probability; the information criteria were applied to the same linear regression framework used in the above bayes factor computation. in model selection terminology, each criterion selects either model hi or model for the ith gene, but we instead averaged the estimates corresponding to the two models for each gene as follows. we first applied the bayesian information criterion. up to a factor of - 1/ and a constant term, the bic approximates the logarithm of the prior predictive probability density given a statistical model and a sufficiently diffuse proper prior distribution under the given model without requiring specification of such a prior. with a prior mass on each model considered, the bic leads to an approximation of a posterior probability that is less conservative than that of the above bayes factor. the general formula for the bic under a model with normal errors is where n is the number of data points and k is the number of parameters in the model. for paired data, n = n; under hi the only parameter is the data variance, giving k = while under the model includes both the data mean and data variance, giving k = therefore the bic for each hypothesis is with ssrh as defined in. for independent data, n = n+ n'; under hi the model includes a single mean log-expression level and the data variance, giving k = while under the model includes two distinct mean log-expression levels and the data variance, giving k = therefore the bic for each hypothesis is with ssrh as defined in. since we again use p = p (), the bic approximation of the posterior odds is equal to its approximation of bayes factors corresponding to a wide class of priors on the model parameters. transformed from the logarithmic scale to the probability scale, the result is an equation of the same form as, the second information criterion we assessed was the akaike information criterion corrected for small samples. while - aicc/ plus a constant term is in general only an approximately unbiased estimator of the expected kullback-leibler distance between the model/hypothesis and the unknown true data generating distribution, it is exactly unbiased for linear regression models with normal errors, a class that includes the present non-hierarchical models. under the name of akaike weights, it and other alc-like criteria have been used to generate predictions that take model uncertainty into account in a manner exactly analogous to bayesian model averaging, giving rise to an equation of the same form as. the general formula for the aicc under a model with normal errors is the particular values of n and k for paired and independent data under and hi are the same as those given above for the bic. for paired data, the aicc values of the hypotheses or models are with ssrh as defined in; for independent data, the aicc values are with ssrh as defined in. transforming from the logarithmic scale yields the effective probability where is the ratio of akaike weights. these algorithms were chosen as representatives of various classes of possible approaches. whereas the fold-change-dependent algorithms represent algorithms that take no account of the data variance, the information criterion algorithms and the non-hierarchical bayesian algorithm represent algorithms that do take data variance into account but do not share information across genes. the local fdr algorithms based on classical p-values share information across genes for the purpose of determining false discovery rates, thus accounting for multiple comparisons, but do not share information for estimating data variance. algorithms employing the moderated t-statistic share information across genes to account for multiple comparisons and also to estimate data variance. methods of assessing gene selection algorithms each of the next subsections describes a different method of quantifying the performance of gene selection algorithms. the first, cross validation, has the advantage that it is an unbiased estimator of squared prediction error without assuming any parametric model. the second, the computation of posterior predictive loss, takes advantage of the knowledge that gene expression is approximately lognormal and that relatively few genes will have substantial differential expression, the vast majority being equivalently expressed for all practical purposes. the two methods will differ in results; if nearly all genes have only negligible differential expression, the latter is deemed more reliable except in the case of extensive biological replication since the former achieves low bias by admitting a high variance of performance estimates. cross validation algorithm α's best prediction of future values of - xi is the posterior expected degree of expression, the term is the best estimator of the degree of expression conditional on definite knowledge that gene i is differentially expressed; it is multiplied by ), the posterior probability of differential expression. the posterior expected degree of expression has been compared to a method of correcting estimates for gene selection bias. for a new observation of gene i, the squared prediction error is, the squared prediction error does not directly target the question of which genes are differentially expressed; instead, it addresses the question of what the value of the next observation will be. however, good performance of one algorithm relative to another on either of these questions implies good performance on the other, as can be seen by considering that in general the mean squared prediction error is the sum of an algorithm's squared predictive bias and the data variance. the squared predictive bias term summarizes the ability of an algorithm to correctly distinguish differentially expressed genes from equivalently expressed genes. it is more fexible than the 0/ loss in that it penalizes algorithms not just for being wrong, but for how wrong they are. the data variance sets the scale for "wrongness", in that for one algorithm to appear significantly worse than another, its squared predictive bias must dominate the data variance term. under the "all nulls false" reference algorithm, the best prediction of future values of - xi for all genes is the maximum likelihood estimator. other algorithms make gains over this reference by correctly assigning equivalently expressed genes, thereby avoiding the contribution of the variance of the mle to the squared prediction error. under the "all nulls true" reference algorithm, the best prediction of future values of - xi for all genes is other algorithms make gains over this reference by correctly assigning differentially expressed genes, thereby avoiding the contribution of the squared bias ]2) to the squared prediction error. the squared prediction error criterion therefore quantifies the relative costs of false positives and false negatives in terms of the bias-variance trade-off. to estimate the squared prediction error, we used leave-one-out cross validation, if n = n' and is paired with xi,j or if and xi are independent, where means the jth replicate is omitted:, and. for example, suppose that is paired with xi,,j and the data for gene i were = and xi =. for j from to =, and using the fold change shrinkage calculation of equation - πα ) =. ).) the individual terms in the sum in equation are and ) and their mean is. if the given data were independent instead of paired, the calculation would involve each of the subsets obtained by leaving out one perturbation data point and one control data point. we considered measuring error relative to always predicting that - xi = on a gene-wise basis using the ratio with two measures of central tendency, is a fast, robust estimator of the mode that is suitable as for asymmetric distributions. it is implemented as the hsm function in the modeest package of r.) we also considered an absolute error criterion, this measure is relative to a base model such as the "all nulls true" model or the "all nulls false" model because we expect only the relative performances of the estimators to be meaningful. we found that the relative error mean essentially reproduced the absolute error relative to the "all nulls true" model, and the relative error mode often evaluated estimators as not practically different from the "all nulls true" benchmark. therefore, we show only the results for the absolute error measure. the use of cross-validation for estimation of classification error, appropriate for the problem of categorizing samples or microarrays given known classifications for use in the training and test sets, differs from the use cross-validation for estimation of squared prediction error, appropriate for the distinct problem of determining which genes are differentially expressed without knowledge of which genes are differentially expressed for use in the training and test sets. jeffery et al. used a cross-validation approach to estimate the predictive error of a variety of gene selection algorithms, but with microarray classification error rather than equations - as the performance criterion. such classification error depends not only on the gene selection algorithm, but also on the particular classifier for which that algorithm selects features. since our interest lies strictly in identifying differentially expressed genes, our methods instead quantify performance in terms of predicting new measurements. we have also addressed the problem using estimation error in place of prediction error. posterior predictive expected squared error the local fdr shrinkage algorithm can be used to define an estimator's posterior predictive expected squared error. in general, the posterior predictive expected squared error is where and xi,new are random variables for new observations, is algorithm α's point prediction for - xi,new, and eposterior and varposterior are the expectation and variance with respect to the posterior distribution. the effective posterior distribution that leads to estimators of the form is which has variance, we use the local fdr estimator with t-statistics and theoretical null distribution as our gold standard model for the computation of πα; this model will be accurate under the reasonable assumption that few genes are differentially expressed at appreciable levels. to fully express the posterior predictive loss, we must define the posterior predictive distribution for - xnew under both the null and alternative hypotheses for both paired and non-paired data. conditional on each hypothesis, we use improper prior distributions for convenience. strictly speaking, this is inconsistent with our choice of πα, an empirical bayes approximation to a posterior probability; under a full bayesian analysis, posterior probabilities of hypotheses can only be computed under proper priors for the parameters conditional on each hypothesis, as in the bayes factor algorithm of equation. our choice of πα enables sharing information across genes to give a sensible empirical bayes posterior probability for the hypotheses but otherwise relies on the same assumptions as our conditional prior distributions. for paired data under the null hypotheses, - xnew has a normal sampling distribution with zero mean and sampling variance estimated from the data. under the usual improper prior for the sampling variance ∝ σ-2), the posterior distribution for the sampling variance is a scaled-inverse-χ distribution with degrees of freedom n and scale. the posterior predictive density is the expectation of the sampling density with respect to the posterior distribution of the sampling variance, where n is the normal distribution parameterized in terms of mean and variance, and tv is a shifted, scaled version of the t distribution with v degrees of freedom, center c, and scale factor s., then /s is distributed as the usual tv distribution.) for paired data under the alternative hypothesis, - xnew has a normal sampling distribution with both mean and sampling variance estimated from the data. it can be shown that under the usual improper joint prior for mean μ ) and the sampling variance ∝ σ-2), the posterior predictive distribution for - xnew is, where, i.e., s is the usual unbiased variance estimator. for non-paired data under the null hypothesis, if the treatment and control data are modeled as having distinct sampling variances ) then the posterior predictive distribution is where and σ are the sampling variance for treatment and control data respectively. this integral is intractable because πposterior 2)has a non-standard form. we estimated it by drawing samples from πposterior 2) using markov chain monte carlo and then calculating the mcmc average, where the subscript k indicates the kth mcmc draw of parameter values and k is the total number of draws. in the present case, the mcmc algorithm we use is an inherently multi-chain procedure; we used chains. we used a burn-in of samples per chain, followed by samples per chain, for a total of k = samples. for each gene in a randomly chosen subset of genes from the complete data set, a contour plot of the posterior density was superimposed on a scatter plot of the mcmc draws of parameter values. the scatter plots visually conformed to the contours of the posterior densities, verifying that the mcmc draws of parameter values provided a good approximation to the posterior distributions. for non-paired data under the alternative hypothesis, and xnew each have a normal sampling distribution with both mean and sampling variance estimated from the data. it can be shown that under the usual improper joint prior for the individual means and sampling variances, the posterior predictive distributions for and xnew are and therefore to summarize gene-wise posterior predictive expected squared error over all genes in a data set, we considered quantities analogous to the relative errors and absolute errors of equations -, with gene-wise posterior predictive expected squared errors replacing cross-validation-derived prediction errors. again, we found that the relative error mean essentially replicated the results of the absolute error relative to the "all nulls true" benchmark; relative error mode evaluated the performance of all estimators as identical to the "all nulls true" benchmark. therefore, we show only the results for the absolute error measure for posterior predictive expected squared error.  to illustrate the proposed methods of quantifying the performance of gene selection algorithms, we applied them to two example data sets, one relevant to agriculture and the other to medicine. since this study is limited to the evaluation of high-level algorithms of detecting differentially expression, we did not consider multiple pre-processing schemes. the agricultural data sets were processed as described in; the medical data sets were pre-processed according to the specifications of the chip manufacturer. agricultural data dual-channel microarrays were used to measure in tomatoes the expression ratios of m = genes at the breaker stage of ripening and at and days thereafter. each of the later two stages has six biological replicates, but one of the biological replicates is missing at the breaker stage of ripening. the next subsection compares algorithms of determining which genes are differentially expressed between mutant and wild type at each point in time, whereas subsection uses the same data to instead compare algorithms of determining which genes are differentially expressed between one point in time and another point in time. pairing across microarray channels in order to determine the genes for which expected values of logarithms of mutant-to-wild-type ratios differ from let be the expression level of the mutant sample with mrna hybridized to the same microarray as that of a wild type sample with expression level xi,j at or days after the breaker stage. then - xi,j is the logarithm of the observed ratio for the ith gene and jth microarray. due to this dependence structure, paired t-tests and wilcoxon signed-rank tests were used to obtain p-values, and equation was used to estimate prediction error. we measured absolute error relative to the local fdr using the t-statistic and the theoretical null because this model had the best or near-best performance in seven out of the nine data sets considered in this paper. thus, its use as the base model facilitated the plotting of multiple data sets in a single figure; this same model was used as the base model for all subsequent figure. the estimated prediction errors for all algorithms mentioned above are displayed as figure two independent groups in order to determine which genes differ in mutant-to-wild-type ratios between different periods of time after the breaker stage, let and xi,j denote the logarithms of ratios observed at two different points in time for gene i and for microarrays j' and j. since the measurement errors of observations made at one time point are independent of those made at the other time point, 2-sample t-tests and wilcoxon rank-sum tests were used to obtain p-values, and equation was used to average estimated prediction error. figure shows, for each non-paired tomato data set, the total posterior predictive expected squared error ) for each estimator relative to that of the gold standard model. biomedical data maqc researchers measured gene expression responses to a rat liver treatment on four different platforms: applied biosystems, affymetrix, agilent, and ge healthcare. each data set has six treatment biological replicates and six control biological replicates. as in subsection. observations in the treatment group are not paired with those of the control group. the applied biosystems data set and the two affymetrix data sets were used to assess gene selection criteria on the basis of prediction error. figure shows, for each maqc data set, the total posterior predictive expected squared error ) for each estimator relative to that of the gold standard model. discussion fold change versus testing fold change performs about as well as simple model selection criteria except when it is penalized by the imposition of a hard threshold. algorithms based on hard thresholds for fold change are outperformed by shrinkage fold-change and by all other non-trivial algorithms that are not restricted by arbitrary thresholds: tables and show that hard-threshold algorithms are never ranked in the top four by either cross validation or posterior predictive expected loss. while the best local-fdr-based methods outperform shrinkage fold-change, as can be seen in figures and shrinkage fold change has performance comparable to simple model selection criteria as represented by the bayes factor, bic, and aicc methods. herein we examined only algorithms that fall into one of two distinct categories: the shrinkage and hard-threshold fold-change algorithms are based on estimated fold change without regard for statistical significance or estimates of variance. all other algorithms of the present study compute levels of significance without regard for fold change estimates. since these categories represent opposite extremes, their algorithms might be outperformed by those that instead employ both fold-change information and variance/significance information. our observation that fold change performs as well as simple model-selection criteria suggests consideration of less extreme algorithms that combine the advantages of the ones studied herein. investigators reported that the estimation of fold-change following a non-stringent significance filter performs better than does either type of algorithm alone, and have recently demonstrated that further improvement is possible by smoothly shrinking estimates of fold change according to statistical significance levels. adjusting fold-change estimates according to significance levels is not the only way to combine the two types of information. a complementary strategy instead adjusts significance levels according to fold-change thresholds. in fact, the seemingly inferior performance of statistical methods that do not make use of fold-change estimates has been explained in terms of a distinction between statistical and biological significance, which would call for the incorporation of the lowest fold change considered biologically relevant into the statistical hypotheses under consideration. recent statistical methods designed to find genes expressed at biologically important levels include those utilizing false discovery rates, empirical and full bayesian analyses, and the likelihood paradigm of measuring the strength of statistical evidence. corroboration of cross validation by posterior predictive expected loss in general, cross-validation is subject to high variance when sample sizes are small. if each of the features had independent data of finite variance, the central limit theorem would nonetheless guarantee a small variance in the overall measure of performance. in the present case, however, due to gene-gene interactions, the numerator and denominator of the overall measure of performance are sums of positively correlated quantities. to address this concern, we performed an additional assessment of the differential-expression-detection algorithms using posterior predictive expected squared error methodology. the posterior predictive expected squared error requires the choice of a particular gold standard model, a bayesian model consisting of a family of sampling distributions refecting knowledge about the biological system and a prior distribution. here, we based our posterior predictive expected squared error on the implicit bayesian model approximated by the local-fdr mean expression estimator calculated using t-statistics and the theoretical null distribution. the key assumption of the model is that few genes are differentially expressed at any notable level; the model also assumes that gene expression ratios are lognormally distributed. the model accommodates unequal variances for non-paired data using conventional improper priors under each hypothesis since we have little prior information about the specific parameter values. naturally, the model's corresponding estimator had the lowest posterior predictive expected squared error, but provided the assumptions encoded in the model hold, the posterior predictive expected squared error will nonetheless be a good way to rank the performance of the estimators. the fitting of the gold-standard model generated estimates for the proportions of equivalently expressed genes, allowing the verification of the assumption that most genes were equivalently expressed. for the days, days, and days data sets, the estimated proportion of equivalently expressed genes were,, and, respectively; for the days vs. days, days vs. days, and days vs. days data sets, they were,, and, respectively; and for the applied biosystems, affymetrix and affymetrix data sets, they were,, and, respectively. the six tomato data sets have relatively high proportions, showing that these data sets more closely satisfy the assumption of a proportion close to therefore, the local-fdr-based rankings for the estimators in these data sets should be accurate. the maqc data sets have lower proportions, indicating that the model assumption is a poor approximation. it is not surprising that the maqc data sets have many differentially expressed genes, as they are derived from liver tissue treated with a potent toxin. as noted before, the cross-validation performance measure ranks the gold standard model highly for the tomato data sets, that is, for the data sets that we expect good estimation from the gold standard model. furthermore, a careful inspection of figures k k and k revealed that the rankings of the estimators according to the posterior predictive assessment and the cross-validation assessment largely agreed., and the moderated t-statistic with limma, which was rated highly by cross-validation but poorly by posterior predictive expected loss for the days, days, days vs. days, and days vs. days data sets.) in addition, the cross-validation performance measure does not rank the gold standard model as highly for the affymetrix data sets; the gold standard model itself has determined that its assumption of a high proportion of equivalently expressed genes fails for precisely those data sets. these observations suggest that the cross-validation methodology was able to accurately rank estimators even though the number of biological replicates was small.  the posterior predictive methodology helped to confirm that the cross-validation methodology was effective for measuring estimators' relative performances. the results support the use of local-fdr-based statistical algorithms over both conventional model-select ion criteria and over algorithms based only on fold change. in particular, the estimator based on the local fdr calculated using t-statistics and the theoretical null had the overall best performance when the proportion of equivalently expressed genes was high. as a second choice, the estimator based on the local fdr calculated using moderated t-statistics also performed quite well. tables and show that it was ranked in the top four for eight data sets out of nine, including all three maqc data sets. the fact that fold-change shrinkage performed as well as conventional model selection criteria calls for investigating algorithms that combine the strengths of significance testing and fold-change estimation. authors' contributions cmy selected the bayes factor algorithm, implemented the bayes factor, bic, and aicc algorithms, and implemented the posterior predictive expected squared loss assessment. drb conceived the study, selected the data sets, and applied the fold change and empirical bayes algorithms. each author made substantial contributions to writing the paper. both authors read and approved the final manuscript. supplementary material additional file 1 this file contains a heuristic overview and detailed derivation of our bayes factor approach to calculating probabilities of differential expression. click here for file acknowledgements we thank pei-chun hsieh for preparing the biomedical data for analysis, xuemei tang for providing the fruit development microarray data, and the reviewers for many helpful suggestions. the biobase package of bioconductor facilitated management of the expression data. this work was partially supported by the canada foundation for innovation, the ministry of research and innovation of ontario, and the faculty of medicine of the university of ottawa.
the fact that fold-change shrinkage performed as well as conventional model selection criteria calls for investigating algorithms that combine the strengths of significance testing and fold-change estimation. sustained research on the problem of determining which genes are differentially expressed on the basis of microarray data has yielded a plethora of statistical algorithms, each justified by theory, simulation, or ad hoc validation and yet differing in practical results from equally justified algorithms. as a nonparametric method of estimating prediction error from observed expression levels, cross validation provides an empirical approach to assessing algorithms for detecting differential gene expression that is fully justified for large numbers of biological replicates. under conditions in which its approximations are valid, it corroborates the results of cross validation. for the data sets considered, estimating prediction error by cross validation demonstrates that empirical bayes methods based on hierarchical models tend to outperform algorithms based on selecting genes by their fold changes or by non-hierarchical model-selection criteria. algorithms for detecting differential gene expression may be compared by estimating each algorithm's error in predicting expression ratios, whether such ratios are defined across microarray channels or between two independent groups. this method has the advantage of basing its assessment solely on the results of real data analyses, but as it requires examining gene lists of given sizes, it may be unstable. because it leverages the knowledge that only a small portion of genes are differentially expressed, the posterior predictive method is expected to provide more reliable estimates of algorithm performance, allaying concerns about limited biological replication. recently, a concordance method that measures agreement among gene lists have been introduced to assess various aspects of differential gene expression detection.
 sustained research on the problem of determining which genes are differentially expressed on the basis of microarray data has yielded a plethora of statistical algorithms, each justified by theory, simulation, or ad hoc validation and yet differing in practical results from equally justified algorithms. recently, a concordance method that measures agreement among gene lists have been introduced to assess various aspects of differential gene expression detection. this method has the advantage of basing its assessment solely on the results of real data analyses, but as it requires examining gene lists of given sizes, it may be unstable.  two methodologies for assessing predictive error are described: a cross-validation method and a posterior predictive method. as a nonparametric method of estimating prediction error from observed expression levels, cross validation provides an empirical approach to assessing algorithms for detecting differential gene expression that is fully justified for large numbers of biological replicates. because it leverages the knowledge that only a small portion of genes are differentially expressed, the posterior predictive method is expected to provide more reliable estimates of algorithm performance, allaying concerns about limited biological replication. in practice, the posterior predictive method can assess when its approximations are valid and when they are inaccurate. under conditions in which its approximations are valid, it corroborates the results of cross validation. both comparison methodologies are applicable to both single-channel and dual-channel microarrays. for the data sets considered, estimating prediction error by cross validation demonstrates that empirical bayes methods based on hierarchical models tend to outperform algorithms based on selecting genes by their fold changes or by non-hierarchical model-selection criteria. the posterior predictive assessment corroborates these findings.  algorithms for detecting differential gene expression may be compared by estimating each algorithm's error in predicting expression ratios, whether such ratios are defined across microarray channels or between two independent groups. according to two distinct estimators of prediction error, algorithms using hierarchical models outperform the other algorithms of the study. the fact that fold-change shrinkage performed as well as conventional model selection criteria calls for investigating algorithms that combine the strengths of significance testing and fold-change estimation. 