BACKGROUND
in recent years there has been a rapidly growing understanding of how gene expression reflects and determines biological states.
this has come about through the widespread use of microarray expression profiling  <cit> .
yet there have been concerns about the accuracy and reproducibility of the technology.
some early studies reported poor reproducibility and dramatic differences between platforms  <cit> .
although other studies have generally reported better accuracy and agreement  <cit> , especially later studies using more developed statistical methods  <cit> , the early concern has contributed to an explosion in the number of publications comparing microarray platforms or assessing microarray accuracy.
despite the growing number of publications, only a limited number of methods are available to assess the accuracy of genome-scale expression platforms.
before discussing the strengths and weakness of the various strategies, it is necessary to dissect exactly what is meant by accuracy.
there are several dimensions to platform accuracy.
the first major dimension is consistency, i.e., the ability of the platform to agree with itself.
this can be further divided into reproducibility or precision on one hand, and dynamic range or sensitivity on the other.
these features determine the ability of the platform to distinguish differentially expressed transcripts from those which are not.
the sensitivity can be further examined to check whether the measured probe intensities increase linearly with transcript expression level.
this determines the ability of the platform to return accurate estimates of the fold changes for differentially expression genes.
the second major dimension is probe annotation.
accurate annotation determines the ability of the platform to agree with independent measures of differential expression for the same genes.
it might be for example that a microarray platform accurately measures the expression level for some gene, but the probe is incorrectly annotated as another gene.
another possibility is that two microarray platforms might both accurately measure expression for the correct gene, but might nevertheless disagree because they respond to different splice-variants or isoforms of that gene  <cit> .
annotation accuracy is likely to improve for all platforms as knowledge of the genome improves.
we can view self-consistency as the innate accuracy of the platform because it can be improved only by a change in the underlying technology.
many platform comparison articles use variability between technical replicates to measure precision  <cit> , but this doesn't measure sensitivity or linearity.
to measure sensitivity, it is necessary to introduce genes which are known to be differentially expressed.
pcr is the traditional method for validating microarray discoveries, so some studies use quantitative pcr to determine the true differential expression status for a subset of genes  <cit> .
this approach is practical only for a small proportion of the probes, and has some other disadvantages which are discussed below.
another method of introducing known fold changes is to spike-in a small number of artificial genes into the rna sample at known concentrations  <cit> .
this technique requires that alien control probes be printed onto the arrays and the corresponding transcripts spiked into the rna samples before labeling and hybridization.
spike-in controls however are inevitably platform specific and hence are more suited to comparing different processing methods for a given platform than for comparing different platforms.
even for a given platform, spike-in controls suffer from a number of shortcomings.
only a small number of distinct transcripts can be spiked-in and these are often at a higher message abundance than the endogenous genes and at relatively large fold changes.
finally, the fact that the spike-in rna is not extracted with the main rna sample and has to be added separately means that the spike-in probes will not follow the same intensity-dependent normalization curve as the regular probes.
this means that the spike-in probes will generally need to be normalized separately to the other probes and will not necessarily be representative of the precision of the other probes.
most platform comparison studies begin by matching probes between platforms, and then measuring the agreement between platforms for the common genes  <cit> .
although natural, this approach suffers from a number of problems.
firstly, there is no perfect or unambiguous strategy for matching probes.
matching on unigene id is clearly insufficient because each platform may have multiple probes with possibly different expression profiles mapped to the same id.
the same problem applies to other gene classification systems.
more detailed sequence analysis may attempt to position the probes at particular loci in the genome  <cit> , but even so an unambiguous matching of probes between platforms is highly unlikely.
when platforms disagree, one cannot easily identify the source of disagreement, whether this is due to innate imprecision of the platforms or incorrect annotation.
more basically, when two platforms disagree the comparison does not tell us which platform is correct, or even if either is correct.
even the validation of microarray results by pcr can be subject to these problems, especially if the splice variants for the gene in question are not completely described.
a small number of studies have used a dilution series of rna samples to construct samples with a range of fold changes  <cit> .
two other studies have taken the approach mixing mrna from two sources in known quantities for the same purpose  <cit> .
in this article we use a series of mixed mrna samples and introduce a more systematic method of analysis.
mixtures of mrna were derived from two cell lines in varying proportions to cover a large dynamic range in gene expression.
a statistical methodology is introduced for evaluating the precision, sensitivity and linearity of microarray platforms.
the novel design allows the series of arrays to be analyzed as a whole.
the mixture series give predictable changes in fold change against which the actual pattern of observed log-ratios can be compared.
there are several advantages to this design.
firstly, each microarray platform is evaluated on a whole-array basis.
every probe on the arrays adds information in assessing the precision of the platform as each probe will behave according to the mixtures series.
secondly, there is no need to match probes across platforms as each platform yields its own measure.
on the other hand, if the additional step of matching probes across platforms is performed, then our method provides a decomposition of variability into within- and between-platform components for each gene.
this provides, for the first time, a quantitative assessment of the degree of systematic disagreement between the platforms.
our method could be applied to evaluate the precision of any whole-genome gene expression technology, not only microarrays.
we use our methodology to assess and compare four widely used microarray platforms: two commercial platforms, affymetrix and agilent, and two academically produced platforms, spotted cdna and oligonucleotide arrays.
this selection includes three two-color platforms and one single-channel platform .
our comparison includes careful consideration of special issues which affect the different technologies including dye-effects for the two-color platforms and experimental design differences between the two-color and single-channel platforms.
our intention is to evaluate the best-case scenario for each platform rather than to evaluate laboratory to laboratory variation as several other studies have done  <cit> .
hence we hybridized all the two-color microarrays in a single experienced laboratory.
of equal importance as the cross-platform issues is the evaluation of alternative data processing algorithms for a given platform.
pre-processing algorithms by which expression measures are quantified from microarray images include background correction, normalization and summarization methods  <cit> .
there is a slowly emerging appreciation in the literature of the size of the noise reduction that can be achieved by state-of-the-art pre-processing methods  <cit> .
for this study we used what we consider to be best-practice pre-processing methods for each platform.
our data and associated statistical analysis provide an objective means to compare different pre-processing methods for each platform.
to illustrate this we compare several popular methods of pre-processing affymetrix genechip expression values and show that the method used in our study performs better than the alternatives.
RESULTS
the titration series design
six rna mixes were prepared ranging from pure mcf <dig>  through four mixtures of mcf <dig> and jurkat  to pure jurkat  .
for each of the three two-color platforms, twelve arrays were hybridized .
each of the six rna mixes was compared to pure jurkat , with a dye-swap pair of arrays used for each comparison.
for the single channel platform, fourteen arrays were hybridized, two for each of rna mixes a1–a <dig> and b. the rationale for this design is that the comparison of pure rnas a <dig> vs b should show considerable differential expression and can be used to evaluate the responsiveness or sensitivity of the platforms, while the same vs same comparison a <dig> vs b should show no differential expression and can be used to evaluate the variability or specificity of the platforms.
the graduated mixes a2–a <dig> can be used to evaluate the ability of each platform to faithfully track fold changes as the relative expression changes.
our analysis is in two major parts.
first we examine the precision, i.e., the repeatability, of each platform individually.
this analysis uses all non-control probes on the arrays.
this analysis does not attempt to match probes between platforms but simply assumes each platform contains a comparable selection of probes.
later we do match probes across the four platforms, to the extent that is possible, and examine the concordance of the four platforms on these common genes.
samples a <dig> through a <dig> were created by mixing total mrna from mcf <dig> breast epithelial cells with mrna derived from the jurkat t cell line in the proportions shown.
the sample b was used as a reference rna in all two-color hybridizations .
for affymetrix hybridizations, samples a <dig> through b were hybridized independently.
nonlinear regressions
for each probe p, let fp be the intensity corresponding to the expression level of the corresponding transcript in mcf <dig> and let jp be the intensity corresponding to its expression in jurkat.
if c is the proportion of mcf <dig> in a particular rna mix, and if the observed intensity is proportional to expression, then the intensity of the mix should be cfp + jp.
also let rp = fp/jp be the expression ratio  between mcf <dig> and jurkat for that probe.
the fold change between the rna mix and the jurkat reference should be {cfp + jp}/jp = crp +  <dig> - c. this shows that, although we do not know the true values of fp or jp for any gene, the log-expression and log-ratio values must change in a predictable way across the titration series.
for the single-channel platform, let epj be the normalized log-intensity for probe p and array j, j =  <dig> ..., <dig>  the theoretical log-intensity for probe p and array j is
αpj = log2{cjfp + jp}
where cj is the proportion of mcf <dig> in the rna mix used for array j. the observed log-intensity epj can be represented as
epj = log <dig> = αpj + εpj
where εpj represents measurement error.
we assume that the εpj are independent with mean zero and with probe dependent standard deviations ϕp.
for the two-color platforms, let mpi be the normalized log-ratio for probe p on array i, i =  <dig> ..., <dig>  the theoretical log-ratio for probe p and array i is
βpi = log2
where ci is the proportion of mcf <dig> in rna mix used for array i. the observed log-ratio can be represented as
mpi = log <dig> = diβpi + εpi
where εpi represents measurement error and di = - <dig> if the array is dye-swapped and  <dig> otherwise.
we assume that the εpi are independent with mean zero and with probe dependent standard derivations σp.
note that the minimum value that the fold change rp can take is zero, for a probe whose transcript is present in jurkat but completely absent in mcf <dig>  the minimum possible value for the log-ratio βpi is therefore log <dig>  the log of the proportion of jurkat in the mix.
the titration series introduces in this way a theoretical lower bound on the range of possible log-ratios.
the horizontal green lines in figure  <dig> represent this bound.
any m-value which falls below this line does so in error.
in general, all four platforms maintain m-values consistently above the line, with the greatest transgressions occurring for the spotted oligonucleotide arrays.
the standard deviations σp and φp are the keys to quantifying the precision of the platforms, because they represent the array-to-array variability which is expected in measuring log-ratios or log-expression values.
our approach is to view equations  and  as nonlinear regression equations for the epj and the mpi, with unknown regression parameters fp and jp for equation  and rp for equation .
these equations are fitted by least squares to the available data for each probe to obtain estimated fold changes and precisions for that probe.
regression equation  was fitted to the  <dig> epj data values for each affymetrix probe-set and equation  was fitted to the  <dig> mpi data values for each probe on each two-color platform.
a complete worked example of the computations for one gene is given in the supplementary materials  <cit> .
examples of fitted curves are shown in figure  <dig>  in each case, the fitted curve is that which minimizes the sum of squared deviations of the data values from the curve.
these calculations yielded an estimated fold change rp for each probe on each two-color platform, which we call the consensus estimate of the fold change between mcf <dig> and jurkat for each probe, because it is derived from the entire series of twelve arrays.
the consensus estimator is more precise than the simple mean log-ratio that could be obtained from the pure mcf <dig> and jurkat arrays alone.
the residual standard error sp from each regression is our estimate for σp.
note that sp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaacqwgzbwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30a3@ is a consistent and approximately unbiased estimator for σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ by standard nonlinear least squares theory  <cit> .
for each affymetrix probe-set, the intensities fp and jp were estimated from the observed log-expression values by non-linear least squares.
this produced consensus estimates of mcf <dig> and jurkat expression for each probe from the entire series fourteen arrays.
the residual standard error from each regression is our estimate of φp.
in this way, precision estimates were obtained for each probe on each platform.
note that we applied least squares to the epj rather than to the un-logged expression values and to the mpi rather than the fold changes because the logged values are more nearly symmetrically and normally distributed.
dye effects
another consideration with the two-color platforms is the possibility of probe-specific dye bias, i.e., the tendency of particular probes to incorporate one of the dyes more readily than the other  <cit> .
dye-swaps were incorporated into our design specifically to detect and balance such effects.
equation  can be modified to allow for a dye-effect as
mpi = δp + diβpi + εpi
where the intercept term δp represents the probe specific dye effect.
if probe p does show a specific dye bias, then including the intercept term will reduce the estimate of σp as the new equation  more closely models the behavior of the probe.
if not, the estimated σp will on average remain the same, although the residual degrees of freedom for the regression will decrease by one.
boxplots of estimated σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ with and without the dye effect term are given in figure  <dig>  it is clear that accounting for dye effects improved the fits for all three platforms.
for the agilent arrays the improvement is particularly noticeable, nearly halving the median estimated σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@.
each of the two-color platforms was hybridized with identical material from a large pool of labeled mrna, so the larger effect for agilent arrays is not due to the labeled material applied to the arrays.
we conclude that all two color platforms show small but noticeable dye effects for a large proportion of the probes.
these dye effects can be estimated and removed in a suitably designed experiment allowing inference to be conducted with improved precision and less bias.
precision of individual platforms
comparing estimated σp values shows that the agilent platform is more precise, i.e., more repeatable, than the two spotted platforms, which are very similar in precision.
the median σp s are  <dig> ,  <dig>  and  <dig>  for the cdna, oligo and agilent platforms respectively, after correction for dye-effects.
these standard deviations correspond to plus/minus fold changes of 17%, 16% and 6% respectively.
the step from the spotted platforms to agilent is substantial.
the numbers suggest that  <dig> =  <dig>  technical replicate oligo arrays are required to estimate a typical fold change with the same precision as one agilent array.
comparing the two-color platforms with affymetrix is not so straightforward, because σp measures the precision of log-ratios while φp measures the precision of log-intensities.
quantitative comparison can only be done in the context of specific experimental designs.
if two affymetrix arrays are used to estimate log-fold changes between two rna targets, the standard deviation of the estimate for probe p should be 2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@φp, assuming the two arrays to be independent and to have equal variances.
with two-color arrays, we need to distinguish between direct comparisons, in which the two rna targets of interest are hybridized competitively to the same array, and indirect comparisons, whereby one channel of each array is hybridized with a common reference sample  <cit> .
if two two-color arrays are used to compare two rna targets indirectly via a common reference, the standard deviation of the estimated log-fold change for probe p should be 2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@σp, again assuming the arrays to be independent with equal variances.
on the other hand, if rna targets are compared directly on the same arrays, the standard deviation would be σp using one array and σp/2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@ using two replicate arrays  <cit> .
table  <dig> gives the precisions, i.e., standard deviations, with which a log-fold change would be estimated from a minimal experiment for each of the platforms.
for the two-color platforms, results are given for both a direct comparison using one array and for an indirect comparison using two arrays.
for affymetrix, the results assume a comparison between two arrays.
agilent is the most precise platform, providing that the dye-effect has been adjusted as indicated above.
the spotted arrays are nearly as precise as affymetrix if they use direct comparison but much less precise if they do not.
the table gives the median standard deviation across probes together with the first and third quartiles.
for the two-color platforms the design may be direct using one array or indirect via a common reference using two arrays.
for affymetrix, two arrays are used.
the two-color calculations assume that estimates of probe-specific dye effects are available, .e.g., from a larger experiment including dye-swaps.
boxplots of the probe-wise standard deviations are shown in figure  <dig>  for the two-color platforms the results assume a direct design and are corrected for probe-specific dye effects, so the situation is the same as for the top half of table  <dig>  the conclusions are the same as for table  <dig>  the agilent arrays perform clearly the best, followed by affymetrix.
for all the platforms there is a considerable range of precisions across the probes.
the full distribution of probe-wise standard deviations for each platform is shown as a density histogram in supplementary figure  <dig> <cit> .
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfdpwcgaqcamaadaaaleaacqwgwbacaeaacqaiyagmaaaaaa@310e@ are plotted, corresponding to a direct comparison with one array after adjustment for probe-specific dye-effects.
for affymetrix, values of log22ϕ^p
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaacqaiyagmiigacuwfvpgagaqcamaabaaaleaacqwgwbacaeqaaaaa@3116@ are plotted corresponding to a comparison of two arrays.
the median lines correspond to the direct design median values in table  <dig>  and the boxplot outlines correspond to the q <dig> and q <dig> given in table  <dig>
it is well known that the variability of expression measures can depend on the abundance of the gene transcript, although the pre-processing methods we used are designed in part to reduce this dependence.
figure  <dig> displays trend lines of σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ and 2ϕp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfvpgadaqhaawcbagaemicaahabagaegomaidaaaaa@3107@ against mean a-value, and these show generally decreasing trends of variability versus intensity.
the spotted arrays are far more variable for lower intensity probes while the commercial platforms maintain precision more evenly over the entire intensity range.
agilent appears to have greater precision than affymetrix for probes in the middle range of intensities.
the precisions of the four platforms tend to converge for the highest intensity probes.
in figure  <dig>  the a-values for the different platforms have been scaled to have the same range.
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ values for the two-color platforms, corresponding to a direct comparison with one array with dye-effect adjustment, and to the logp 2ϕp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfvpgadaqhaawcbagaemicaahabagaegomaidaaaaa@3107@ values for affymetrix, corresponding to a comparison of two arrays.
the horizontal axis is the average log-intensity over the a <dig> arrays for the two color platforms and over the a <dig> and b arrays for affymetrix.
each unit on the vertical axis corresponds to a doubling of statistical information.
pure error and bias
it should be kept in mind when interpreting σp and φp that they are overall measures of residual variation which respond to both the variability between technical replicate arrays  and any failure of the nonlinear regression curves to track the correct fold changes .
although the regression curves give a visually good fit to the empirical data , there may nevertheless be evidence for lack of fit.
such lack of fit might arise from a failure of the microarray spot intensities to respond to expression levels with perfect proportionality, or possibly from small pipetting errors in preparing the titration series.
the pure error and bias components are additive according to σ <dig> = σpure error2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaeeiuaalaeeydaunaeeocainaeeyzaumaeeiiaaiaeeyraukaeeocainaeeocainaee4ba8maeeocaihabagaegomaidaaaaa@3c67@ +  <dig> for each probe.
we estimated the pure error and bias components by decomposing the residual sum of squares from each of the nonlinear regressions  and  into pure error and lack of fit components, and formed f-statistics to test for lack of fit by dividing the lack of fit mean square by the pure error mean square  <cit>  .
the f-statistics were found to be randomly distributed around unity , which is evidence that there is little or no systematic lack of fit.
none of the f-statistics were statistically significant after bonferroni or holm adjustment for multiple testing across probes.
as an extra precaution, in case lack of fit exists but is not detected by the statistical tests because of lack of power, we estimated the size of the lack of fit variance component for each probe.
for all probes which show differentially expression between mcf <dig> and jurkat, the estimated lack of fit variance component  was either zero or extremely small compared with the sums of squares due to the nonlinear regression trend .
we conclude that the nonlinear regressions describe the data more than adequately for our purposes and that pure error is the dominant component of the residual standard errors.
concordance between platforms
we now turn to the question of systematic discordance between the platforms, and for this purpose we need to match probes corresponding to common genes across the platforms.
unigene ids were obtained for all probes, using the unigene build of  <dig> november  <dig>  and  <dig>  genes were found to be present on all four platforms.
this was not sufficient to match probes across platforms because most  of the common genes were represented by more than one probe, or probe set in the case of affymetrix, on at least one of the platforms.
in order to have an objective method of matching probes uniquely across platforms, we chose for each gene the probe on each platform with the highest average expression level.
in other words, we chose the probe whose transcript is most highly represented in the cell lines used in the study.
choosing a single representative probe on each platform in this way was preferable to averaging multiple probes for a given gene, for two reasons.
firstly, different probes for the same gene on the same platform in many cases showed quite different expression profiles, so that averaging would be inappropriate.
secondly, taking averages of different numbers of probes would prevent the platforms from being compared on the same basis.
the probes matched in this way showed broadly similar behavior across the platforms but also noticeable differences .
genes are sorted by average mcf <dig> vs jurkat log-ratio across platforms and columns correspond to individual arrays progressing from a <dig> on the left to a <dig> on the right.
dye-swapped arrays are un-swapped here in silico, i.e., are shown with m-values reversed in sign.
as expected, we observe a progressive reduction in the number of differentially expressed genes from left to right, although with somewhat more variability for the two spotted platforms.
the dye-effect noted above can be seen in the alternating vertical stripes for the two-color platforms, most noticeable for agilent.
the fold-change values rp estimated by the nonlinear regression are also in broad agreement across the platforms, although there is considerable scatter around equality .
points lying on the y =  <dig> line for each pane in figure  <dig> indicate probes which fail to respond in that platform.
all four platforms seem to be returning a similar range of fold changes.
cross-platforms correlations range from  <dig>  between cdna and oligo to  <dig>  between agilent and affymetrix .
affymetrix enjoys the best agreement with the other platforms and cdna the least.
the oligo platform is better correlated with agilent and affymetrix than the cdna platform despite being no more precise, suggesting that the annotation of the compugen probes is superior to that of the cdna probes.
by comparing the estimated fold changes with the platform precisions, the cross-platform variability can be decomposed into within-platform and between-platform components.
that is, the disagreement between any two platforms can be separated into the variation inherent in the two platforms and the variation caused by trying to match probes across different technologies.
let m <dig> and m <dig> be log-ratios for a particular gene from two technologies, i.e., m <dig> is the log-ratio observed on an array of platform  <dig> and m <dig> is observed on an array of platform  <dig>  for affymetrix, the log-ratio would be obtained by comparing two arrays.
the expected squared discrepancy between the two log-ratios is the sum of variance and squared-bias:
e{2} = variance + bias2
= σ12
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ + σ22
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ + {log <dig>  - log2}2
where σ12
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ and σ22
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ are the precisions σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ for this gene on the two platforms and r <dig> and r <dig> are the fold changes rp for this gene on two platforms.
for the a <dig> samples the proportion of mcf <dig> is c =  <dig> so
e{2} = σ12
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ + σ22
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ + {log2} <dig>
the first two terms on the right-hand-side of equation  <dig> are the within-platform components while the third term is the between-platform component, simply the squared log-ratio of the fold-changes for this gene on the two platforms.
for most genes the third term turns out to be considerably larger than the first two, particularly for genes with larger fold-changes.
figure  <dig> plots each of the three components of equation  <dig> against average log-fold change for the  <dig>  matched genes when comparing the cdna and agilent platforms.
for each component, the plot is given as a loess trend line.
the between-platform component is larger than the within platform components at all fold changes but becomes dominant as the fold change increases.
this relationship of within- and between-platform variation is qualitatively the same for all the combinations of platforms .
previous work has shown that that matching genes across platforms using sequences rather than unigene ids reduces the differences seen between platforms  <cit> .
it is unlikely however that the qualitative picture in which the between-platform component dominates would be materially changed.
comparing pre-processing methods for affymetrix data
all results presented above for the affymetrix arrays use expression values for each probe-set on each array produced by the rma software algorithm  <cit> .
most affymetrix expression results published in the literature have been processed however using the microarray analysis suite  software from affymetrix, now replaced by genechip operating software  <cit> .
affymetrix has recently developed a new preprocessing algorithm called plier which incorporates some of the ideas associated with rma.
to examine how these methods might change the results, we repeated all the analysis described above using mas and plier data.
mas  <dig>  and plier were used to extract the data using affymetrix's recommended parameters.
for plier, quantile normalisation was used and background subtraction was selected as pm-mm with an offset of  <dig> =  <dig> added for variance stabilization at the low intensity end .
the nonlinear model was fitted to the mas  <dig> and plier data and the variances σp2
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ were recalculated.
investigation of the variances as a function of probe intensity revealed that for both mas  <dig> and plier the variance increases at lower intensities .
it can be seen that both rma and plier give more precise results than mas over the entire intensity range.
the difference is substantial, corresponding to a four-fold increase in statistical information for most probes.
for high intensities, rma and plier give almost identical variance measures.
at low intensities, rma gives lower variances than plier.
at these intensities, rma is also returning smaller fold changes than plier, so it appears to be attenuating the fold changes in this intensity range in order to achieve the observed reduction in variability.
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfvpgagaqcaaaa@2e8f@ versus the average log-intensity of mcf <dig> and jurkat for the three pre-processing algorithms mas <dig> , plier  and rma .
it can be seen that mas  <dig> gives higher variance than the other algorithms over all intensities while rma and plier are almost identical at high intensities.
at low intensities the rma algorithm is more precise than plier, perhaps at the cost of greater bias.
discussion and
CONCLUSIONS
this article has described a new approach to evaluate the reliability and accuracy of microarray platforms.
the method consists of an easy to construct titration series and an associated statistical analysis.
the global performance of each microarray platform is determined by examining the deviation of every probe on the array from its expected behavior.
this differs from most previous work where the behavior of a gene is not an expected value, but rather is subject to post hoc validation.
implicit in much previous work is that the behavior of the whole array is inferred from a relatively small number of validated expression values.
a series of mixtures of two distinct rna sources is used to induce a graduated series of fold changes for each probe.
our design uses one of the pure rna samples as a common reference.
this strategy ensures that the analysis can be conducted entirely in terms of log-ratios for the two-color platforms.
nonlinear regression is used to model the expected pattern by which expression must change as a function of the mixing proportions.
this appears to be the first application of nonlinear regression to microarray data.
the residual variation from the probe-wise regressions provides a measure of precision and the lack of fit of the curves measures deviation from response linearity for each platform.
the nonlinear regression approach involves a novel concept, the idea that a consensus measure of fold change between the rna sources can be obtained from the entire series of arrays, even though most of the arrays do not compare the pure samples.
the consensus fold change provides a measure of dynamic range for each platform.
for the titration series to be most effective, the two rna sources should contain many differentially expressed probes including a wide range of fold changes.
the rnas chosen in this article were mcf <dig> breast epithelial cells  <cit>  and jurkat t-cells  <cit> .
previous work involving more than  <dig> replicates of mcf <dig> versus jurkat hybridization, as part of a large microarray quality control program, showed these lines to have a very divergent pattern of gene expression consistent with their diverse tissues of origin  <cit> .
four platforms were compared in this study.
the ink-jet printed two-color agilent arrays and the single channel in situ synthesized affymetrix arrays gave the most consistent results with respect to the predicted behavior.
academically produced spotted cdna and oligonucleotide arrays give somewhat more variable expression measurements, as would be expected given their lower cost and quality control.
all the platforms yielded acceptable precision however.
even the least precise cdna platform gave a typical standard deviation of only  <dig>  for measured log-ratios from a single array , which translates into variation of ± 17% for the fold changes .
precision is intensity dependent for the spotted arrays but far less so for the commercial arrays.
the spotted arrays are virtually as precise as the more expensive platforms for high-intensity probes but the commercial platforms appear better able to resolve differential expression at lower expression levels.
the aim of this study was to evaluate the best-case scenario for each platform.
in practice, the platforms may perform differently in other laboratories, particularly in less experienced hands.
the hybridization process is more automated for the commercial platforms, especially affymetrix, so these platforms can be expected to give stable results for most laboratories.
our experience in analyzing microarray data from many laboratories suggests that the academic array platforms are more subject to operator and laboratory variability and to loss of quality in inexperienced hands.
the effective precision of the two-color platforms was found to be substantially improved by allowing for probe-specific dye effects in the statistical model.
this implies that microarray experiments using two-color arrays should be designed with sufficient alternation of dye assignment to allow probe-specific dye effects to be identified.
the results also imply that experimenters should routinely allow for probe-specific effects in model-based analyses of two-color microarray data  <cit> .
it was a design criterion of our study that all two-color microarrays were hybridized with identical labeled material.
each point in the titration series was labeled as a pool and then divided three ways between the platforms.
this approach was taken to control for any possible titration inaccuracy or variation between probe labeling protocols.
the agilent protocol calls for labeled riboprobes generated by a proprietary kit.
it is possible that our decision to standardize the labeling protocol across platforms may have contributed to the large probe-specific dye effects observed with the agilent arrays.
nevertheless we have found in separate studies that the two protocols produce very similar results , suggesting that any bias introduced by the labeling protocol is likely to be small.
obtaining expression values from microarrays requires several post-hybridization data analysis steps including background correction and normalization.
these steps are often collectively called pre-processing to distinguish them from subsequent data analysis steps.
the methodology developed in this article provides a means to objectively compare different pre-processing methods for any given platform.
it is our intention to make the raw expression data generated by this study publicly available.
a website will be established through which researchers can interactively evaluate, for any of the four platforms, any pre-processing method which is implemented as a function in the r software environment.
this is analogous to but more general than the resource provided by the affycomp software package for the affymetrix platform  <cit> .
it can be argued that the molecular biology literature has been slow to appreciate the extent to which the choice of pre-processing algorithm can impact on the precision of microarray data.
the emphasis has often been on filtering highly variable spots rather than on ensuring low variability in the first place.
this study compared three pre-processing methods for affymetrix data and showed that the differences are substantial.
the propriety affymetrix pre-processing algorithm, the most frequently used in the literature, was found to be easily the worst of the three methods.
the differences were so large that affymetrix quantified with mas <dig>  was the worst of all the platforms considered in this study whereas affymetrix quantified with rma was nearly the best.
the loss of information of mas <dig>  compared to rma or plier varies from about 2-fold for highly expressed genes to 16-fold at low intensities.
similarly unfavorable comparisons could have been made with commonly used pre-processing methods for the two-color platforms .
this highlights the importance of using state-of-the-art pre-processing methods to get reliable data from microarray technologies.
after probe matching, the four platforms were found to broadly agree, with log-ratios across probes giving correlations of around  <dig> .
the differences are large enough though to establish that the platforms do differ systematically and, in fact, the majority of inter-platform disagreement is due to systematic disagreement.
this agrees with qualitative observations in other studies  <cit> .
a quantitative decomposition of variation into within- and between-platform components is given for the first time.
in this study, probes were matched across platforms on unigene id.
where there were multiple probes, or probe-sets, on a platform for the same unigene id, the most highly expressed probe was chosen.
it is probable that better agreement could have been obtained between the platforms by blast searching  <cit>  each individual probe sequence and, where there were multiple probes, attempting to match probes based on sequence similarity or on locus within a gene.
careful probe sequence matching of this type is likely to be valuable in the future for improving cross-platform agreement, but it would be unlikely to change the qualitative results presented in this study.
the presence of systematic disagreement between the platforms emphasizes the need for improved annotation and, in particular, for identification of the splice variants and other isoforms for each gene.
systematic differences have implications for how well published expression profiles for a particular phenotype will be predictive using other platforms  <cit> .
it is widely accepted that there is a high error rate in annotation of early cdna libraries, and despite some efforts at re-annotation, for example in the nia  <dig> k mouse clone set  <cit> , errors have tended to propagate along with the library.
recent work analyzing affymetrix probe-sets showed a surprisingly high level of mismatches between probe sequences and the refseq clone sequence the probes purportedly interrogate  <cit> , with only 62% of human u133a probe sequences being perfectly matched.
the consequence of mis-annotated or mismatched probes has been shown in experiments where correlations between platforms derived from sequence verified probes are generally higher than for all probes  <cit> .
recent technologies such as exon and tiling arrays offer potential for elucidation of splice variants and finer annotation of microarray probes  <cit> .
quantifying the degree of between-platform disagreement for each gene can serve to prioritize probes for re-annotation.

 in recent years there has been a rapidly growing understanding of how gene expression reflects and determines biological states. this has come about through the widespread use of microarray expression profiling. yet there have been concerns about the accuracy and reproducibility of the technology. some early studies reported poor reproducibility and dramatic differences between platforms. although other studies have generally reported better accuracy and agreement, especially later studies using more developed statistical methods, the early concern has contributed to an explosion in the number of publications comparing microarray platforms or assessing microarray accuracy. despite the growing number of publications, only a limited number of methods are available to assess the accuracy of genome-scale expression platforms. before discussing the strengths and weakness of the various strategies, it is necessary to dissect exactly what is meant by accuracy. there are several dimensions to platform accuracy. the first major dimension is consistency, i.e., the ability of the platform to agree with itself. this can be further divided into reproducibility or precision on one hand, and dynamic range or sensitivity on the other. these features determine the ability of the platform to distinguish differentially expressed transcripts from those which are not. the sensitivity can be further examined to check whether the measured probe intensities increase linearly with transcript expression level. this determines the ability of the platform to return accurate estimates of the fold changes for differentially expression genes. the second major dimension is probe annotation. accurate annotation determines the ability of the platform to agree with independent measures of differential expression for the same genes. it might be for example that a microarray platform accurately measures the expression level for some gene, but the probe is incorrectly annotated as another gene. another possibility is that two microarray platforms might both accurately measure expression for the correct gene, but might nevertheless disagree because they respond to different splice-variants or isoforms of that gene. annotation accuracy is likely to improve for all platforms as knowledge of the genome improves. we can view self-consistency as the innate accuracy of the platform because it can be improved only by a change in the underlying technology. many platform comparison articles use variability between technical replicates to measure precision, but this doesn't measure sensitivity or linearity. to measure sensitivity, it is necessary to introduce genes which are known to be differentially expressed. pcr is the traditional method for validating microarray discoveries, so some studies use quantitative pcr to determine the true differential expression status for a subset of genes. this approach is practical only for a small proportion of the probes, and has some other disadvantages which are discussed below. another method of introducing known fold changes is to spike-in a small number of artificial genes into the rna sample at known concentrations. this technique requires that alien control probes be printed onto the arrays and the corresponding transcripts spiked into the rna samples before labeling and hybridization. spike-in controls however are inevitably platform specific and hence are more suited to comparing different processing methods for a given platform than for comparing different platforms. even for a given platform, spike-in controls suffer from a number of shortcomings. only a small number of distinct transcripts can be spiked-in and these are often at a higher message abundance than the endogenous genes and at relatively large fold changes. finally, the fact that the spike-in rna is not extracted with the main rna sample and has to be added separately means that the spike-in probes will not follow the same intensity-dependent normalization curve as the regular probes. this means that the spike-in probes will generally need to be normalized separately to the other probes and will not necessarily be representative of the precision of the other probes. most platform comparison studies begin by matching probes between platforms, and then measuring the agreement between platforms for the common genes. although natural, this approach suffers from a number of problems. firstly, there is no perfect or unambiguous strategy for matching probes. matching on unigene id is clearly insufficient because each platform may have multiple probes with possibly different expression profiles mapped to the same id. the same problem applies to other gene classification systems. more detailed sequence analysis may attempt to position the probes at particular loci in the genome, but even so an unambiguous matching of probes between platforms is highly unlikely. when platforms disagree, one cannot easily identify the source of disagreement, whether this is due to innate imprecision of the platforms or incorrect annotation. more basically, when two platforms disagree the comparison does not tell us which platform is correct, or even if either is correct. even the validation of microarray results by pcr can be subject to these problems, especially if the splice variants for the gene in question are not completely described. a small number of studies have used a dilution series of rna samples to construct samples with a range of fold changes. two other studies have taken the approach mixing mrna from two sources in known quantities for the same purpose. in this article we use a series of mixed mrna samples and introduce a more systematic method of analysis. mixtures of mrna were derived from two cell lines in varying proportions to cover a large dynamic range in gene expression. a statistical methodology is introduced for evaluating the precision, sensitivity and linearity of microarray platforms. the novel design allows the series of arrays to be analyzed as a whole. the mixture series give predictable changes in fold change against which the actual pattern of observed log-ratios can be compared. there are several advantages to this design. firstly, each microarray platform is evaluated on a whole-array basis. every probe on the arrays adds information in assessing the precision of the platform as each probe will behave according to the mixtures series. secondly, there is no need to match probes across platforms as each platform yields its own measure. on the other hand, if the additional step of matching probes across platforms is performed, then our method provides a decomposition of variability into within- and between-platform components for each gene. this provides, for the first time, a quantitative assessment of the degree of systematic disagreement between the platforms. our method could be applied to evaluate the precision of any whole-genome gene expression technology, not only microarrays. we use our methodology to assess and compare four widely used microarray platforms: two commercial platforms, affymetrix and agilent, and two academically produced platforms, spotted cdna and oligonucleotide arrays. this selection includes three two-color platforms and one single-channel platform. our comparison includes careful consideration of special issues which affect the different technologies including dye-effects for the two-color platforms and experimental design differences between the two-color and single-channel platforms. our intention is to evaluate the best-case scenario for each platform rather than to evaluate laboratory to laboratory variation as several other studies have done. hence we hybridized all the two-color microarrays in a single experienced laboratory. of equal importance as the cross-platform issues is the evaluation of alternative data processing algorithms for a given platform. pre-processing algorithms by which expression measures are quantified from microarray images include background correction, normalization and summarization methods. there is a slowly emerging appreciation in the literature of the size of the noise reduction that can be achieved by state-of-the-art pre-processing methods. for this study we used what we consider to be best-practice pre-processing methods for each platform. our data and associated statistical analysis provide an objective means to compare different pre-processing methods for each platform. to illustrate this we compare several popular methods of pre-processing affymetrix genechip expression values and show that the method used in our study performs better than the alternatives.  the titration series design six rna mixes were prepared ranging from pure mcf through four mixtures of mcf and jurkat to pure jurkat. for each of the three two-color platforms, twelve arrays were hybridized. each of the six rna mixes was compared to pure jurkat, with a dye-swap pair of arrays used for each comparison. for the single channel platform, fourteen arrays were hybridized, two for each of rna mixes a1–a and b. the rationale for this design is that the comparison of pure rnas a vs b should show considerable differential expression and can be used to evaluate the responsiveness or sensitivity of the platforms, while the same vs same comparison a vs b should show no differential expression and can be used to evaluate the variability or specificity of the platforms. the graduated mixes a2–a can be used to evaluate the ability of each platform to faithfully track fold changes as the relative expression changes. our analysis is in two major parts. first we examine the precision, i.e., the repeatability, of each platform individually. this analysis uses all non-control probes on the arrays. this analysis does not attempt to match probes between platforms but simply assumes each platform contains a comparable selection of probes. later we do match probes across the four platforms, to the extent that is possible, and examine the concordance of the four platforms on these common genes. samples a through a were created by mixing total mrna from mcf breast epithelial cells with mrna derived from the jurkat t cell line in the proportions shown. the sample b was used as a reference rna in all two-color hybridizations. for affymetrix hybridizations, samples a through b were hybridized independently. nonlinear regressions for each probe p, let fp be the intensity corresponding to the expression level of the corresponding transcript in mcf and let jp be the intensity corresponding to its expression in jurkat. if c is the proportion of mcf in a particular rna mix, and if the observed intensity is proportional to expression, then the intensity of the mix should be cfp + jp. also let rp = fp/jp be the expression ratio between mcf and jurkat for that probe. the fold change between the rna mix and the jurkat reference should be {cfp + jp}/jp = crp + - c. this shows that, although we do not know the true values of fp or jp for any gene, the log-expression and log-ratio values must change in a predictable way across the titration series. for the single-channel platform, let epj be the normalized log-intensity for probe p and array j, j =..., the theoretical log-intensity for probe p and array j is αpj = log2{cjfp + jp} where cj is the proportion of mcf in the rna mix used for array j. the observed log-intensity epj can be represented as epj = log = αpj + εpj     where εpj represents measurement error. we assume that the εpj are independent with mean zero and with probe dependent standard deviations ϕp. for the two-color platforms, let mpi be the normalized log-ratio for probe p on array i, i =..., the theoretical log-ratio for probe p and array i is βpi = log2 where ci is the proportion of mcf in rna mix used for array i. the observed log-ratio can be represented as mpi = log = diβpi + εpi     where εpi represents measurement error and di = - if the array is dye-swapped and otherwise. we assume that the εpi are independent with mean zero and with probe dependent standard derivations σp. note that the minimum value that the fold change rp can take is zero, for a probe whose transcript is present in jurkat but completely absent in mcf the minimum possible value for the log-ratio βpi is therefore log the log of the proportion of jurkat in the mix. the titration series introduces in this way a theoretical lower bound on the range of possible log-ratios. the horizontal green lines in figure represent this bound. any m-value which falls below this line does so in error. in general, all four platforms maintain m-values consistently above the line, with the greatest transgressions occurring for the spotted oligonucleotide arrays. the standard deviations σp and φp are the keys to quantifying the precision of the platforms, because they represent the array-to-array variability which is expected in measuring log-ratios or log-expression values. our approach is to view equations and as nonlinear regression equations for the epj and the mpi, with unknown regression parameters fp and jp for equation and rp for equation. these equations are fitted by least squares to the available data for each probe to obtain estimated fold changes and precisions for that probe. regression equation was fitted to the epj data values for each affymetrix probe-set and equation was fitted to the mpi data values for each probe on each two-color platform. a complete worked example of the computations for one gene is given in the supplementary materials. examples of fitted curves are shown in figure in each case, the fitted curve is that which minimizes the sum of squared deviations of the data values from the curve. these calculations yielded an estimated fold change rp for each probe on each two-color platform, which we call the consensus estimate of the fold change between mcf and jurkat for each probe, because it is derived from the entire series of twelve arrays. the consensus estimator is more precise than the simple mean log-ratio that could be obtained from the pure mcf and jurkat arrays alone. the residual standard error sp from each regression is our estimate for σp. note that sp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaacqwgzbwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30a3@ is a consistent and approximately unbiased estimator for σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ by standard nonlinear least squares theory. for each affymetrix probe-set, the intensities fp and jp were estimated from the observed log-expression values by non-linear least squares. this produced consensus estimates of mcf and jurkat expression for each probe from the entire series fourteen arrays. the residual standard error from each regression is our estimate of φp. in this way, precision estimates were obtained for each probe on each platform. note that we applied least squares to the epj rather than to the un-logged expression values and to the mpi rather than the fold changes because the logged values are more nearly symmetrically and normally distributed. dye effects another consideration with the two-color platforms is the possibility of probe-specific dye bias, i.e., the tendency of particular probes to incorporate one of the dyes more readily than the other. dye-swaps were incorporated into our design specifically to detect and balance such effects. equation can be modified to allow for a dye-effect as mpi = δp + diβpi + εpi     where the intercept term δp represents the probe specific dye effect. if probe p does show a specific dye bias, then including the intercept term will reduce the estimate of σp as the new equation more closely models the behavior of the probe. if not, the estimated σp will on average remain the same, although the residual degrees of freedom for the regression will decrease by one. boxplots of estimated σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ with and without the dye effect term are given in figure it is clear that accounting for dye effects improved the fits for all three platforms. for the agilent arrays the improvement is particularly noticeable, nearly halving the median estimated σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@. each of the two-color platforms was hybridized with identical material from a large pool of labeled mrna, so the larger effect for agilent arrays is not due to the labeled material applied to the arrays. we conclude that all two color platforms show small but noticeable dye effects for a large proportion of the probes. these dye effects can be estimated and removed in a suitably designed experiment allowing inference to be conducted with improved precision and less bias. precision of individual platforms comparing estimated σp values shows that the agilent platform is more precise, i.e., more repeatable, than the two spotted platforms, which are very similar in precision. the median σp s are, and for the cdna, oligo and agilent platforms respectively, after correction for dye-effects. these standard deviations correspond to plus/minus fold changes of 17%, 16% and 6% respectively. the step from the spotted platforms to agilent is substantial. the numbers suggest that = technical replicate oligo arrays are required to estimate a typical fold change with the same precision as one agilent array. comparing the two-color platforms with affymetrix is not so straightforward, because σp measures the precision of log-ratios while φp measures the precision of log-intensities. quantitative comparison can only be done in the context of specific experimental designs. if two affymetrix arrays are used to estimate log-fold changes between two rna targets, the standard deviation of the estimate for probe p should be 2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@φp, assuming the two arrays to be independent and to have equal variances. with two-color arrays, we need to distinguish between direct comparisons, in which the two rna targets of interest are hybridized competitively to the same array, and indirect comparisons, whereby one channel of each array is hybridized with a common reference sample. if two two-color arrays are used to compare two rna targets indirectly via a common reference, the standard deviation of the estimated log-fold change for probe p should be 2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@σp, again assuming the arrays to be independent with equal variances. on the other hand, if rna targets are compared directly on the same arrays, the standard deviation would be σp using one array and σp/2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaadagcaaqaaiabikdayawcbeaaaaa@2db9@ using two replicate arrays. table gives the precisions, i.e., standard deviations, with which a log-fold change would be estimated from a minimal experiment for each of the platforms. for the two-color platforms, results are given for both a direct comparison using one array and for an indirect comparison using two arrays. for affymetrix, the results assume a comparison between two arrays. agilent is the most precise platform, providing that the dye-effect has been adjusted as indicated above. the spotted arrays are nearly as precise as affymetrix if they use direct comparison but much less precise if they do not. the table gives the median standard deviation across probes together with the first and third quartiles. for the two-color platforms the design may be direct using one array or indirect via a common reference using two arrays. for affymetrix, two arrays are used. the two-color calculations assume that estimates of probe-specific dye effects are available,.e.g., from a larger experiment including dye-swaps. boxplots of the probe-wise standard deviations are shown in figure for the two-color platforms the results assume a direct design and are corrected for probe-specific dye effects, so the situation is the same as for the top half of table the conclusions are the same as for table the agilent arrays perform clearly the best, followed by affymetrix. for all the platforms there is a considerable range of precisions across the probes. the full distribution of probe-wise standard deviations for each platform is shown as a density histogram in supplementary figure. mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfdpwcgaqcamaadaaaleaacqwgwbacaeaacqaiyagmaaaaaa@310e@ are plotted, corresponding to a direct comparison with one array after adjustment for probe-specific dye-effects. for affymetrix, values of log22ϕ^p mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaacqaiyagmiigacuwfvpgagaqcamaabaaaleaacqwgwbacaeqaaaaa@3116@ are plotted corresponding to a comparison of two arrays. the median lines correspond to the direct design median values in table and the boxplot outlines correspond to the q and q given in table it is well known that the variability of expression measures can depend on the abundance of the gene transcript, although the pre-processing methods we used are designed in part to reduce this dependence. figure displays trend lines of σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ and 2ϕp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfvpgadaqhaawcbagaemicaahabagaegomaidaaaaa@3107@ against mean a-value, and these show generally decreasing trends of variability versus intensity. the spotted arrays are far more variable for lower intensity probes while the commercial platforms maintain precision more evenly over the entire intensity range. agilent appears to have greater precision than affymetrix for probes in the middle range of intensities. the precisions of the four platforms tend to converge for the highest intensity probes. in figure the a-values for the different platforms have been scaled to have the same range. mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ values for the two-color platforms, corresponding to a direct comparison with one array with dye-effect adjustment, and to the logp 2ϕp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfvpgadaqhaawcbagaemicaahabagaegomaidaaaaa@3107@ values for affymetrix, corresponding to a comparison of two arrays. the horizontal axis is the average log-intensity over the a arrays for the two color platforms and over the a and b arrays for affymetrix. each unit on the vertical axis corresponds to a doubling of statistical information. pure error and bias it should be kept in mind when interpreting σp and φp that they are overall measures of residual variation which respond to both the variability between technical replicate arrays and any failure of the nonlinear regression curves to track the correct fold changes. although the regression curves give a visually good fit to the empirical data, there may nevertheless be evidence for lack of fit. such lack of fit might arise from a failure of the microarray spot intensities to respond to expression levels with perfect proportionality, or possibly from small pipetting errors in preparing the titration series. the pure error and bias components are additive according to σ = σpure error2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaeeiuaalaeeydaunaeeocainaeeyzaumaeeiiaaiaeeyraukaeeocainaeeocainaee4ba8maeeocaihabagaegomaidaaaaa@3c67@ + for each probe. we estimated the pure error and bias components by decomposing the residual sum of squares from each of the nonlinear regressions and into pure error and lack of fit components, and formed f-statistics to test for lack of fit by dividing the lack of fit mean square by the pure error mean square. the f-statistics were found to be randomly distributed around unity, which is evidence that there is little or no systematic lack of fit. none of the f-statistics were statistically significant after bonferroni or holm adjustment for multiple testing across probes. as an extra precaution, in case lack of fit exists but is not detected by the statistical tests because of lack of power, we estimated the size of the lack of fit variance component for each probe. for all probes which show differentially expression between mcf and jurkat, the estimated lack of fit variance component was either zero or extremely small compared with the sums of squares due to the nonlinear regression trend. we conclude that the nonlinear regressions describe the data more than adequately for our purposes and that pure error is the dominant component of the residual standard errors. concordance between platforms we now turn to the question of systematic discordance between the platforms, and for this purpose we need to match probes corresponding to common genes across the platforms. unigene ids were obtained for all probes, using the unigene build of november and genes were found to be present on all four platforms. this was not sufficient to match probes across platforms because most of the common genes were represented by more than one probe, or probe set in the case of affymetrix, on at least one of the platforms. in order to have an objective method of matching probes uniquely across platforms, we chose for each gene the probe on each platform with the highest average expression level. in other words, we chose the probe whose transcript is most highly represented in the cell lines used in the study. choosing a single representative probe on each platform in this way was preferable to averaging multiple probes for a given gene, for two reasons. firstly, different probes for the same gene on the same platform in many cases showed quite different expression profiles, so that averaging would be inappropriate. secondly, taking averages of different numbers of probes would prevent the platforms from being compared on the same basis. the probes matched in this way showed broadly similar behavior across the platforms but also noticeable differences. genes are sorted by average mcf vs jurkat log-ratio across platforms and columns correspond to individual arrays progressing from a on the left to a on the right. dye-swapped arrays are un-swapped here in silico, i.e., are shown with m-values reversed in sign. as expected, we observe a progressive reduction in the number of differentially expressed genes from left to right, although with somewhat more variability for the two spotted platforms. the dye-effect noted above can be seen in the alternating vertical stripes for the two-color platforms, most noticeable for agilent. the fold-change values rp estimated by the nonlinear regression are also in broad agreement across the platforms, although there is considerable scatter around equality. points lying on the y = line for each pane in figure indicate probes which fail to respond in that platform. all four platforms seem to be returning a similar range of fold changes. cross-platforms correlations range from between cdna and oligo to between agilent and affymetrix. affymetrix enjoys the best agreement with the other platforms and cdna the least. the oligo platform is better correlated with agilent and affymetrix than the cdna platform despite being no more precise, suggesting that the annotation of the compugen probes is superior to that of the cdna probes. by comparing the estimated fold changes with the platform precisions, the cross-platform variability can be decomposed into within-platform and between-platform components. that is, the disagreement between any two platforms can be separated into the variation inherent in the two platforms and the variation caused by trying to match probes across different technologies. let m and m be log-ratios for a particular gene from two technologies, i.e., m is the log-ratio observed on an array of platform and m is observed on an array of platform for affymetrix, the log-ratio would be obtained by comparing two arrays. the expected squared discrepancy between the two log-ratios is the sum of variance and squared-bias: e{2} = variance + bias2 = σ12 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ + σ22 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ + {log - log2}2 where σ12 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ and σ22 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ are the precisions σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ for this gene on the two platforms and r and r are the fold changes rp for this gene on two platforms. for the a samples the proportion of mcf is c = so e{2} = σ12 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegymaedabagaegomaidaaaaa@3085@ + σ22 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaegomaidabagaegomaidaaaaa@3087@ + {log2}     the first two terms on the right-hand-side of equation are the within-platform components while the third term is the between-platform component, simply the squared log-ratio of the fold-changes for this gene on the two platforms. for most genes the third term turns out to be considerably larger than the first two, particularly for genes with larger fold-changes. figure plots each of the three components of equation against average log-fold change for the matched genes when comparing the cdna and agilent platforms. for each component, the plot is given as a loess trend line. the between-platform component is larger than the within platform components at all fold changes but becomes dominant as the fold change increases. this relationship of within- and between-platform variation is qualitatively the same for all the combinations of platforms. previous work has shown that that matching genes across platforms using sequences rather than unigene ids reduces the differences seen between platforms. it is unlikely however that the qualitative picture in which the between-platform component dominates would be materially changed. comparing pre-processing methods for affymetrix data all results presented above for the affymetrix arrays use expression values for each probe-set on each array produced by the rma software algorithm. most affymetrix expression results published in the literature have been processed however using the microarray analysis suite software from affymetrix, now replaced by genechip operating software. affymetrix has recently developed a new preprocessing algorithm called plier which incorporates some of the ideas associated with rma. to examine how these methods might change the results, we repeated all the analysis described above using mas and plier data. mas and plier were used to extract the data using affymetrix's recommended parameters. for plier, quantile normalisation was used and background subtraction was selected as pm-mm with an offset of = added for variance stabilization at the low intensity end. the nonlinear model was fitted to the mas and plier data and the variances σp2 mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemicaahabagaegomaidaaaaa@30fe@ were recalculated. investigation of the variances as a function of probe intensity revealed that for both mas and plier the variance increases at lower intensities. it can be seen that both rma and plier give more precise results than mas over the entire intensity range. the difference is substantial, corresponding to a four-fold increase in statistical information for most probes. for high intensities, rma and plier give almost identical variance measures. at low intensities, rma gives lower variances than plier. at these intensities, rma is also returning smaller fold changes than plier, so it appears to be attenuating the fold changes in this intensity range in order to achieve the observed reduction in variability. mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfvpgagaqcaaaa@2e8f@ versus the average log-intensity of mcf and jurkat for the three pre-processing algorithms mas, plier and rma. it can be seen that mas gives higher variance than the other algorithms over all intensities while rma and plier are almost identical at high intensities. at low intensities the rma algorithm is more precise than plier, perhaps at the cost of greater bias. discussion and  this article has described a new approach to evaluate the reliability and accuracy of microarray platforms. the method consists of an easy to construct titration series and an associated statistical analysis. the global performance of each microarray platform is determined by examining the deviation of every probe on the array from its expected behavior. this differs from most previous work where the behavior of a gene is not an expected value, but rather is subject to post hoc validation. implicit in much previous work is that the behavior of the whole array is inferred from a relatively small number of validated expression values. a series of mixtures of two distinct rna sources is used to induce a graduated series of fold changes for each probe. our design uses one of the pure rna samples as a common reference. this strategy ensures that the analysis can be conducted entirely in terms of log-ratios for the two-color platforms. nonlinear regression is used to model the expected pattern by which expression must change as a function of the mixing proportions. this appears to be the first application of nonlinear regression to microarray data. the residual variation from the probe-wise regressions provides a measure of precision and the lack of fit of the curves measures deviation from response linearity for each platform. the nonlinear regression approach involves a novel concept, the idea that a consensus measure of fold change between the rna sources can be obtained from the entire series of arrays, even though most of the arrays do not compare the pure samples. the consensus fold change provides a measure of dynamic range for each platform. for the titration series to be most effective, the two rna sources should contain many differentially expressed probes including a wide range of fold changes. the rnas chosen in this article were mcf breast epithelial cells and jurkat t-cells. previous work involving more than replicates of mcf versus jurkat hybridization, as part of a large microarray quality control program, showed these lines to have a very divergent pattern of gene expression consistent with their diverse tissues of origin. four platforms were compared in this study. the ink-jet printed two-color agilent arrays and the single channel in situ synthesized affymetrix arrays gave the most consistent results with respect to the predicted behavior. academically produced spotted cdna and oligonucleotide arrays give somewhat more variable expression measurements, as would be expected given their lower cost and quality control. all the platforms yielded acceptable precision however. even the least precise cdna platform gave a typical standard deviation of only for measured log-ratios from a single array, which translates into variation of ± 17% for the fold changes. precision is intensity dependent for the spotted arrays but far less so for the commercial arrays. the spotted arrays are virtually as precise as the more expensive platforms for high-intensity probes but the commercial platforms appear better able to resolve differential expression at lower expression levels. the aim of this study was to evaluate the best-case scenario for each platform. in practice, the platforms may perform differently in other laboratories, particularly in less experienced hands. the hybridization process is more automated for the commercial platforms, especially affymetrix, so these platforms can be expected to give stable results for most laboratories. our experience in analyzing microarray data from many laboratories suggests that the academic array platforms are more subject to operator and laboratory variability and to loss of quality in inexperienced hands. the effective precision of the two-color platforms was found to be substantially improved by allowing for probe-specific dye effects in the statistical model. this implies that microarray experiments using two-color arrays should be designed with sufficient alternation of dye assignment to allow probe-specific dye effects to be identified. the results also imply that experimenters should routinely allow for probe-specific effects in model-based analyses of two-color microarray data. it was a design criterion of our study that all two-color microarrays were hybridized with identical labeled material. each point in the titration series was labeled as a pool and then divided three ways between the platforms. this approach was taken to control for any possible titration inaccuracy or variation between probe labeling protocols. the agilent protocol calls for labeled riboprobes generated by a proprietary kit. it is possible that our decision to standardize the labeling protocol across platforms may have contributed to the large probe-specific dye effects observed with the agilent arrays. nevertheless we have found in separate studies that the two protocols produce very similar results, suggesting that any bias introduced by the labeling protocol is likely to be small. obtaining expression values from microarrays requires several post-hybridization data analysis steps including background correction and normalization. these steps are often collectively called pre-processing to distinguish them from subsequent data analysis steps. the methodology developed in this article provides a means to objectively compare different pre-processing methods for any given platform. it is our intention to make the raw expression data generated by this study publicly available. a website will be established through which researchers can interactively evaluate, for any of the four platforms, any pre-processing method which is implemented as a function in the r software environment. this is analogous to but more general than the resource provided by the affycomp software package for the affymetrix platform. it can be argued that the molecular biology literature has been slow to appreciate the extent to which the choice of pre-processing algorithm can impact on the precision of microarray data. the emphasis has often been on filtering highly variable spots rather than on ensuring low variability in the first place. this study compared three pre-processing methods for affymetrix data and showed that the differences are substantial. the propriety affymetrix pre-processing algorithm, the most frequently used in the literature, was found to be easily the worst of the three methods. the differences were so large that affymetrix quantified with mas was the worst of all the platforms considered in this study whereas affymetrix quantified with rma was nearly the best. the loss of information of mas compared to rma or plier varies from about 2-fold for highly expressed genes to 16-fold at low intensities. similarly unfavorable comparisons could have been made with commonly used pre-processing methods for the two-color platforms. this highlights the importance of using state-of-the-art pre-processing methods to get reliable data from microarray technologies. after probe matching, the four platforms were found to broadly agree, with log-ratios across probes giving correlations of around. the differences are large enough though to establish that the platforms do differ systematically and, in fact, the majority of inter-platform disagreement is due to systematic disagreement. this agrees with qualitative observations in other studies. a quantitative decomposition of variation into within- and between-platform components is given for the first time. in this study, probes were matched across platforms on unigene id. where there were multiple probes, or probe-sets, on a platform for the same unigene id, the most highly expressed probe was chosen. it is probable that better agreement could have been obtained between the platforms by blast searching each individual probe sequence and, where there were multiple probes, attempting to match probes based on sequence similarity or on locus within a gene. careful probe sequence matching of this type is likely to be valuable in the future for improving cross-platform agreement, but it would be unlikely to change the qualitative results presented in this study. the presence of systematic disagreement between the platforms emphasizes the need for improved annotation and, in particular, for identification of the splice variants and other isoforms for each gene. systematic differences have implications for how well published expression profiles for a particular phenotype will be predictive using other platforms. it is widely accepted that there is a high error rate in annotation of early cdna libraries, and despite some efforts at re-annotation, for example in the nia k mouse clone set, errors have tended to propagate along with the library. recent work analyzing affymetrix probe-sets showed a surprisingly high level of mismatches between probe sequences and the refseq clone sequence the probes purportedly interrogate, with only 62% of human u133a probe sequences being perfectly matched. the consequence of mis-annotated or mismatched probes has been shown in experiments where correlations between platforms derived from sequence verified probes are generally higher than for all probes. recent technologies such as exon and tiling arrays offer potential for elucidation of splice variants and finer annotation of microarray probes. quantifying the degree of between-platform disagreement for each gene can serve to prioritize probes for re-annotation. 
concerns are often raised about the accuracy of microarray technologies and the degree of cross-platform agreement, but there are yet no methods which can unambiguously evaluate precision and sensitivity for these technologies on a whole-array basis. the effective precision of the two-color platforms is improved by allowing for probe-specific dye-effects in the statistical model. the method evaluates the precision and responsiveness of each microarray platform on a whole-array basis, i.e., using all the probes, without the need to match probes across platforms. the method consists of an easy-to-construct titration series of rna samples and an associated statistical analysis using non-linear regression. all four platforms are shown to have satisfactory precision but the commercial platforms are superior for resolving differential expression for genes at lower expression levels. an experiment is conducted to assess and compare four widely used microarray platforms. the results demonstrate good precision and sensitivity for all the platforms, but highlight the need for improved probe annotation. for probes which can be matched across platforms, the cross-platform variability is decomposed into within-platform and between-platform components, showing that platform disagreement is almost entirely systematic rather than due to measurement variability.
 concerns are often raised about the accuracy of microarray technologies and the degree of cross-platform agreement, but there are yet no methods which can unambiguously evaluate precision and sensitivity for these technologies on a whole-array basis.  a methodology is described for evaluating the precision and sensitivity of whole-genome gene expression technologies such as microarrays. the method consists of an easy-to-construct titration series of rna samples and an associated statistical analysis using non-linear regression. the method evaluates the precision and responsiveness of each microarray platform on a whole-array basis, i.e., using all the probes, without the need to match probes across platforms. an experiment is conducted to assess and compare four widely used microarray platforms. all four platforms are shown to have satisfactory precision but the commercial platforms are superior for resolving differential expression for genes at lower expression levels. the effective precision of the two-color platforms is improved by allowing for probe-specific dye-effects in the statistical model. the methodology is used to compare three data extraction algorithms for the affymetrix platforms, demonstrating poor performance for the commonly used proprietary algorithm relative to the other algorithms. for probes which can be matched across platforms, the cross-platform variability is decomposed into within-platform and between-platform components, showing that platform disagreement is almost entirely systematic rather than due to measurement variability.  the results demonstrate good precision and sensitivity for all the platforms, but highlight the need for improved probe annotation. they quantify the extent to which cross-platform measures can be expected to be less accurate than within-platform comparisons for predicting disease progression or outcome. 