BACKGROUND
dna microarray technology has rapidly seduced scientists and clinicians with the ability to simultaneously measure the expression of tens of thousands of transcripts, enabling data-driven, holistic comparisons of groups or populations of cells, subtyping tissues, or predicting prognosis  <cit> .
however, as with any method, sound experimental design is essential to generate robust results from microarray experiments, particularly given the issues of high dimensionality  <cit> .
sufficient care must be taken to identify and correct for sources of experimental bias alongside a cautious interpretation of the importance of reported differentially expressed genes  <cit> .
efforts to promote the routine formalisation and control of all stages of the experimental workflow have seen success and are increasingly promoted by journals and microarray data repositories  <cit> .
more recent work suggests the need for the inclusion of more detailed information concerning the statistical treatment of data in order for results to be independently validated post-publication  <cit> .
such standardisation is essential to researchers wishing to re-analyse published data or combine multiple datasets in a meta-analysis.
however the utility of these standards to the individual researcher gathering, analysing, and interpreting the data in the first instance is largely overlooked.
despite all efforts towards standardisation, it is still not possible to account for all potential sources of variation in the experiment workflow; identical experiments performed at different sites have produced significantly different results  <cit> .
inconsistencies between results generated using different microarray platforms  <cit>  or generations of array  <cit>  have been highlighted and multiplicative, systematic biases have been shown to be introduced at many stages of the experimental process, even when using a single array platform  <cit> .
the common practice of hybridising samples with no technical replication  is a result of the relatively high cost of arrays, the perceived improvement in array manufacturing quality, and the difficulties of obtaining sufficient amounts of high quality mrna from some clinical samples.
this practice is fundamentally reliant on the assumption that the intra-experiment variability is of a small enough magnitude not to undermine the power of the assay to resolve interesting biological differences that may exist between predefined groups of samples.
there is, however, mounting evidence  <cit>  to suggest that this assumption may be flawed and that the technical variation between replicate samples should not be ignored.
a large amount of effort has been expended in assessing the reliability, reproducibility, and compatibility of results generated by a number of array platforms within and between laboratory sites.
the microarray quality control  project, a us food and drug administration initiative  <cit> , explored that the intra- and inter-platform consistency of microarrays using two reference rna samples  from stratagene comprised of high-quality rna from a mixture of  <dig> different human cell-lines  and a human brain reference rna from ambion) and primary samples processed on six microarray platforms at three different sites.
the results of the maqc and other studies highlight the fact that, despite the generally good consensus between results, data generated from different platforms, in different laboratories, by different investigators can be negatively affected by dataset-wide batch variation in the reported expression levels  <cit> .
several methods that can remove these batch differences have been proposed, tested, and evaluated.
batch effects have been shown to be minimised with correction methods such as, singular value decomposition  <cit> , distance weighted discrimination  <cit> , mean-centring  <cit> , and combat  <cit> .
it is slowly becoming accepted that batch effects are to be expected when combining data generated across different labs, by different researchers, or using different platforms  <cit> .
there is a strong motivation to integrate multiple studies for meta-analyses that have increased statistical power afforded by larger sample-sizes, which can help to overcome basic limitations such as the inherent heterogeneity between biological subjects.
combined datasets can swell to include thousands of tumours and have been shown to lead to improved results and consensus findings  <cit> .
some researchers are now aware of bias arising due to analysis of samples at different sites or the use of different microarray platforms.
the maqc project  <cit> , for example, was a multi-site and multi-platform comparison study, while others deal exclusively with the integration of data generated at geographically distributed locations.
this study, to the best of our knowledge, is the first to assess the propensity for introduction of batch-processing effects at the same site and using the same protocol, making use of the multi-array illumina beadchip platform.
we go further than the maqc study by analysing both a commercial reference rna and primary clinical material.
this approach enabled us to demonstrate that it is possible to generate robust and reliable results, without the need for technical replication of starting rna, but only when batch-processing effects are identified and suitably minimised.
in this study we demonstrate compelling evidence for the existence of confounding batch-processing effects within a single experiment, using rna prepared in the same laboratory, arrays hybridised and scanned at a single site, using a single protocol, and quantified on a single platform.
we investigated intra-experiment batch-processing variability on the illumina beadchip  <cit>  platform, as multiple arrays on each chip allow an investigation of intra- and inter-run variation.
this was achieved through the hybridisation of a sample of uhrr to a single array on each chip along with duplicate preparations of crna from fresh frozen breast tumour samples that formed part of a recent clinical study .
intra-experiment variation is common in other assays, such as quantitative rt-pcr , where technical replicates and inter-plate calibrators are used to increase statistical resolution.
RESULTS
data quality
a qualitative measure of the performance of the beadchips used in this study is provided by a measurement of the fraction of probes that are consistently called to be detected or undetected over all arrays.
analysis of detection consistency in the uhrr data in this study was comparable with the maqc results  <cit>  with 60-70% probes consistently called all-detected, and 80-90% genes consistently called as either all-detected or all-undetected, across all arrays in each run .
the coefficient of variation  between and within the runs of the experiment was also consistent with the findings of the maqc study, with a mean cv in quantile normalised data of around  <dig> % .
the illumina arrays used throughout the maqc study were the human- <dig>  beadchips, which differ from the human- <dig>  beadchips used in this study in terms of the number of features represented.
the human- <dig>  chips contain twice the number of probesets available on human- <dig> , however a large percentage of these additional probesets have been found to be unreliable  <cit>  and are all contained within a completely separate strip on the chip leading to normalisation issues  <cit> .
the high level of agreement in the observed cv and detection calls suggest any differences between the array versions at the probe-level are small.
inter- and intra-run variation of the replicate uhrr samples
a clear batch-specific effect was observed in the raw data when the correlations of identical uhrr samples were assessed over all available pairs across the five runs processed on different days as illustrated in figure  <dig>  generally, the level of correlation was high , however several clusters of samples were observed that corresponded to the batch in which the arrays were processed.
in particular the samples in run  <dig> appeared to be very tightly correlated with each other but poorly correlated with samples in run  <dig> .
quantile normalisation was found to have only a marginal improvement in the overall correlation of the samples and anomalies, such as that between runs  <dig> and  <dig>  were conserved .
only on application of specialised batch-correction methods, such as mean-centring  and combat  <cit>  , were these run-specific disparities shown to be substantially reduced.
the correlations  for quantile normalised data ranged from  <dig> - <dig> , whereas following a combat correction this was increased to  <dig> - <dig> .
the probe-wise standard deviations of the raw expressions were found to be consistently small across the uhrr arrays .
using the nested analysis of variance described in methods, 60%  of the variability was due to that between runs and less than 40% to that within each run.
the magnitude of the variation was marginally increased by detection-filtering , which would be expected due to the preferential filtering of probes with low signal.
the application of quantile normalisation had a positive effect, decreasing the standard deviation to half that of the raw data.
however both after detection filtering and quantile normalisation the relative contributions of the inter- and intra-run components to the total standard deviation remained approximately unchanged.
of a selection of other normalisation methods, loess, and cubic-spline performed similarly to quantile and all of these methods out-performed simple median normalisation .
in all cases a further correction step is required after normalisation to correct for the batch effect.
both mean-centring and combat reduced inter-run variation to such an extent that it could no longer be accurately detected by the nested-anova method .
the only observable difference between the two methods was that the combat corrected data also showed a slight reduction in the intra-run component of variation .
the sequence in which the data were quantile-normalised and batch-corrected appeared to produce only marginal differences in the resulting variance components; as a result, all remaining corrections using mean-centring and combat were performed after quantile normalisation for consistency and to comply with the statistical assumptions of the latter  <cit> .
the differences in measured expression between all combinations of pairs of uhrr samples that straddled the five runs  were calculated for raw, quantile-normalised, mean-centred, and combat corrected data .
the distribution of differences in the raw data did not resemble the expected form of a gaussian centred at the origin; instead it was skewed towards the positive .
this was largely corrected after quantile normalisation and subsequent application of mean-centring and combat further narrowed the distribution reflecting the previously observed improvement in correlation.
similar improvements were observed in the differences between samples that were processed in the same run .
a full illustration of the intra-run pairwise differences can be found in additional file  <dig>
duplicate clinical breast-tumour samples
the sixty-three duplicate clinical samples provided a means to assess inter- and intra-run variation using samples more representative of those commonly analysed using microarray technology.
the differences in the measured expressions between each of the duplicate pairs of the clinical samples that straddled the five runs  were calculated for raw, quantile-normalised, mean-centred, and combat corrected data .
as with the uhrr samples, moderate differences were observed between the raw expressions of duplicate hybridisations and quantile normalisation was found to reduce, but not eliminate, the differences between the duplicate samples.
the distributions are similar to uhrr samples, although the raw data showed a clear positive skew that was again successively improved following quantile-normalisation, mean-centring, and combat, respectively.
for completeness, intra-run distribution of differences between the duplicate samples was assessed for the seven pairs of samples in run five .
pearson rank-products were calculated to assess the correlation between the duplicate samples.
as with the uhrr the clinical samples were generally very highly correlated , although the samples on beadchips 13/ <dig> and 14/ <dig> were found to be less similar than the others ; this is consistent with the effect observed in run  <dig> using the uhrr.
batch correction by either mean-centring or combat increased the correlation for all samples except for two arrays on beadchips 1/ <dig> in the first run and all arrays on beadchips 17/ <dig> in the final run.
comparing duplicate tumour samples as a repeated dataset to assess reproducibility of gene-lists
of the  <dig> duplicate, paired clinical samples obtained from matched-biopsies before and after treatment with the mtor inhibitor rad <dig>   <dig> were of sufficient quality to be used in an analysis to reveal differentially expressed genes .
using these samples we further assessed the impact of the intra-experiment variation in terms of the differences between lists of differentially expressed genes reported by each half of the duplicate samples.
the hybridisation plan for the  <dig> pairs of pre- and post-treatment samples in each duplicate-group is illustrated in figure 1; in the figure, triangles to the left of the sample represent pre-treatment samples and triangles to the right represent post-treatment samples.
the first hybridisation of each duplicate sample is represented by a trailing 'a' and the second represented by a trailing 'b'.
the 'a' and 'b' duplicate sample groups, containing the 'a' and 'b' hybridisations of each sample, respectively, were considered as two completely independent datasets  in order to assess the extent to which run-specific processing bias can influence the identification of differentially expressed genes.
these datasets were independently filtered by detection calls, quantile-normalised, and, where stated, batch corrected by mean-centering or combat before generating lists of differentially expressed genes.
two bioconductor packages, limma and siggenes, were used to perform the statistical analyses .
using the same stringency in the assessment of differential expression  and using quantile normalised data, many more probes were found to be differentially expressed between pre- and post-treatment samples in sample group a  than in group b .
following batch correction with combat the number of differentially expressed genes identified in the two groups was more consistent  and the overlap, in terms of probes reported in both groups, increased from just  <dig> % to  <dig> %, however the use of mean-centred data only increased the overlap marginally to  <dig> % .
differentially expressed genes were identified using limma and sam as described in the text with quantile-normalisation , mean-centring , and combat .
the uhrr was used as an inter-batch calibrator.
similar results were seen with less stringent criteria , which consequently led to larger numbers of probes, but similar proportions of overlapping probes were reported .
the analysis was repeated using significance analysis of microarrays  at a predicted false discovery rate of 5% and generated very similar results to those obtained using limma, increasing the overlap between groups of samples from just  <dig> % in quantile-normalised data to  <dig> % following combat batch-correction.
see table  <dig> for a full summary of these results.
the heatmaps in figure  <dig>  also highlight how all the pairs of duplicate samples cluster together following combat correction and the clustering is far less affected by processing runs.
the dependency between the choice of combat, mean-centring, or quantile normalisation on the number of genes identified as differentially expressed in each replicate group was very strong in both the limma and sam analyses  p-value <<  <dig> ).
in addition to these independent analyses, the a and b groups were combined to create a third group of samples, 'c'.
this group was analysed for differential expression in the same way and the results summarised in terms of the number of genes reported in any one, or any combination, of the three lists.
the percentage of genes consistently reported by limma as differentially expressed in all three groups after combat correction was  <dig> % compared to  <dig> % after quantile normalisation alone and  <dig> % after mean-centering .
the percentage of genes identified in the pooled group c compared with those consistently reported in all three groups increased from  <dig> % after quantile normalisation to  <dig> % after combat.
again, very similar results were observed using sam .
these analyses were repeated using the uhrr as inter-batch calibrator, designating it as a covariate in both the mean-centring and combat corrections.
the inclusion of uhrr during quantile normalisation produced only a small difference in the number of differentially expressed genes identified in each of the three sample groups.
however, the inclusion of the uhrr as a covariate in the mean-centring and combat corrections gave very different results.
in both methods there was a large reduction in the total number of genes reported in each list, in terms of the consensus between the a and b groups, the agreement dropped to  <dig> % following mean-centring, but increased to  <dig> % after correction by combat .
the dependency between choice of batch correction method and number of genes reported in either replicate group was stronger when uhrr was included in the correction in both the limma and sam analyses  p-value <<<  <dig> ).
discussion
batch-processing effects in microarray experiments are commonly encountered when combining datasets from different studies, different labs, or different technologies.
in this study we have demonstrated that batch effects can arise within a single study, at a single lab, using a single technology and that these can have a significant impact on reported gene-lists.
the magnitude of the variation in the observed expression of replicate samples derived from the uhrr in this study is consistent with that reported in other studies assessing the quality of microarray data, such as the maqc  <cit> .
we have also shown that the correlation of replicate uhrr samples is similar to that between duplicate pairs of samples derived from clinical breast-tissue biopsies and that this correlation is generally high.
however, when duplicate groups of clinical samples were independently analysed to identify differentially expressed genes the consistency of the resulting gene-lists was found to be very poor.
the predicted false discovery rate of 5% using sam was far lower than the observed proportion of genes that failed to be consistently reported over the replicate analyses .
whilst these two values are not directly equivalent, our results suggest that the predicted fdr may imply greater consistency than would be measured if duplicate samples are available.
specialised corrections for run-bias were more successful in reducing the magnitude of variability attributed to the inter-run batch effect in both uhrr and clinical samples.
the reliability of results generated from the duplicate clinical samples was also greatly increased following batch-correction with a much greater proportion of genes consistently reported as differentially expressed in both sets of samples.
use of single samples
there are many stages of sample-processing prior to conducting any gene-expression experiment and each is vulnerable to the introduction of systematic processing errors  <cit> .
opportunities to quantify this variation, prior to the microarray data analysis itself, are extremely limited and generally the only available option is an assessment of rna quality.
other methods of quantifying gene expression, that are equally susceptible to the introduction of processing error, rely on the use of technical replicates to minimise confounding variation and maximise statistical resolution to the biological processes under investigation  <cit> .
in this respect the routine practice of analysing each expression array sample as a singleton, regardless of the amount of rna loaded, is an unusual scientific approach.
whilst beadchip technology has a degree of built-in replication , this is no substitute for biological replicates, especially when a large degree of the observed error can be attributed to noise at the sample level, rather than at the probe level.
in the context of primary breast tumour samples, which have been repeatedly shown to have highly heterogeneous mrna expression profiles, there is much greater variation between the rna profiles from different individuals than within tumours  <cit> ; either when comparing different tumour sections, biopsies and the tumour or ffpe and frozen  <cit> , which effectively characterises the 'intrinsic profile' of subtype classification.
on this basis and the grounds of cost and scarcity of primary material it could be argued that replicates are unnecessary.
however a lack of replicates limits the investigator in terms of their ability to assess whether the observed variation is of biological or technical origin and the extent to which it influences the resulting gene-lists.
in this respect both biological and technical replicates are desirable to allow generated data to be screened for bias and batch-correction applied where appropriate.
this is particularly important in the clinical setting if samples for large trials are processed in multiple labs.
using the duplicate-experiment approach we were able to demonstrate that single samples can generate reliable data, particularly when batch correction is performed to minimise processing bias.
however the genes reported to be differentially expressed in the pooled duplicate samples in group c were more robust in terms of their agreement with those identified in groups a and b, especially following batch-correction.
use of uhrr controls
in addition to the technical replicates commonly used in other assays, in cases where the execution of the assay is split into several runs, it is very common for an inter-run calibrator to be used to quantify the variation introduced by the splitting of the experiment and to normalise for it.
despite the uhrr samples used in this study showing very similar variation and correlation to that previously reported, we found that the samples were of limited utility as predictors of the batch variation amongst the clinical samples.
however the replicate uhrr samples were found to slightly improve the consensus between the results of the duplicate experiments when used in conjunction with the combat correction.
although the uhrr has been reported to be useful as a standard for microarray experiments and suitable for monitoring the performance of genome-wide expression platforms  <cit> , it has also been reported to not be a suitable representative as a normal sample for colon epithelial rna  <cit> ; similarly, the uhrr does not contain breast tumour rna .
a more reliable control sample with which to improve the batch-correction might be provided by an mrna sample more representative of that under investigation; in this case, a pool of tumour rna rather than the uhrr.
we found that the pre-treatment samples were good predictors of the batch variation amongst the post-treatment samples and so would likely make a better control  than the uhrr.
experimental design
there is no reason to believe that the batch-processing effects observed here are limited to the illumina beadchip platform.
many previous investigations of other platforms have postulated potential factors responsible for the introduction of processing errors in microarray experiments  <cit> .
other experiments at our facility using the more recent illumina human ht- <dig> and mouse ref  <dig> beadchips exhibit similar batch effects to those in this study; samples are observed to cluster preferentially with others processed in the same run, rather than by the biological differences between them, even after quantile normalisation .
specialised batch-corrections appear to remove the bias, however without replicates such as those described in the current study, this cannot be fully evaluated.
regardless of the platform chosen, it is clear that compensation for processing variation is beneficial and can only be achieved by incorporating the design of the experiment into the downstream data analyses.
if all pre-treatment samples had been processed in one batch and all post-treatment samples in a second batch, it would not be possible to rule out confounding differences between treatment and batch processing.
'real' differences due to the common variable of interest may have been partially or completely obscured by the batch effect.
design oversights of this type are beginning to be highlighted  <cit>  and demonstrate the need to record the batches or processing runs in which data is generated.
some raw files contain metadata, such as the date in which they were generated, embedded within them.
acknowledgement and identification of the propensity for processing variation can be used to maximise the efficacy of batch-correction methods through a more informed design of the hybridisation-plan that includes, for example, randomisation and/or blocking of samples.
our results support the notion that analysis of gene expression data should begin with an evaluation of batch effects.
if the possibility of batch effects has been anticipated and confounding factors separated, then it should be possible to remove the bias to generate more robust results.
as with other studies, ours is limited by the samples that were used in the evaluation of processing variation.
we would have liked to test the applicability of our findings in other published datasets, however we were unable to find comparable datasets that include technical replicates and details of hybridisation 'batches' in the existing data repositories.
in terms of cost and practicalities it is understandable why most researchers do not perform replicates in clinical studies, our results indeed suggest they may not be necessary; however providing a hybridisation plan along with the raw data, would make the processing of data more transparent.
CONCLUSIONS
in summary, intra-experiment bias can distort the findings of gene expression studies.
replicate samples were found to be beneficial in both the identification and reduction of processing bias and lead to increased consensus in reported gene-lists, especially following specialised batch-corrections.
we conclude that single samples can generate reliable data, although an appreciation for sources of intra-experiment variation during the design of the experiment is required to maximise the efficacy of specialised corrections in order to minimise susceptibility to potentially confounding intra-experiment batch-effects.
finally, based on the discrepancy between the lists of differentially expressed genes in each group of duplicate tumour samples, the observed rate of falsely-reported genes was consistently and significantly larger than that predicted by sam.
therefore, based on the results of this study, a healthy degree of skepticism is advised when interpreting published results of microarray experiments that do not include validation by technical replication or, preferably, by another technique such as qpcr.
in the absence of large numbers of biological replicates, it is our opinion that technical replication should be encouraged in order to provide robust, reliable, and credible expression-profiles.

 dna microarray technology has rapidly seduced scientists and clinicians with the ability to simultaneously measure the expression of tens of thousands of transcripts, enabling data-driven, holistic comparisons of groups or populations of cells, subtyping tissues, or predicting prognosis. however, as with any method, sound experimental design is essential to generate robust results from microarray experiments, particularly given the issues of high dimensionality. sufficient care must be taken to identify and correct for sources of experimental bias alongside a cautious interpretation of the importance of reported differentially expressed genes. efforts to promote the routine formalisation and control of all stages of the experimental workflow have seen success and are increasingly promoted by journals and microarray data repositories. more recent work suggests the need for the inclusion of more detailed information concerning the statistical treatment of data in order for results to be independently validated post-publication. such standardisation is essential to researchers wishing to re-analyse published data or combine multiple datasets in a meta-analysis. however the utility of these standards to the individual researcher gathering, analysing, and interpreting the data in the first instance is largely overlooked. despite all efforts towards standardisation, it is still not possible to account for all potential sources of variation in the experiment workflow; identical experiments performed at different sites have produced significantly different results. inconsistencies between results generated using different microarray platforms or generations of array have been highlighted and multiplicative, systematic biases have been shown to be introduced at many stages of the experimental process, even when using a single array platform. the common practice of hybridising samples with no technical replication is a result of the relatively high cost of arrays, the perceived improvement in array manufacturing quality, and the difficulties of obtaining sufficient amounts of high quality mrna from some clinical samples. this practice is fundamentally reliant on the assumption that the intra-experiment variability is of a small enough magnitude not to undermine the power of the assay to resolve interesting biological differences that may exist between predefined groups of samples. there is, however, mounting evidence to suggest that this assumption may be flawed and that the technical variation between replicate samples should not be ignored. a large amount of effort has been expended in assessing the reliability, reproducibility, and compatibility of results generated by a number of array platforms within and between laboratory sites. the microarray quality control project, a us food and drug administration initiative, explored that the intra- and inter-platform consistency of microarrays using two reference rna samples from stratagene comprised of high-quality rna from a mixture of different human cell-lines and a human brain reference rna from ambion) and primary samples processed on six microarray platforms at three different sites. the results of the maqc and other studies highlight the fact that, despite the generally good consensus between results, data generated from different platforms, in different laboratories, by different investigators can be negatively affected by dataset-wide batch variation in the reported expression levels. several methods that can remove these batch differences have been proposed, tested, and evaluated. batch effects have been shown to be minimised with correction methods such as, singular value decomposition, distance weighted discrimination, mean-centring, and combat. it is slowly becoming accepted that batch effects are to be expected when combining data generated across different labs, by different researchers, or using different platforms. there is a strong motivation to integrate multiple studies for meta-analyses that have increased statistical power afforded by larger sample-sizes, which can help to overcome basic limitations such as the inherent heterogeneity between biological subjects. combined datasets can swell to include thousands of tumours and have been shown to lead to improved results and consensus findings. some researchers are now aware of bias arising due to analysis of samples at different sites or the use of different microarray platforms. the maqc project, for example, was a multi-site and multi-platform comparison study, while others deal exclusively with the integration of data generated at geographically distributed locations. this study, to the best of our knowledge, is the first to assess the propensity for introduction of batch-processing effects at the same site and using the same protocol, making use of the multi-array illumina beadchip platform. we go further than the maqc study by analysing both a commercial reference rna and primary clinical material. this approach enabled us to demonstrate that it is possible to generate robust and reliable results, without the need for technical replication of starting rna, but only when batch-processing effects are identified and suitably minimised. in this study we demonstrate compelling evidence for the existence of confounding batch-processing effects within a single experiment, using rna prepared in the same laboratory, arrays hybridised and scanned at a single site, using a single protocol, and quantified on a single platform. we investigated intra-experiment batch-processing variability on the illumina beadchip platform, as multiple arrays on each chip allow an investigation of intra- and inter-run variation. this was achieved through the hybridisation of a sample of uhrr to a single array on each chip along with duplicate preparations of crna from fresh frozen breast tumour samples that formed part of a recent clinical study. intra-experiment variation is common in other assays, such as quantitative rt-pcr, where technical replicates and inter-plate calibrators are used to increase statistical resolution.  data quality a qualitative measure of the performance of the beadchips used in this study is provided by a measurement of the fraction of probes that are consistently called to be detected or undetected over all arrays. analysis of detection consistency in the uhrr data in this study was comparable with the maqc results with 60-70% probes consistently called all-detected, and 80-90% genes consistently called as either all-detected or all-undetected, across all arrays in each run. the coefficient of variation between and within the runs of the experiment was also consistent with the findings of the maqc study, with a mean cv in quantile normalised data of around %. the illumina arrays used throughout the maqc study were the human- beadchips, which differ from the human- beadchips used in this study in terms of the number of features represented. the human- chips contain twice the number of probesets available on human-, however a large percentage of these additional probesets have been found to be unreliable and are all contained within a completely separate strip on the chip leading to normalisation issues. the high level of agreement in the observed cv and detection calls suggest any differences between the array versions at the probe-level are small. inter- and intra-run variation of the replicate uhrr samples a clear batch-specific effect was observed in the raw data when the correlations of identical uhrr samples were assessed over all available pairs across the five runs processed on different days as illustrated in figure generally, the level of correlation was high, however several clusters of samples were observed that corresponded to the batch in which the arrays were processed. in particular the samples in run appeared to be very tightly correlated with each other but poorly correlated with samples in run. quantile normalisation was found to have only a marginal improvement in the overall correlation of the samples and anomalies, such as that between runs and were conserved. only on application of specialised batch-correction methods, such as mean-centring and combat, were these run-specific disparities shown to be substantially reduced. the correlations for quantile normalised data ranged from -, whereas following a combat correction this was increased to -. the probe-wise standard deviations of the raw expressions were found to be consistently small across the uhrr arrays. using the nested analysis of variance described in methods, 60% of the variability was due to that between runs and less than 40% to that within each run. the magnitude of the variation was marginally increased by detection-filtering, which would be expected due to the preferential filtering of probes with low signal. the application of quantile normalisation had a positive effect, decreasing the standard deviation to half that of the raw data. however both after detection filtering and quantile normalisation the relative contributions of the inter- and intra-run components to the total standard deviation remained approximately unchanged. of a selection of other normalisation methods, loess, and cubic-spline performed similarly to quantile and all of these methods out-performed simple median normalisation. in all cases a further correction step is required after normalisation to correct for the batch effect. both mean-centring and combat reduced inter-run variation to such an extent that it could no longer be accurately detected by the nested-anova method. the only observable difference between the two methods was that the combat corrected data also showed a slight reduction in the intra-run component of variation. the sequence in which the data were quantile-normalised and batch-corrected appeared to produce only marginal differences in the resulting variance components; as a result, all remaining corrections using mean-centring and combat were performed after quantile normalisation for consistency and to comply with the statistical assumptions of the latter. the differences in measured expression between all combinations of pairs of uhrr samples that straddled the five runs were calculated for raw, quantile-normalised, mean-centred, and combat corrected data. the distribution of differences in the raw data did not resemble the expected form of a gaussian centred at the origin; instead it was skewed towards the positive. this was largely corrected after quantile normalisation and subsequent application of mean-centring and combat further narrowed the distribution reflecting the previously observed improvement in correlation. similar improvements were observed in the differences between samples that were processed in the same run. a full illustration of the intra-run pairwise differences can be found in additional file duplicate clinical breast-tumour samples the sixty-three duplicate clinical samples provided a means to assess inter- and intra-run variation using samples more representative of those commonly analysed using microarray technology. the differences in the measured expressions between each of the duplicate pairs of the clinical samples that straddled the five runs were calculated for raw, quantile-normalised, mean-centred, and combat corrected data. as with the uhrr samples, moderate differences were observed between the raw expressions of duplicate hybridisations and quantile normalisation was found to reduce, but not eliminate, the differences between the duplicate samples. the distributions are similar to uhrr samples, although the raw data showed a clear positive skew that was again successively improved following quantile-normalisation, mean-centring, and combat, respectively. for completeness, intra-run distribution of differences between the duplicate samples was assessed for the seven pairs of samples in run five. pearson rank-products were calculated to assess the correlation between the duplicate samples. as with the uhrr the clinical samples were generally very highly correlated, although the samples on beadchips 13/ and 14/ were found to be less similar than the others; this is consistent with the effect observed in run using the uhrr. batch correction by either mean-centring or combat increased the correlation for all samples except for two arrays on beadchips 1/ in the first run and all arrays on beadchips 17/ in the final run. comparing duplicate tumour samples as a repeated dataset to assess reproducibility of gene-lists of the duplicate, paired clinical samples obtained from matched-biopsies before and after treatment with the mtor inhibitor rad were of sufficient quality to be used in an analysis to reveal differentially expressed genes. using these samples we further assessed the impact of the intra-experiment variation in terms of the differences between lists of differentially expressed genes reported by each half of the duplicate samples. the hybridisation plan for the pairs of pre- and post-treatment samples in each duplicate-group is illustrated in figure 1; in the figure, triangles to the left of the sample represent pre-treatment samples and triangles to the right represent post-treatment samples. the first hybridisation of each duplicate sample is represented by a trailing'a' and the second represented by a trailing'b'. the'a' and'b' duplicate sample groups, containing the'a' and'b' hybridisations of each sample, respectively, were considered as two completely independent datasets in order to assess the extent to which run-specific processing bias can influence the identification of differentially expressed genes. these datasets were independently filtered by detection calls, quantile-normalised, and, where stated, batch corrected by mean-centering or combat before generating lists of differentially expressed genes. two bioconductor packages, limma and siggenes, were used to perform the statistical analyses. using the same stringency in the assessment of differential expression and using quantile normalised data, many more probes were found to be differentially expressed between pre- and post-treatment samples in sample group a than in group b. following batch correction with combat the number of differentially expressed genes identified in the two groups was more consistent and the overlap, in terms of probes reported in both groups, increased from just % to %, however the use of mean-centred data only increased the overlap marginally to %. differentially expressed genes were identified using limma and sam as described in the text with quantile-normalisation, mean-centring, and combat. the uhrr was used as an inter-batch calibrator. similar results were seen with less stringent criteria, which consequently led to larger numbers of probes, but similar proportions of overlapping probes were reported. the analysis was repeated using significance analysis of microarrays at a predicted false discovery rate of 5% and generated very similar results to those obtained using limma, increasing the overlap between groups of samples from just % in quantile-normalised data to % following combat batch-correction. see table for a full summary of these results. the heatmaps in figure also highlight how all the pairs of duplicate samples cluster together following combat correction and the clustering is far less affected by processing runs. the dependency between the choice of combat, mean-centring, or quantile normalisation on the number of genes identified as differentially expressed in each replicate group was very strong in both the limma and sam analyses p-value << ). in addition to these independent analyses, the a and b groups were combined to create a third group of samples,'c'. this group was analysed for differential expression in the same way and the results summarised in terms of the number of genes reported in any one, or any combination, of the three lists. the percentage of genes consistently reported by limma as differentially expressed in all three groups after combat correction was % compared to % after quantile normalisation alone and % after mean-centering. the percentage of genes identified in the pooled group c compared with those consistently reported in all three groups increased from % after quantile normalisation to % after combat. again, very similar results were observed using sam. these analyses were repeated using the uhrr as inter-batch calibrator, designating it as a covariate in both the mean-centring and combat corrections. the inclusion of uhrr during quantile normalisation produced only a small difference in the number of differentially expressed genes identified in each of the three sample groups. however, the inclusion of the uhrr as a covariate in the mean-centring and combat corrections gave very different results. in both methods there was a large reduction in the total number of genes reported in each list, in terms of the consensus between the a and b groups, the agreement dropped to % following mean-centring, but increased to % after correction by combat. the dependency between choice of batch correction method and number of genes reported in either replicate group was stronger when uhrr was included in the correction in both the limma and sam analyses p-value <<< ). discussion batch-processing effects in microarray experiments are commonly encountered when combining datasets from different studies, different labs, or different technologies. in this study we have demonstrated that batch effects can arise within a single study, at a single lab, using a single technology and that these can have a significant impact on reported gene-lists. the magnitude of the variation in the observed expression of replicate samples derived from the uhrr in this study is consistent with that reported in other studies assessing the quality of microarray data, such as the maqc. we have also shown that the correlation of replicate uhrr samples is similar to that between duplicate pairs of samples derived from clinical breast-tissue biopsies and that this correlation is generally high. however, when duplicate groups of clinical samples were independently analysed to identify differentially expressed genes the consistency of the resulting gene-lists was found to be very poor. the predicted false discovery rate of 5% using sam was far lower than the observed proportion of genes that failed to be consistently reported over the replicate analyses. whilst these two values are not directly equivalent, our results suggest that the predicted fdr may imply greater consistency than would be measured if duplicate samples are available. specialised corrections for run-bias were more successful in reducing the magnitude of variability attributed to the inter-run batch effect in both uhrr and clinical samples. the reliability of results generated from the duplicate clinical samples was also greatly increased following batch-correction with a much greater proportion of genes consistently reported as differentially expressed in both sets of samples. use of single samples there are many stages of sample-processing prior to conducting any gene-expression experiment and each is vulnerable to the introduction of systematic processing errors. opportunities to quantify this variation, prior to the microarray data analysis itself, are extremely limited and generally the only available option is an assessment of rna quality. other methods of quantifying gene expression, that are equally susceptible to the introduction of processing error, rely on the use of technical replicates to minimise confounding variation and maximise statistical resolution to the biological processes under investigation. in this respect the routine practice of analysing each expression array sample as a singleton, regardless of the amount of rna loaded, is an unusual scientific approach. whilst beadchip technology has a degree of built-in replication, this is no substitute for biological replicates, especially when a large degree of the observed error can be attributed to noise at the sample level, rather than at the probe level. in the context of primary breast tumour samples, which have been repeatedly shown to have highly heterogeneous mrna expression profiles, there is much greater variation between the rna profiles from different individuals than within tumours; either when comparing different tumour sections, biopsies and the tumour or ffpe and frozen, which effectively characterises the'intrinsic profile' of subtype classification. on this basis and the grounds of cost and scarcity of primary material it could be argued that replicates are unnecessary. however a lack of replicates limits the investigator in terms of their ability to assess whether the observed variation is of biological or technical origin and the extent to which it influences the resulting gene-lists. in this respect both biological and technical replicates are desirable to allow generated data to be screened for bias and batch-correction applied where appropriate. this is particularly important in the clinical setting if samples for large trials are processed in multiple labs. using the duplicate-experiment approach we were able to demonstrate that single samples can generate reliable data, particularly when batch correction is performed to minimise processing bias. however the genes reported to be differentially expressed in the pooled duplicate samples in group c were more robust in terms of their agreement with those identified in groups a and b, especially following batch-correction. use of uhrr controls in addition to the technical replicates commonly used in other assays, in cases where the execution of the assay is split into several runs, it is very common for an inter-run calibrator to be used to quantify the variation introduced by the splitting of the experiment and to normalise for it. despite the uhrr samples used in this study showing very similar variation and correlation to that previously reported, we found that the samples were of limited utility as predictors of the batch variation amongst the clinical samples. however the replicate uhrr samples were found to slightly improve the consensus between the results of the duplicate experiments when used in conjunction with the combat correction. although the uhrr has been reported to be useful as a standard for microarray experiments and suitable for monitoring the performance of genome-wide expression platforms, it has also been reported to not be a suitable representative as a normal sample for colon epithelial rna; similarly, the uhrr does not contain breast tumour rna. a more reliable control sample with which to improve the batch-correction might be provided by an mrna sample more representative of that under investigation; in this case, a pool of tumour rna rather than the uhrr. we found that the pre-treatment samples were good predictors of the batch variation amongst the post-treatment samples and so would likely make a better control than the uhrr. experimental design there is no reason to believe that the batch-processing effects observed here are limited to the illumina beadchip platform. many previous investigations of other platforms have postulated potential factors responsible for the introduction of processing errors in microarray experiments. other experiments at our facility using the more recent illumina human ht- and mouse ref beadchips exhibit similar batch effects to those in this study; samples are observed to cluster preferentially with others processed in the same run, rather than by the biological differences between them, even after quantile normalisation. specialised batch-corrections appear to remove the bias, however without replicates such as those described in the current study, this cannot be fully evaluated. regardless of the platform chosen, it is clear that compensation for processing variation is beneficial and can only be achieved by incorporating the design of the experiment into the downstream data analyses. if all pre-treatment samples had been processed in one batch and all post-treatment samples in a second batch, it would not be possible to rule out confounding differences between treatment and batch processing.'real' differences due to the common variable of interest may have been partially or completely obscured by the batch effect. design oversights of this type are beginning to be highlighted and demonstrate the need to record the batches or processing runs in which data is generated. some raw files contain metadata, such as the date in which they were generated, embedded within them. acknowledgement and identification of the propensity for processing variation can be used to maximise the efficacy of batch-correction methods through a more informed design of the hybridisation-plan that includes, for example, randomisation and/or blocking of samples. our results support the notion that analysis of gene expression data should begin with an evaluation of batch effects. if the possibility of batch effects has been anticipated and confounding factors separated, then it should be possible to remove the bias to generate more robust results. as with other studies, ours is limited by the samples that were used in the evaluation of processing variation. we would have liked to test the applicability of our findings in other published datasets, however we were unable to find comparable datasets that include technical replicates and details of hybridisation'batches' in the existing data repositories. in terms of cost and practicalities it is understandable why most researchers do not perform replicates in clinical studies, our results indeed suggest they may not be necessary; however providing a hybridisation plan along with the raw data, would make the processing of data more transparent.  in summary, intra-experiment bias can distort the findings of gene expression studies. replicate samples were found to be beneficial in both the identification and reduction of processing bias and lead to increased consensus in reported gene-lists, especially following specialised batch-corrections. we conclude that single samples can generate reliable data, although an appreciation for sources of intra-experiment variation during the design of the experiment is required to maximise the efficacy of specialised corrections in order to minimise susceptibility to potentially confounding intra-experiment batch-effects. finally, based on the discrepancy between the lists of differentially expressed genes in each group of duplicate tumour samples, the observed rate of falsely-reported genes was consistently and significantly larger than that predicted by sam. therefore, based on the results of this study, a healthy degree of skepticism is advised when interpreting published results of microarray experiments that do not include validation by technical replication or, preferably, by another technique such as qpcr. in the absence of large numbers of biological replicates, it is our opinion that technical replication should be encouraged in order to provide robust, reliable, and credible expression-profiles. 
correlation between replicate uhrr samples improved by two orders of magnitude following batch-correction using combat and increased the consistency of the gene-lists from the duplicate clinical samples, from % in quantile normalised data to % in batch-corrected data. the use of uhrr as an inter-batch calibrator provided a small additional benefit when used in conjunction with combat, further increasing the agreement between the two gene-lists, up to %. microarray technology is a popular means of producing whole genome transcriptional profiles, however high cost and scarcity of mrna has led many studies to be conducted based on the analysis of single samples. we exploit the design of the illumina platform, specifically multiple arrays on each chip, to evaluate intra-experiment technical variation using repeated hybridisations of universal human reference rna and duplicate hybridisations of primary breast tumour samples from a clinical study. a clear batch-specific bias was detected in the measured expressions of both the uhrr and clinical samples. however, when mean-centering or empirical bayes batch-correction methods were applied to the data, inter-batch variation in the uhrr and clinical samples were greatly reduced. in the interests of practicalities and cost, these results suggest that single samples can generate reliable data, but only after careful compensation for technical bias in the experiment.
 microarray technology is a popular means of producing whole genome transcriptional profiles, however high cost and scarcity of mrna has led many studies to be conducted based on the analysis of single samples. we exploit the design of the illumina platform, specifically multiple arrays on each chip, to evaluate intra-experiment technical variation using repeated hybridisations of universal human reference rna and duplicate hybridisations of primary breast tumour samples from a clinical study.  a clear batch-specific bias was detected in the measured expressions of both the uhrr and clinical samples. this bias was found to persist following standard microarray normalisation techniques. however, when mean-centering or empirical bayes batch-correction methods were applied to the data, inter-batch variation in the uhrr and clinical samples were greatly reduced. correlation between replicate uhrr samples improved by two orders of magnitude following batch-correction using combat and increased the consistency of the gene-lists from the duplicate clinical samples, from % in quantile normalised data to % in batch-corrected data. the use of uhrr as an inter-batch calibrator provided a small additional benefit when used in conjunction with combat, further increasing the agreement between the two gene-lists, up to %.  in the interests of practicalities and cost, these results suggest that single samples can generate reliable data, but only after careful compensation for technical bias in the experiment. we recommend that investigators appreciate the propensity for such variation in the design stages of a microarray experiment and that the use of suitable correction methods become routine during the statistical analysis of the data. 