BACKGROUND
dna microarray technology provides a powerful tool for characterizing gene expression on a genome scale.
while the technology has been widely used in discovery-based medical and basic biological research, its direct application in clinical practice and regulatory decision-making has been questioned  <cit> .
a few key issues, including the reproducibility, reliability, compatibility and standardization of microarray analysis and results, must be critically addressed before any routine usage of microarrays in clinical laboratory and regulated areas can occur.
considerable effort has been dedicated to investigate these important issues, most of which focused on the compatibility across different laboratories and analytical methods, as well as the correlation between different microarray platforms.
in this study we investigate some of these issues using the applied biosystems human genome survey microarrays.
the microarrays contain  <dig>  60-mer oligonucleotide probes representing  <dig>  individual human genes, and uses chemiluminescence  to identify and measure gene expression levels in cells and tissues.
in addition to the unique 60-mer probe, an internal control probe  is co-spotted with the 60-mer probe on the microarray and labeled with a complementary oligo containing the fluorescent liz® dye  during the hybridization of the microarray.
in this study, we analyzed the gene expression profiles of two human tissues: brain and universal human reference sample .
five technical replicates in three different sites were performed on the same total rna samples according to manufacturer's standard protocols.
five different methods, quantile  <cit> , median  <cit> , scale <cit> , vsn  <cit>  and cyclic loess  <cit>  were used to normalize ab microarray data within each site.
since fold change and variance dependency with intensity is platform dependent  <cit>  we were interested in evaluating the performance of these methods applied to ab microarray data, making this study the first one from this perspective.
we restricted our attention on these five methods for the following reasons.
these methods are most frequently used normalization methods for ab microarray data.
in addition, the microarrays used in this study contain one probe for each gene , this design restricting the number of normalization methods to be used and making methods based on replicated measurements for each gene  inapplicable.
other normalization methods that would also be inapplicable include those explicitly developed for two color technology, or replicated measurements.
<dig>  genes spanning a wide dynamic range in gene expression levels were selected for real-time pcr validation.
using the taqman® assays data as the reference set, the performance of the five normalization methods was evaluated focusing on the following criteria:  sensitivity and reproducibility in detection of expression;  fold change correlation with real-time pcr data;  sensitivity and specificity in detection of differential expression;  reproducibility of differentially expressed gene lists.
the data set analyzed in this manuscript has been reported elsewhere  <cit>  and made publicly available via geo accession number gse <dig> using the platform gpl  <dig> for taqman® assays data and gpl  <dig> for applied biosystems human genome survey microarrays data.
RESULTS
target selection for real-time pcr validation
in order to conduct a comprehensive survey of the arrays' performance, gene targets for real-time pcr validation were selected based on the following criteria:  ensure a large enough number of validation targets to provide representative overviews of the microarray performance;  select genes spanning a wide range of expression levels and  fold changes .
<dig> taqman® gene expression assays were used in this study, covering  <dig> genes  .
over 90% of these genes were selected from a subset of  <dig>  refseq common to the various microarray platforms .
this selection was designed so that the genes would cover the entire intensity and fold change ranges and include any bias due to refseq itself.
a subset of  genes were included based on tissue-specificity .
sensitivity and reproducibility in detection of expression
the dynamic range for the ab microarray platform spans 3– <dig> orders of magnitude  <cit> , while taqman based real-time pcr can achieve 7– <dig> orders of magnitude dynamic range  <cit> .
the larger dynamic range imparts taqman® assays with higher detection sensitivity ; we therefore used the taqman® assays data set as the reference set to evaluate the performance of microarrays in terms of detection sensitivity and accuracy.
first, genes that are detectable  and not detectable  were determined for each sample according to manufacturer's recommendations .
figure  <dig> shows the relationship between percent genes detected by the microarrays out of the ones detected by taqman® assays as a function of ct measurement .
gene expression levels were ordered according to taqman® assay measurements .
a sliding window containing  <dig> consecutive genes was constructed and moved one gene at a time to cover the whole range of ct values.
within each sliding window, the percent of genes detected as present in at least half of the replicates of individual samples by the microarray platform was computed out the total of those detected by taqman® assays, and plotted as a function of mean ct value of the  <dig> genes in the given window.
the overall sensitivity ) and specificity ), are presented in table  <dig>  and are  <dig> % and  <dig> %, respectively in the uhr sample measured in test site  <dig>
for each platform, sample and site, a gene is declared detected  if it is detected according to the platform specifications, in more than half of the replicates.
concordance between detection calls are presented in these tables for uhr and brain samples in site  <dig> .
for each normalization method, the impact on signal level was determined using genes detected by taqman® assays .
there are almost no differences in the signal levels when each of these normalization methods is used with the exception of vsn which results in a small increase in signal level for low expressing genes.
we also used coefficients of variation  of log <dig> to evaluate the effect of the  <dig> normalization methods on signal reproducibility, both within and between sites .
reproducibility of technical replicates for the five normalization approaches for site  <dig> is illustrated in figure  <dig> for both brain and uhr samples.
panel a, where all  <dig>  genes are represented, shows the coefficient of variation  across the  <dig> technical replicates, as a function of expression level when the data is normalized using the quantile approach.
panel b shows the coefficients of variation, only for genes with taqman® assays as a function of taqman ct. lines represent the lowess smoothing fitting curves  <cit>  of all data points from each normalization method.
as expected, cv's showed a strong dependence on expression level, decreasing from 10% for low expressers or absent genes, to 1% for high expressers.
all normalization methods improved the coefficient of variation observed in the raw data over the entire range of expression levels.
a small improvement in reproducibility for genes expressed at lower levels was observed in vsn normalization, in both representations, and for both samples.
signal reproducibility between the  <dig> testing sites is represented in figure  <dig>  within sites cvs  and between sites cvs  of all  <dig>  genes, show similar trends for all five normalization methods.
the vsn normalization which showed some improvement in within sites variability of low expression level genes performed similarly to the other normalization methods when between sites variability is considered.
fold change concordance with taqman® assays
to evaluate the concordance of fold changes between microarray and real-time pcr data, we performed regression analysis of fold differences between the uhr sample  and brain sample .
fold change metrics are more meaningful as they tend to cancel out systematic platform biases in absolute signal values, moreover they are more biologically relevant.
fold change  was computed as the difference in mean expression level of the five technical replicates measured within each site for each sample.
genes were filtered based on real-time pcr detection thresholds  and only genes detected in both samples  were used.
fold changes between brain and uhr samples were estimated and plots between  fold changes determined from ab microarray data and taqman® assays , are presented in figure  <dig>  the linear regression fitting curve for all data points was performed for each scatter plot .
the r <dig>  slope and intercept for taqman® assays versus microarray data are presented in table  <dig>  the  <dig> normalization methods used for the microarrays showed similar linear correlation characteristics with real-time pcr measurements with r <dig> values ranging between  <dig> – <dig>  and slopes ranging from  <dig>  to  <dig>  suggesting the existence of some ratio compression in the microarray data.
vsn showed slightly higher compression than the other normalization methods, most likely caused by the higher signal levels they impart for low expressers .
a similar fold-change comparison between the five normalization methods and taqman® assays was also performed using lowess smoothing , which does not assume a linear relationship of fold-change values between platforms.
the estimated range of fold changes , for taqman assays, is from - <dig> to  <dig>  while for ab microarrays is - <dig> to  <dig>  in order to better understand the cause of fold change compression additional analysis was performed on genes with similar expression levels in the two samples.
genes were binned into low/medium/high according to taqman® assays ct measurements .
only genes having expression level in the same bin in both samples a and b are included.
boxplots of the fold change for each normalization method and taqman® assays are presented in figure  <dig>  the range of fold changes measured by the microarrays is significantly lower for low expression level genes and somewhat lower for high expression level genes when compared to the range of fold changes measured by taqman® assays.
for medium expression level genes, the two platforms have a higher agreement on the magnitude of fold differences between the  <dig> samples.
sensitivity and specificity in detection of differential expression
we evaluated the performance of the different normalization methods in detecting differential expression between the two samples using multiple statistical approaches.
traditionally, analysis of accuracy is carried out by analyzing the true positive rate  and false discovery rate .
in this case, the actual rates are unknown.
for this reason, we used taqman® as the reference platform.
only genes detected by taqman® assays in both samples were used for this section.
using the assay calls as the reference, we constructed contingency tables against microarray data, in which the concordance was determined and both the p-value significance of the t-test controlling false discovery rate  at 5% level  <cit>  , and fold-change directionality  were taken into consideration.
specifically, true positives  are genes differentially expressed  in both taqman® and microarray platforms with similar direction of the fold change; true negatives  are genes not differentially expressed in both platforms; false positives , consist of two sets of genes:  genes not differentially expressed in taqman® and differentially expressed in microarrays, or  genes differentially expressed in both platforms with opposite fold change direction; false negatives , genes differentially expressed for taqman and not for microarrays.
genes were first ranked according to their average ct value in uhr and brain samples.
for each bin of  <dig> consecutive genes , we compare the results from each normalization method with the ones obtained with the taqman® assays.
tpr defined as tpr = tp/, represents the percentage of genes detected differentially expressed in microarray data out of the ones detected by taqman® assays.
fdr was defined as fp/, and represents the percentage of differentially expressed genes detected only by the microarrays out of all genes differentially expressed in microarrays.
as shown in figure  <dig>  at the highest expression level, all normalization methods displayed reasonably good sensitivities: 70–75% tpr; the performance drops as the expression level decreases, and at the lowest expression level, the tpr is ~20% .
the opposite trend can be seen for the fdr plots : a relative constant level of false findings  was observed for genes with high and medium expression levels , after which fdr increases up to 25–30% for genes with low expression level.
the overall accuracy of each of the five normalization methods is presented in table  <dig>  tpr and fdr indicate that the performance of the microarray platform is not dependent on the normalization method used.
the observed fdr of  <dig> % is slightly higher than the one expected from the fdr control we used to select differentially expressed genes , the 2% overestimate  being possibly explained by genes incorrectly called differentially expressed by the taqman® assays .
genes detected in both samples by taqman® assays  are used for this comparison.
t-test is used to detect significantly differentially expressed genes, controlling fdr at 5% level.
we compare the results from each normalization method, for all sites, with the ones from taqman® assays keeping track of up/down regulation in each platform.
additionally, we investigated four commonly used methods for identification of differentially expressed genes in microarray data: simple t-test , t-test combined with fold-change , t-test with fdr and fc control  and sam  <cit>  , to determine their impact on the detection of differentially expressed genes.
only data from site  <dig> was used for this case.
figure  <dig> shows tpr and fdr plots comparing genes differentially expressed for each statistical method applied to each normalization approach with taqman® data used as a reference as previously described .
table  <dig> presents the overall concordance for quantile normalized data.
the performance of the five methods does not change with the normalization method used.
it is important, however, when interpreting these results to take into consideration one particularity of this experiment.
the two samples compared, human brain and uhr being extremely divergent tissue types, display big differences in gene expression levels, and so the vast majority of genes used  showed significant changes .
from this perspective, it is expected that fdr control, for genes found differentially expressed by microarrays, will have little impact on fp rates.
this explains why the results obtained from the simple t-test and the t-test with fdr control are very similar.
in a previous comparative study  <cit> ) where samples with smaller differences were used, we have seen that these two methods show bigger differences in specificity.
on the other hand, restrictions on the magnitude of fold change reduce both the number of true positives and false positives.
finally, sam produces even more specific results  penalizing some of the low expressers that the taqman® assays find differentially expressed.
we use different methods to detect significantly differentially expressed genes for data quantile normalized:  t-test ,  t-test controlling fdr at 5% level,  t-test  and fc <  <dig> ,  t-test controlling fdr at 5% level and fc <  <dig> , or  sam q <  <dig> .
we compare the results for data generated by site  <dig>  from each normalization method, with the ones from taqman® assays for which differential expression is detected using t-test and controlling fdr at 5% level.
we keep track of up/down regulation in each platform.
reproducibility of differentially expressed gene lists
a fundamental step in most microarray experiments is determining lists of differentially expressed genes that distinguish biological conditions.
reproducibility of differentially expressed genes across highly similar experiments is one of the important aspects of assessing reliability of microarray results .
we used percentage of overlapping genes  between differentially expressed genes lists as the measure of reproducibility  <cit> .
for each testing site and each normalization method, we declared genes differentially expressed again using t-test and controlling fdr at 5% level.
in this way, for each site we generated a list of differentially expressed genes.
figure  <dig> shows the overlap between these lists of genes for each normalization method.
table  <dig> summarizes both percentages and counts of gene overlapping between either pairs of sites or all three sites.
one can see that the pog obtained from different normalization methods are similar, ranging from  <dig> %  to  <dig> %  when all three sites are compared.
site  <dig> shows some differences compared to the other two sites, while the comparison between sites  <dig> and  <dig> shows consistently 83% pog.
very similar results were observed when the other  <dig> statistical methods were used for generating gene lists for microarray data .
it is also important to note that no genes showed discordant results between brain and uhr, i.e.
a significant fold change in opposite direction, when different normalization methods were used.
we use t-test controlling fdr at 5% level within each site to detect significantly differentially expressed genes for normalized data.
we compare the results for each normalization method, from the  <dig> sites.
first table contains percentage of concordant genes between sites.
second table contains the counts of genes differentially expressed.
discussion
one unanswered question in the microarray field has always been the effect of various normalization as well as statistical methods on the end results of a profiling experiment and more explicitly whether using different normalization or statistical approaches results in different gene lists of less concordance between different microarray platforms.
in this study we have assessed the performance of five different normalization methods using the applied biosystems expression array system.
our results show a high level of concordance between these normalization methods.
this is true, regardless of whether signals, variation or fold change measurements were interrogated.
in addition, these five normalization methods showed similar performance of signal reproducibility between the three testing sites used for this study.
furthermore, we used taqman® assays as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range .
tpr was directly correlated to gene expression levels whereas fdr was inversely correlated.
this is not completely surprising as the two platforms have different dynamic ranges and sensitivity levels, with the detection levels of the microarrays being lower than those of taqman® assays.
these differences more than likely explain the lower tp rates and higher fp rates for the genes at the low expression levels.
these effects were also observed for several other microarray platforms in a separate study  <cit> .
one conclusion of this study is that, at least for the microarray platform tested in this study, the current normalization approaches have little impact on the signal, detection levels as well as tp and fp rates in detection of differentially expressed genes.
these results are consistent with the findings of the maqc study .
in addition we also explored the contribution of several statistical approaches commonly used in the field on the tp and fp rates.
as expected in this case, with approaches which relax the stringency in differential expression, better detection and differential expression concordance is observed, concomitant with a higher percentage of false positives.
at the opposite end of the spectrum, fdr control and sam methods, which are more restrictive in detection of differential expression, produce gene lists with fewer false positives.
sam, as expected, shows a reduced number of false positives for low expressers, at the expense of missing some differentially expressed genes.
the expected percentage of false positives in these lists is close to the one observed when comparing results to taqman® assays.
unfortunately, it seems that the full strength of these statistical methods is obscured by the fact that the majority of the genes chosen for taqman® validation show significant fold changes between samples, minimizing the effect of fdr on the fp rate.
more importantly however, applying the different normalization approaches to the various statistical methods tried, had no significant impact on identifying differentially expressed genes.
finally, when comparing the overlap in gene lists generated by each of these statistical methods, a concordance of  <dig> – <dig> % was observed between all three sites, and  <dig> – <dig> % between sites  <dig> and  <dig>  indicating little effect of the analysis approach used on the final gene list obtained.
this result is, however, sensitive to the cut-offs used in determining the gene lists and can affect the degree of overlap observed  <cit> .
we were pleasantly surprised, however, of the little effect observed by the various normalization on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field.
CONCLUSIONS
in this study we have assessed the performance of five different normalization methods using data generated with the applied biosystems expression array system.
our results show a high level of concordance between these normalization methods.
this is true, regardless of whether signals, variation, site reproducibility or fold change measurements were interrogated.
the same similarity is observed when taqman® assays were used as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range.
in addition we also explored the contribution of several statistical approaches commonly used in the field on the detection of differential expression.
little effect is observed by the various normalization methods on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field, particularly when used in conjunction with the applied biosystems microarrays.

 dna microarray technology provides a powerful tool for characterizing gene expression on a genome scale. while the technology has been widely used in discovery-based medical and basic biological research, its direct application in clinical practice and regulatory decision-making has been questioned. a few key issues, including the reproducibility, reliability, compatibility and standardization of microarray analysis and results, must be critically addressed before any routine usage of microarrays in clinical laboratory and regulated areas can occur. considerable effort has been dedicated to investigate these important issues, most of which focused on the compatibility across different laboratories and analytical methods, as well as the correlation between different microarray platforms. in this study we investigate some of these issues using the applied biosystems human genome survey microarrays. the microarrays contain 60-mer oligonucleotide probes representing individual human genes, and uses chemiluminescence to identify and measure gene expression levels in cells and tissues. in addition to the unique 60-mer probe, an internal control probe is co-spotted with the 60-mer probe on the microarray and labeled with a complementary oligo containing the fluorescent liz® dye during the hybridization of the microarray. in this study, we analyzed the gene expression profiles of two human tissues: brain and universal human reference sample. five technical replicates in three different sites were performed on the same total rna samples according to manufacturer's standard protocols. five different methods, quantile, median, scale, vsn and cyclic loess were used to normalize ab microarray data within each site. since fold change and variance dependency with intensity is platform dependent we were interested in evaluating the performance of these methods applied to ab microarray data, making this study the first one from this perspective. we restricted our attention on these five methods for the following reasons. these methods are most frequently used normalization methods for ab microarray data. in addition, the microarrays used in this study contain one probe for each gene, this design restricting the number of normalization methods to be used and making methods based on replicated measurements for each gene inapplicable. other normalization methods that would also be inapplicable include those explicitly developed for two color technology, or replicated measurements. genes spanning a wide dynamic range in gene expression levels were selected for real-time pcr validation. using the taqman® assays data as the reference set, the performance of the five normalization methods was evaluated focusing on the following criteria: sensitivity and reproducibility in detection of expression; fold change correlation with real-time pcr data; sensitivity and specificity in detection of differential expression; reproducibility of differentially expressed gene lists. the data set analyzed in this manuscript has been reported elsewhere and made publicly available via geo accession number gse using the platform gpl for taqman® assays data and gpl for applied biosystems human genome survey microarrays data.  target selection for real-time pcr validation in order to conduct a comprehensive survey of the arrays' performance, gene targets for real-time pcr validation were selected based on the following criteria: ensure a large enough number of validation targets to provide representative overviews of the microarray performance; select genes spanning a wide range of expression levels and fold changes. taqman® gene expression assays were used in this study, covering genes. over 90% of these genes were selected from a subset of refseq common to the various microarray platforms. this selection was designed so that the genes would cover the entire intensity and fold change ranges and include any bias due to refseq itself. a subset of genes were included based on tissue-specificity. sensitivity and reproducibility in detection of expression the dynamic range for the ab microarray platform spans 3– orders of magnitude, while taqman based real-time pcr can achieve 7– orders of magnitude dynamic range. the larger dynamic range imparts taqman® assays with higher detection sensitivity; we therefore used the taqman® assays data set as the reference set to evaluate the performance of microarrays in terms of detection sensitivity and accuracy. first, genes that are detectable and not detectable were determined for each sample according to manufacturer's recommendations. figure shows the relationship between percent genes detected by the microarrays out of the ones detected by taqman® assays as a function of ct measurement. gene expression levels were ordered according to taqman® assay measurements. a sliding window containing consecutive genes was constructed and moved one gene at a time to cover the whole range of ct values. within each sliding window, the percent of genes detected as present in at least half of the replicates of individual samples by the microarray platform was computed out the total of those detected by taqman® assays, and plotted as a function of mean ct value of the genes in the given window. the overall sensitivity ) and specificity ), are presented in table and are % and %, respectively in the uhr sample measured in test site for each platform, sample and site, a gene is declared detected if it is detected according to the platform specifications, in more than half of the replicates. concordance between detection calls are presented in these tables for uhr and brain samples in site. for each normalization method, the impact on signal level was determined using genes detected by taqman® assays. there are almost no differences in the signal levels when each of these normalization methods is used with the exception of vsn which results in a small increase in signal level for low expressing genes. we also used coefficients of variation of log to evaluate the effect of the normalization methods on signal reproducibility, both within and between sites. reproducibility of technical replicates for the five normalization approaches for site is illustrated in figure for both brain and uhr samples. panel a, where all genes are represented, shows the coefficient of variation across the technical replicates, as a function of expression level when the data is normalized using the quantile approach. panel b shows the coefficients of variation, only for genes with taqman® assays as a function of taqman ct. lines represent the lowess smoothing fitting curves of all data points from each normalization method. as expected, cv's showed a strong dependence on expression level, decreasing from 10% for low expressers or absent genes, to 1% for high expressers. all normalization methods improved the coefficient of variation observed in the raw data over the entire range of expression levels. a small improvement in reproducibility for genes expressed at lower levels was observed in vsn normalization, in both representations, and for both samples. signal reproducibility between the testing sites is represented in figure within sites cvs and between sites cvs of all genes, show similar trends for all five normalization methods. the vsn normalization which showed some improvement in within sites variability of low expression level genes performed similarly to the other normalization methods when between sites variability is considered. fold change concordance with taqman® assays to evaluate the concordance of fold changes between microarray and real-time pcr data, we performed regression analysis of fold differences between the uhr sample and brain sample. fold change metrics are more meaningful as they tend to cancel out systematic platform biases in absolute signal values, moreover they are more biologically relevant. fold change was computed as the difference in mean expression level of the five technical replicates measured within each site for each sample. genes were filtered based on real-time pcr detection thresholds and only genes detected in both samples were used. fold changes between brain and uhr samples were estimated and plots between fold changes determined from ab microarray data and taqman® assays, are presented in figure the linear regression fitting curve for all data points was performed for each scatter plot. the r slope and intercept for taqman® assays versus microarray data are presented in table the normalization methods used for the microarrays showed similar linear correlation characteristics with real-time pcr measurements with r values ranging between – and slopes ranging from to suggesting the existence of some ratio compression in the microarray data. vsn showed slightly higher compression than the other normalization methods, most likely caused by the higher signal levels they impart for low expressers. a similar fold-change comparison between the five normalization methods and taqman® assays was also performed using lowess smoothing, which does not assume a linear relationship of fold-change values between platforms. the estimated range of fold changes, for taqman assays, is from - to while for ab microarrays is - to in order to better understand the cause of fold change compression additional analysis was performed on genes with similar expression levels in the two samples. genes were binned into low/medium/high according to taqman® assays ct measurements. only genes having expression level in the same bin in both samples a and b are included. boxplots of the fold change for each normalization method and taqman® assays are presented in figure the range of fold changes measured by the microarrays is significantly lower for low expression level genes and somewhat lower for high expression level genes when compared to the range of fold changes measured by taqman® assays. for medium expression level genes, the two platforms have a higher agreement on the magnitude of fold differences between the samples. sensitivity and specificity in detection of differential expression we evaluated the performance of the different normalization methods in detecting differential expression between the two samples using multiple statistical approaches. traditionally, analysis of accuracy is carried out by analyzing the true positive rate and false discovery rate. in this case, the actual rates are unknown. for this reason, we used taqman® as the reference platform. only genes detected by taqman® assays in both samples were used for this section. using the assay calls as the reference, we constructed contingency tables against microarray data, in which the concordance was determined and both the p-value significance of the t-test controlling false discovery rate at 5% level, and fold-change directionality were taken into consideration. specifically, true positives are genes differentially expressed in both taqman® and microarray platforms with similar direction of the fold change; true negatives are genes not differentially expressed in both platforms; false positives, consist of two sets of genes: genes not differentially expressed in taqman® and differentially expressed in microarrays, or genes differentially expressed in both platforms with opposite fold change direction; false negatives, genes differentially expressed for taqman and not for microarrays. genes were first ranked according to their average ct value in uhr and brain samples. for each bin of consecutive genes, we compare the results from each normalization method with the ones obtained with the taqman® assays. tpr defined as tpr = tp/, represents the percentage of genes detected differentially expressed in microarray data out of the ones detected by taqman® assays. fdr was defined as fp/, and represents the percentage of differentially expressed genes detected only by the microarrays out of all genes differentially expressed in microarrays. as shown in figure at the highest expression level, all normalization methods displayed reasonably good sensitivities: 70–75% tpr; the performance drops as the expression level decreases, and at the lowest expression level, the tpr is ~20%. the opposite trend can be seen for the fdr plots: a relative constant level of false findings was observed for genes with high and medium expression levels, after which fdr increases up to 25–30% for genes with low expression level. the overall accuracy of each of the five normalization methods is presented in table tpr and fdr indicate that the performance of the microarray platform is not dependent on the normalization method used. the observed fdr of % is slightly higher than the one expected from the fdr control we used to select differentially expressed genes, the 2% overestimate being possibly explained by genes incorrectly called differentially expressed by the taqman® assays. genes detected in both samples by taqman® assays are used for this comparison. t-test is used to detect significantly differentially expressed genes, controlling fdr at 5% level. we compare the results from each normalization method, for all sites, with the ones from taqman® assays keeping track of up/down regulation in each platform. additionally, we investigated four commonly used methods for identification of differentially expressed genes in microarray data: simple t-test, t-test combined with fold-change, t-test with fdr and fc control and sam, to determine their impact on the detection of differentially expressed genes. only data from site was used for this case. figure shows tpr and fdr plots comparing genes differentially expressed for each statistical method applied to each normalization approach with taqman® data used as a reference as previously described. table presents the overall concordance for quantile normalized data. the performance of the five methods does not change with the normalization method used. it is important, however, when interpreting these results to take into consideration one particularity of this experiment. the two samples compared, human brain and uhr being extremely divergent tissue types, display big differences in gene expression levels, and so the vast majority of genes used showed significant changes. from this perspective, it is expected that fdr control, for genes found differentially expressed by microarrays, will have little impact on fp rates. this explains why the results obtained from the simple t-test and the t-test with fdr control are very similar. in a previous comparative study ) where samples with smaller differences were used, we have seen that these two methods show bigger differences in specificity. on the other hand, restrictions on the magnitude of fold change reduce both the number of true positives and false positives. finally, sam produces even more specific results penalizing some of the low expressers that the taqman® assays find differentially expressed. we use different methods to detect significantly differentially expressed genes for data quantile normalized: t-test, t-test controlling fdr at 5% level, t-test and fc <, t-test controlling fdr at 5% level and fc <, or sam q <. we compare the results for data generated by site from each normalization method, with the ones from taqman® assays for which differential expression is detected using t-test and controlling fdr at 5% level. we keep track of up/down regulation in each platform. reproducibility of differentially expressed gene lists a fundamental step in most microarray experiments is determining lists of differentially expressed genes that distinguish biological conditions. reproducibility of differentially expressed genes across highly similar experiments is one of the important aspects of assessing reliability of microarray results. we used percentage of overlapping genes between differentially expressed genes lists as the measure of reproducibility. for each testing site and each normalization method, we declared genes differentially expressed again using t-test and controlling fdr at 5% level. in this way, for each site we generated a list of differentially expressed genes. figure shows the overlap between these lists of genes for each normalization method. table summarizes both percentages and counts of gene overlapping between either pairs of sites or all three sites. one can see that the pog obtained from different normalization methods are similar, ranging from % to % when all three sites are compared. site shows some differences compared to the other two sites, while the comparison between sites and shows consistently 83% pog. very similar results were observed when the other statistical methods were used for generating gene lists for microarray data. it is also important to note that no genes showed discordant results between brain and uhr, i.e. a significant fold change in opposite direction, when different normalization methods were used. we use t-test controlling fdr at 5% level within each site to detect significantly differentially expressed genes for normalized data. we compare the results for each normalization method, from the sites. first table contains percentage of concordant genes between sites. second table contains the counts of genes differentially expressed. discussion one unanswered question in the microarray field has always been the effect of various normalization as well as statistical methods on the end results of a profiling experiment and more explicitly whether using different normalization or statistical approaches results in different gene lists of less concordance between different microarray platforms. in this study we have assessed the performance of five different normalization methods using the applied biosystems expression array system. our results show a high level of concordance between these normalization methods. this is true, regardless of whether signals, variation or fold change measurements were interrogated. in addition, these five normalization methods showed similar performance of signal reproducibility between the three testing sites used for this study. furthermore, we used taqman® assays as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range. tpr was directly correlated to gene expression levels whereas fdr was inversely correlated. this is not completely surprising as the two platforms have different dynamic ranges and sensitivity levels, with the detection levels of the microarrays being lower than those of taqman® assays. these differences more than likely explain the lower tp rates and higher fp rates for the genes at the low expression levels. these effects were also observed for several other microarray platforms in a separate study. one conclusion of this study is that, at least for the microarray platform tested in this study, the current normalization approaches have little impact on the signal, detection levels as well as tp and fp rates in detection of differentially expressed genes. these results are consistent with the findings of the maqc study. in addition we also explored the contribution of several statistical approaches commonly used in the field on the tp and fp rates. as expected in this case, with approaches which relax the stringency in differential expression, better detection and differential expression concordance is observed, concomitant with a higher percentage of false positives. at the opposite end of the spectrum, fdr control and sam methods, which are more restrictive in detection of differential expression, produce gene lists with fewer false positives. sam, as expected, shows a reduced number of false positives for low expressers, at the expense of missing some differentially expressed genes. the expected percentage of false positives in these lists is close to the one observed when comparing results to taqman® assays. unfortunately, it seems that the full strength of these statistical methods is obscured by the fact that the majority of the genes chosen for taqman® validation show significant fold changes between samples, minimizing the effect of fdr on the fp rate. more importantly however, applying the different normalization approaches to the various statistical methods tried, had no significant impact on identifying differentially expressed genes. finally, when comparing the overlap in gene lists generated by each of these statistical methods, a concordance of – % was observed between all three sites, and – % between sites and indicating little effect of the analysis approach used on the final gene list obtained. this result is, however, sensitive to the cut-offs used in determining the gene lists and can affect the degree of overlap observed. we were pleasantly surprised, however, of the little effect observed by the various normalization on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field.  in this study we have assessed the performance of five different normalization methods using data generated with the applied biosystems expression array system. our results show a high level of concordance between these normalization methods. this is true, regardless of whether signals, variation, site reproducibility or fold change measurements were interrogated. the same similarity is observed when taqman® assays were used as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range. in addition we also explored the contribution of several statistical approaches commonly used in the field on the detection of differential expression. little effect is observed by the various normalization methods on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field, particularly when used in conjunction with the applied biosystems microarrays. 
a few key issues, including the reproducibility, reliability, compatibility and standardization of microarray analysis and results, must be critically addressed before any routine usage of microarrays in clinical laboratory and regulated areas can occur. five different methods, quantile, median, scale, vsn and cyclic loess were used to normalize ab microarray data within each site. in this study we investigate some of these issues for the applied biosystems human genome survey microarrays. we analyzed the gene expression profiles of two samples: brain and universal human reference, a mixture of rnas from cancer cell lines, using the applied biosystems human genome survey microarrays. five technical replicates in three different sites were performed on the same total rna samples according to manufacturer's standard protocols. our results showed a high level of concordance between these normalization methods. little impact is observed on the tp and fp rates in detection of differentially expressed genes. additionally, little effect was observed by the various normalization methods on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field, particularly when used in conjunction with the applied biosystems gene expression system. furthermore, we used taqman® assays as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range. while the technology has been widely used in discovery-based medical and basic biological research, its direct application in clinical practice and regulatory decision-making has been questioned. dna microarray technology provides a powerful tool for characterizing gene expression on a genome scale. genes spanning a wide dynamic range in gene expression levels were selected for real-time pcr validation. using the taqman® assays data set as the reference set, the performance of the five normalization methods was evaluated focusing on the following criteria: sensitivity and reproducibility in detection of expression; fold change correlation with real-time pcr data; sensitivity and specificity in detection of differential expression; reproducibility of differentially expressed gene lists.
 dna microarray technology provides a powerful tool for characterizing gene expression on a genome scale. while the technology has been widely used in discovery-based medical and basic biological research, its direct application in clinical practice and regulatory decision-making has been questioned. a few key issues, including the reproducibility, reliability, compatibility and standardization of microarray analysis and results, must be critically addressed before any routine usage of microarrays in clinical laboratory and regulated areas can occur. in this study we investigate some of these issues for the applied biosystems human genome survey microarrays.  we analyzed the gene expression profiles of two samples: brain and universal human reference, a mixture of rnas from cancer cell lines, using the applied biosystems human genome survey microarrays. five technical replicates in three different sites were performed on the same total rna samples according to manufacturer's standard protocols. five different methods, quantile, median, scale, vsn and cyclic loess were used to normalize ab microarray data within each site. genes spanning a wide dynamic range in gene expression levels were selected for real-time pcr validation. using the taqman® assays data set as the reference set, the performance of the five normalization methods was evaluated focusing on the following criteria: sensitivity and reproducibility in detection of expression; fold change correlation with real-time pcr data; sensitivity and specificity in detection of differential expression; reproducibility of differentially expressed gene lists.  our results showed a high level of concordance between these normalization methods. this is true, regardless of whether signal, detection, variation, fold change measurements and reproducibility were interrogated. furthermore, we used taqman® assays as a reference, to generate tpr and fdr plots for the various normalization methods across the assay range. little impact is observed on the tp and fp rates in detection of differentially expressed genes. additionally, little effect was observed by the various normalization methods on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field, particularly when used in conjunction with the applied biosystems gene expression system. 