BACKGROUND
endogenous reference alse referred to as house keeping genes defines in biology the theoretical assumption that certain genes are ubiquitously expressed in nucleated cells possibly because their stable expression is essential for cell survival and welfare in all physio-pathological circumstances.
in practical terms, endogenous reference genes provide a useful constant reference to normalize the expression of test genes in different tissues and in different conditions.
this is obviously important when estimates of gene expression are provided in relative terms rather than absolute units of measurement.
thus, endogenous reference genes are used as common denominator in biological fractions where the expression of a test gene is described as the relative ratio over an arbitrarily selected internal control presumed to be stably expressed in all circumstances relevant to the experiment  <cit> .
most frequently, glyceraldehydes-3-phosphate dehydrogenase   <cit> , albumin   <cit> , β-, γ-actins  <cit> , cyclophilin  <cit> , α-, β-tubulins  <cit> , hypoxantine phosphoribosyltransferase   <cit> , l <dig>  <cit>  and 18s, 28s ribosomal rna   <cit>  have been used as endogenous reference genes.
depending upon the experimental design, endogenous reference genes have been used individually or in combination for northern blot analysis, reverse transcription polymerase chain reaction  and quantitative real-time pcr  analysis  <cit> .
with the development cdna microarray technology endogenous reference genes have been used for array data normalization.
however, accumulation of extensive data bases suggests that the expression of frequently used endogenous reference genes can vary substantially according to materials and conditions studied  <cit> .
powerful insights in patterns of gene expression could be attained recently through cdna or oligonucleotide-based global transcript analysis tools that apply a constant reference system to determine ratios of gene expression across large data sets  <cit> .
the constant reference is provided for each gene in question by consistently co-hybridizing individual test samples with a differentially labeled reference sample maintained identical throughout all the hybridization experiments.
gene expression data are then expressed as the ratio of expression between test and reference samples for each gene.
by keeping the reference sample identical the resulting ratio represents a precise estimate of the relative expression of each gene across the various conditions tested bypassing the need to normalize with endogenous reference genes.
this holds true if the hybridization kinetics between test and reference sample are accurately reproducible.
we will refer to this concept as "reference concordance" and in the results we will discuss how reference concordance was used to validate the reproducibility of cdna array data from which putative candidate endogenous reference genes were identified.
in the present study, we tested a set of  <dig> consecutive experiments performed on a  <dig>  gene cdna array platform to which rna from neoplastic or normal tissues were consistently co-hybridized with a differentially-labeled reference rna derived from peripheral blood mononuclear cells  pooled from six normal donors.
the following steps were pursued: 1) reproducibility assessment of the data set through determination of reference concordance.
this was achieved by repeating  <dig> reference experiments using the melanoma cell line a <dig> as test sample  co-hybridized with pooled pbmc as reference .
to test for inter-array and printing variation, slide number one and every other  <dig> slides in sequential order of printing  were used for the repetitive a <dig> / pooled pbmc hybridizations.
in addition, to assess labeling bias, reciprocal labeling was alternatively applied as previously described  <cit> .
in this fashion a pool of genes expressed with high level of reference concordance was selected.
2) identification of putative endogenous reference genes was performed on  <dig> array experiments of unequivocal quality by selecting genes that had demonstrated high reference concordance  and ranking them from the lowest to the highest variance of log <dig> test / reference ratios across all array experiments.
3) validation of the candidate endogenous reference genes as predictor of relative gene expression in large data series.
for this purpose, we tested the relative estimates of expression of the melanocytic lineage-restricted melanoma differentiation antigen gp100/pmel <dig>   <cit>  in melanocytic and non-melanocytic tissues.
estimates of expression of gp <dig> were compared after normalization with different endogenous reference genes.
for this analysis, we used a new tool that spots cdna libraries from different tissues on an array platform to allow high-throughput evaluation of individual gene expression across broad tissue collections.
we termed this tool: "transcriptome array".
4) validation of endogenous reference gene-based normalization of gene expression by qrt-pcr and protein expression.
RESULTS
data processing
the genepix pro  <dig>  software was used for array image analysis and calibration.
all statistical analyses were performed with the splus package .
thirteen arrays with missing data in more than 30% of the spots and  <dig> arrays with irregular distribution according to m-a plots   <cit>  were excluded.
spots in which > 50% of the pixels reached saturation in either channel, flagged spots and spots with intensity ≤  <dig> in one channel and ≤  <dig> in the other channel were filtered out.
if the intensity in one channel was <  <dig> but that of the other channel was >  <dig>  we arbitrarily assigned an intensity of  <dig> to the channel with the lowest intensity.
the log2cy5/cy <dig> ratios were normalized by approximating median values to zero.
spots with ratios >  <dig> or < 1/ <dig> were truncated at  <dig> and 1/ <dig> respectively.
data normalization was done by median centering log2ratio of all genes for each array.
for data normalization we used certain arbitrary cutoff criteria to remove spots with weak signal.
weak signal approximating background fluorescence is not reliable and the corresponding log ratios are often distorted resulting in disproportionately high or low values that would bias the statistical results.
because the interpretation of spots with low signal is difficult to make we have adopted the policy of excluding them to focus the analysis on genes whose expression pattern is more reliably tracked by the array tool.
in regard to spots with a ratio of > <dig> or <-1/ <dig>  most of them are due to extremely weak signal in one channel which generates a disproportionately extreme value.
although there is no published "gold standard" for the selection of such cutoffs, in practice the parameters that we used for this study are most commonly accepted by investigators and reasonable because allow retention of most of the data in the array excluding only the most extreme and least reliable information.
validation of the data set through analysis of "reference concordance"
to select from the data base the most reliable genes, we first assessed reproducibility by testing the level of concordance in  <dig> reference experiments.
reference concordance relies on the expectation that results obtained by repeatedly hybridizing the same test and reference material should perfectly collimate and the degree of deviation from such prediction estimates experimental variance.
concordance can be easily measured by periodically re-hybridizing the same test sample with the constant reference sample.
we analyzed a matrix of  <dig> forward and  <dig> reciprocally-labeled replicate array experiments hybridized periodically every other  <dig> cdna array slides.
reciprocal labeling was applied to measure labeling bias  <cit> .
data generated by reciprocal experiments were mathematically reversed into the same labeling direction for data analysis.
genes that were discordant due to labeling bias were identified by student's t test as those with ratios significantly different between the  <dig> forward and  <dig> reciprocal experiments after reversal of the reciprocal values  and the median ratio difference between the  <dig> forward and  <dig> reciprocal experiments was larger than  <dig>  fold.
the genes whose variances of the log ratio across all experiments were among the top  <dig> percentile of variance of all genes were identified as discordant due to hybridization bias.
in addition, genes with more than 50% missing values were excluded.
overall  <dig>  out of  <dig>  were judged potentially discordant and were excluded from further analysis.
the remaining  <dig>  concordant genes were used for subsequent analyses.
endogenous reference gene identification
we studied  <dig> of  <dig> consecutive array experiments remaining after the exclusion of the  <dig> reference concordance experiments and  <dig> experiments judged of poor quality due to missing data or irregular distribution by m-a plot analysis.
selection of candidate endogenous reference genes followed these steps.
first, the seven experiments used for the analysis of reference concordance in which labeling was done identically to the rest of the experimental samples  were used to calculate experimental variance .
median of variances across the seven replicate arrays was calculated for all genes with average intensity in both channels >  <dig> .
this parameter provides an estimate of the variance due to experimental variation  since theoretically no differences in gene expression should be noted by using the same material.
based on the assessment of the seven repeat experiments, we defined differential expression as >  <dig> standard deviations  from the mean of the  <dig> arrays which is equivalent to  <dig> -fold change .
consequently, genes with fold changes <  <dig>  from zero across all  <dig> arrays were categorized as candidate endogenous reference genes and were ranked according to ascending values of sd of log ratio .
among these genes, we further selected candidate endogenous reference genes according to the following criteria: in at least 95% of the  <dig> arrays the log <dig> ratio had to be within  <dig> sd from zero and the average intensities of both channels across all samples need to be higher than  <dig> .
the intensity parameter was added to ensure that the selected endogenous reference genes were expressed at a relatively high level in most tissues to mitigate excessive fluctuations in log <dig> ratios occurring when a low value is applied as denominator.
this is important when a reference gene is applied as a denominator in the equation used to normalize the ratio of other tests genes; in such cases robust gene expression decreases the range of ratios resulting from the analysis decreasing, therefore, the experimental variance.
sixty-nine genes fit these criteria.
the range of mean log <dig> ratio was from  <dig> to  <dig>  and the sd of mean log <dig> ratio was from  <dig>  to  <dig> .
analysis of  <dig> replicate array based on the same group of candidate endogenous reference genes demonstrated a similar distribution of mean log <dig> ratio and sd of mean log <dig> ratios  suggesting that this variance could be attributed to predominantly experimental noise.
among these genes we further selected for validation  <dig> that had high mean fluorescence intensity .
however, probe preparation and other technical considerations limited to analysis to only  <dig> of these genes which included: need <dig>  hirip <dig>  gpld <dig>  rab <dig>  snrpd <dig>  fzd <dig>  col8a <dig> and afap.
"leave-one-out cross-validations" were used to estimate the error  <cit> .
in each validation experiment one array was left out.
the remaining  <dig> arrays were used to identify endogenous reference genes using the same parameters used for the original analysis .
the median log <dig> ratio of the endogenous reference gene in the left out array was then compared with the "real" normalization factor consisting of the median log <dig> ratio for all genes in that array supposed to approximate a balanced expression of test and reference genes for that array.
this procedure was repeated for each array  <dig> times.
only in  <dig> arrays out of  <dig>  were found errors in endogenous reference gene selection .
the expression of each candidate endogenous reference gene identified in this study was then compared with available information about the expression of the same genes in  <dig> normal human tissues reported by the affymetrix hg-u95a-e probe sets .
we found a good correlation between the expression patterns observed by us and that reported by the affymetrix geenchip array .
in addition, a cluster of endogenous reference genes suggested by applied biosystems for standardization during qrt-pcr was analyzed by evaluating the sd of their log <dig> ratios across the  <dig> arrays.
as data indicated in table  <dig>  although some genes  demonstrated mean sd log <dig> ratios comparable to that of the endogenous reference genes identified in this study, the corresponding percentage of the arrays in which log <dig> ratios were within  <dig> sd from zero ranged between  <dig> – <dig>  which is significantly lower than the frequency in which the same parameter fell for the  <dig> candidate house keeping genes identified here .
in addition, we looked at the variation in our arrays of genes classically used as endogenous reference genes including: ribosomal protein l <dig>  hrpt, β-actin and tubulin-α <dig>  in all cases the mean sd of log2ratio was relatively larger than for the endogenous reference genes identified in table  <dig> with the following respective values:  <dig>  for ribosomal protein l <dig>   <dig>  for hrpt,  <dig>  for β-actin and  <dig>  for laminin-α <dig>  interestingly, however, the mean intensities were relatively higher for all of these genes compared with the ones identified in this study  suggesting that these genes are expressed probably at higher concentrations in tissue and, therefore, although relatively variable in expression in different tissues, they have been a useful marker of rna abundance when only rough estimates are required like, for instance, for northern blotting.
to test the usefulness of various endogenous reference gene we resorted to the well characterized expression of the melanocytic lineage-specific gene gp100/pmel <dig>  <cit>  that is expressed exclusively though heterogeneously in cells of melanocytic lineage  <cit> .
the assumption of this experiment was that in spite of its heterogeneity of expression in samples of melanocytic origin, overall the expression of gp <dig> should be higher in meloanocytic compared with non melanocytic samples with a high degree of significance.
rho-c has been associated with the metastatization process in melanoma but its specificity for melanocytic lineage is unknown  <cit> .
the differentiation control element dice is found in the 3'-utr of numerous eukaryotic mrnas and there is no solid association between its pattern of expression and specific physio-pathological or developmental conditions  <cit> .
therefore, we compared the expression profile of these three genes in  <dig> cdna libraries that included  <dig> melanoma cell lines,  <dig> melanoma metastases,  <dig> benign nevi  and  <dig> miscellaneous samples containing a predominance of primary esophageal, renal and colon cancers paired with normal tissues from the same organ and a large number of circulating mononuclear cells.
in addition, samples from most other normal and cancerous tissues were included although in smaller number .
as endogenous reference genes we chose β-actin and new candidate genes identified by this study .
the test genes  were separately hybridized to the transcriptome array.
each gene labeled with cy <dig> was co-hybridized with individual endogenous reference genes labeled with cy <dig>  we then compared log <dig> cy <dig> / cy <dig> ratios between melanocytic and non-melanocytic tissues.
gp <dig> was, as expected, expressed more in melanocytic lesions with high degree of significance no matter what gp <dig> / endogenous reference gene combination was used.
the range of significance, however, varied extensively depending upon the gp <dig> / endogenous reference gene combination.
this difference was considered representative of the ability of different endogenous reference genes to normalize for tissue-specific gene expression patterns .
in details, all endogenous reference genes could segregate melanocytic from non melanocytic lesions with a high degree of significance .
there was, however, a big range in the discriminatory capacity among endogenous reference genes with nedd <dig>  rab <dig> and fzd <dig> providing the highest predictive value  and β-actin being one of the least reliable .
the large variation in the results obtained using different endogenous reference genes to normalize gp <dig> expression could have been due to a higher stability of the expression of some genes across all samples or to a differential expression of the endogenous reference genes themselves in melanocytic lesion.
in the latter case, the better results obtained could be coincidental and not useful in other experimental situations.
we, therefore, tested whether pooling results obtained with all endogenous reference genes independently of each predictive value in this controlled experimental situation could yield results as informative about gp <dig> pattern of expression as those obtainable with individual endogenous reference genes, particularly those that provided the best prediction.
the advantage of this strategy is that it does not depend on previous knowledge of gene expression patterns for the selection of individual endogenous reference genes not applicable in conditions, in which the suitability of a gene for a given experimental situation is, contrary to gp <dig>  is unknown.
thus, we compared the predictive value of the average of gp <dig> / endogenous reference gene log <dig> ratios obtained with all endogenous reference genes and those obtained using the genes that provided the best five or the best three results .
the use of pooled information from various endogenous reference genes appeared to stabilize the non-melanocytic log <dig> ratios gp <dig> / endogenous reference.
in fact, the sd of thelog <dig> ratios gp <dig> / endogenous reference among non-melanocytic samples derived by pooling endogenous reference gene results were significantly lower than the sd obtainable with any individual endogenous reference gene .
the basis for this test was that non-melanocytic lesions uniformly should not express gp <dig> and, therefore, the log <dig> ratios for any gp <dig> / endogenous reference gene combination should be constant resulting in very low sd.
sd for non-melanocytic lesions were much lower using pooled endogenous reference genes for the normalization  and this was true at extreme levels of significance .
similar results were obtained when results from the best  <dig> and best  <dig> endogenous reference genes were pooled together.
in addition, even if pooling data together did not provide the highest level of discrimination compared to the best results obtained with selected individual endogenous reference genes, the capacity of pooled data sets to discriminate gp <dig> expression between melanocytic and non melanocytic samples was still very high .
interestingly, different endogenous reference genes provided not only different levels of discrimination but also provided different estimates of gp <dig> expression in melanocytic lesions with gp <dig> / endogenous reference gene log <dig> ratio above  or below   <dig> .
this, of course, has no impact on the ability to characterize patterns of expression of different genes in different tissues but rather suggests that the latter group of endogenous reference genes is expressed at higher copy number, therefore, biasing the data toward its own fluorescence channel.
simple adjustment in probe concentration could easily solve this problem.
we then analyzed the expression of rho-c and dice whose pattern of expression in the transcriptome array cannot be predicted based on available information save for the notion that of the expression of rho-c is related to the metastatic process of melanoma  <cit> .
the pattern of rho-c was characterized with all the endogenous reference genes and representative information is presented in figure  <dig>  most endogenous reference genes yielded a pattern suggestive of a preferential expression of rho-c in melanoma metastases.
in addition, averages of rho-c / endogenous reference gene log2ratios demonstrated clearly a specific expression of this gene in melanoma metastases.
this pattern was not necessarily specific for melanoma as the other cancerous tissues spotted in the transcriptome array were obtained from primary lesions.
thus, it is possible that the preferential expression in melanoma metastases was due to the metastatic process rather than the melanocytic lineage.
interestingly, the melanoma cell line mel-a <dig> did not constitutively express rho-c as previously observed by clark et al.
<cit> .
in addition, none of the melanoma cell lines mostly derived from melanoma metastases demonstrated expression of rho-c. this information suggests that rho-c may be involved in the natural metastatic process in vivo but is not constitutively expressed in vitro.
finally, dice did not demonstrate any specificity of expression and appeared variably expressed in all specimens independently of the endogenous reference gene used for normalization.
validation of data by protein analysis and quantitative real-time pcr
we tested whether gp <dig> / endogenous reference gene log <dig> ratios correlated with gp <dig> protein expression level as measured by intra-cellular facs analysis .
the gp <dig> / endogenous reference gene log <dig> ratios were derived from the transcriptome arrays while protein expression was based on previous characterizations of the same cell lines  <cit> .
overall, a good correlation  was noted for gp <dig> values normalized with most endogenous reference genes.
however, with the exception of nedd <dig>  the best correlation was obtained when pooled information was used from the best  <dig> and  <dig> candidate endogenous reference genes as defined before.
interestingly, rab <dig> that scored very high as a predictor of gp <dig> transcriptional expression in melanocytic lesions and β-actin provided the worst correlation with protein expression.
this data suggest that, although different endogenous reference genes may yield better predictive value than others in large data sets, it is likely that their reliability varies in different conditions and possibly the best results can be obtained by pooling information from several of them.
we also tested by quantitative real-time pcr  the discriminatory capacity of endogenous reference genes in samples with known patterns of expression of gp <dig>  we selected  <dig> previously characterized  <cit>  melanoma cell lines four of which were known to express  and two known not to express  gp <dig> at the transcriptional and protein level   <cit> .
in addition,  <dig> fine needle aspirate  samples from melanoma metastases expressing gp <dig>  <cit>  and a series of tumors  known not to express gp <dig> were tested.
absolute expression of gp <dig> was measured and although the lesions supposed to be gp <dig> expressing appeared more positive than those supposed to be gp <dig> negative there was no absolute demarcation among the two groups .
a better differentiation between positive and negative lesions could be obtained when the various endogenous reference genes were used for the analysis or the combined values from all of them .
discussion
although the concept of endogenous reference genes is appealing it may be unrealistic to expect that a gene could be equally expressed in all eukaryotic cells in all physiopathological conditions.
thus, the endogenous reference gene concept can only approximate such ideal biological behavior.
in fact, numerous studies noted that the expression of the ostensible endogenous reference genes varies according to distinct physio-pathological situations and can be affected by experimental manipulation  <cit> .
for instance, the levels of expression of gapdh, cyclophilin and β-actin fluctuate in different tissues, disease stage and are affected by cell behavior like proliferation  <cit> .
in addition, endogenous reference genes expression varies markedly in cancer compared to normal tissue  <cit>  suggesting that oncogenesis may influence their expression  <cit> .
in this study, therefore, we looked for relative stable endogenous reference gene challenging a  <dig>  cdna clone data base against approximately  <dig> specimens that included a broad variety of normal and neoplastic tissues as well as cell lines.
indeed, few previously unnamed genes approximated the ideal endogenous reference gene by being expressed consistently in more than 95% of the specimens analyzed within a variance that was barely and insignificantly different from what could be expected from intrinsic experimental fluctuation .
in fact, the threshold that we selected based on the variance observed by periodically repeating identical experiments was consistent with criteria suggested by others  <cit>  who claimed that genes with fold changes greater than  <dig>  can be classified as differentially expressed at ~95% confidence interval.
this is consistent to our selection criterion that contained endogenous reference genes within a  <dig>  fold maximal variation.
although these genes have not been proposed before as candidate endogenous reference genes, analysis of available data bases suggested that they are relatively stably expressed in various normal tissues .
selection of these candidate endogenous reference genes provided useful information when the expression of a lineage specific gene such as gp <dig> was compared between relevant and irrelevant samples.
in addition, gp <dig> expression normalized according to some of the new endogenous reference genes more closely aligned to lineage specificity than normalization performed using β-actin.
this data, however, confirm that there is no such thing as a perfect endogenous reference gene.
although some endogenous reference genes demonstrated high power of discrimination their ranking was dictated by previous knowledge of the expected experimental results .
however, gene expression analysis is done to determine such specificity and in most cases no previous knowledge of tissue specificity is truly available.
one may argue that, in spite of the extreme variability in normalization power demonstrated by various endogenous reference genes, still significant differences could extracted between the melanocytic and non-melanocytic samples no matter what endogenous reference genes were used.
this statement is correct in this experimental situation where hundreds of samples could be compared with the transcriptome array.
however, in cases when only few samples are analyzed the difference in discriminatory capacity becomes critical.
for instance, when randomly selected gp <dig> / endogenous reference gene log <dig> ratios from  <dig> melanoma metastases were compared with those from  <dig> kidney specimens only nedd <dig>  rab <dig>  fzd <dig>  afap, col <dig>  snrpd and pooled endogenous reference gene-based normalizations yielded significant differences between the two groups while non significant values were obtained for normalizations based on hirip <dig>  β-actin and gpld <dig>
the lowest limit of detection in the transcriptome array of specific probes in term of copy numbers is currently under investigation.
decreasing concentrations of β-actin alone printed the array spots as internal controls suggest that the detection range is from  <dig> ng / spot to  <dig> pg / spot when standard fluorescent intensity range parameters are applied.
the higher limit of detection before saturation can be easily adjusted to avoid under estimates of highly expressed genes, therefore, increasing the dynamic range of detection.
this is achieved by adjusting the fluorescent intensity right below the saturation for both channel and subsequently calculating the abundance of reference gene expression using serially diluted internal control spots as standard curve.
with few exceptions, the expression of most genes can be estimated with relative accuracy using this semi-quantitative approach.
since in most cases endogenous reference genes are selected based on previous knowledge of their usefulness for a particular experimental situations, it is likely that the best consistency could be achieved in such circumstances when averages of several endogenous reference gene normalizations are used.
this may represent a reasonable solution particularly if a cocktail of endogenous reference genes identically labeled could be used as reference within a single experiment.
we are presently testing various combinations of lineage specific genes and pools of endogenous reference genes for hybridization to the transcriptome array and for qrt-pcr.
such attempts, however, have been quite disappointing so far as results obtained using mixture of endogenous reference genes do not yield information comparable to that achievable when results obtained with the same endogenous reference genes used individually are pooled together.
this is possibly due to imbalanced density of various endogenous reference genes in different samples with predominant effects of some over others.
therefore, extensive work in the future will be aimed at equilibrating the interactions among endogenous reference genes in various experimental conditions to test whether mixtures of them could be used in multiplex for normalization of large expression analysis studies.
CONCLUSIONS
our observations, together with previous work by others indicate that much caution should be taken when using endogenous reference to normalize the expression of test genes.
it is likely that previous analyses based on single endogenous reference genes might have been strongly biased by the individual selections and the interpretation of the results should looked at with caution.
in particular, this study suggests that even when the most stably expressed genes in array experiments are used as endogenous reference genes, significant variation in test gene expression estimates may occur and the best normalization is achieved when data from several endogenous reference genes are pooled together to minimize minimal but significant variation among samples.
we are presently optimizing strategies for the preparation of endogenous reference gene mixtures that could yield information comparable to that of data pooled from individual endogenous reference gene normalizations.

 endogenous reference alse referred to as house keeping genes defines in biology the theoretical assumption that certain genes are ubiquitously expressed in nucleated cells possibly because their stable expression is essential for cell survival and welfare in all physio-pathological circumstances. in practical terms, endogenous reference genes provide a useful constant reference to normalize the expression of test genes in different tissues and in different conditions. this is obviously important when estimates of gene expression are provided in relative terms rather than absolute units of measurement. thus, endogenous reference genes are used as common denominator in biological fractions where the expression of a test gene is described as the relative ratio over an arbitrarily selected internal control presumed to be stably expressed in all circumstances relevant to the experiment. most frequently, glyceraldehydes-3-phosphate dehydrogenase, albumin, β-, γ-actins, cyclophilin, α-, β-tubulins, hypoxantine phosphoribosyltransferase, l and 18s, 28s ribosomal rna have been used as endogenous reference genes. depending upon the experimental design, endogenous reference genes have been used individually or in combination for northern blot analysis, reverse transcription polymerase chain reaction and quantitative real-time pcr analysis. with the development cdna microarray technology endogenous reference genes have been used for array data normalization. however, accumulation of extensive data bases suggests that the expression of frequently used endogenous reference genes can vary substantially according to materials and conditions studied. powerful insights in patterns of gene expression could be attained recently through cdna or oligonucleotide-based global transcript analysis tools that apply a constant reference system to determine ratios of gene expression across large data sets. the constant reference is provided for each gene in question by consistently co-hybridizing individual test samples with a differentially labeled reference sample maintained identical throughout all the hybridization experiments. gene expression data are then expressed as the ratio of expression between test and reference samples for each gene. by keeping the reference sample identical the resulting ratio represents a precise estimate of the relative expression of each gene across the various conditions tested bypassing the need to normalize with endogenous reference genes. this holds true if the hybridization kinetics between test and reference sample are accurately reproducible. we will refer to this concept as "reference concordance" and in the results we will discuss how reference concordance was used to validate the reproducibility of cdna array data from which putative candidate endogenous reference genes were identified. in the present study, we tested a set of consecutive experiments performed on a gene cdna array platform to which rna from neoplastic or normal tissues were consistently co-hybridized with a differentially-labeled reference rna derived from peripheral blood mononuclear cells pooled from six normal donors. the following steps were pursued: 1) reproducibility assessment of the data set through determination of reference concordance. this was achieved by repeating reference experiments using the melanoma cell line a as test sample co-hybridized with pooled pbmc as reference. to test for inter-array and printing variation, slide number one and every other slides in sequential order of printing were used for the repetitive a / pooled pbmc hybridizations. in addition, to assess labeling bias, reciprocal labeling was alternatively applied as previously described. in this fashion a pool of genes expressed with high level of reference concordance was selected. 2) identification of putative endogenous reference genes was performed on array experiments of unequivocal quality by selecting genes that had demonstrated high reference concordance and ranking them from the lowest to the highest variance of log test / reference ratios across all array experiments. 3) validation of the candidate endogenous reference genes as predictor of relative gene expression in large data series. for this purpose, we tested the relative estimates of expression of the melanocytic lineage-restricted melanoma differentiation antigen gp100/pmel in melanocytic and non-melanocytic tissues. estimates of expression of gp were compared after normalization with different endogenous reference genes. for this analysis, we used a new tool that spots cdna libraries from different tissues on an array platform to allow high-throughput evaluation of individual gene expression across broad tissue collections. we termed this tool: "transcriptome array". 4) validation of endogenous reference gene-based normalization of gene expression by qrt-pcr and protein expression.  data processing the genepix pro software was used for array image analysis and calibration. all statistical analyses were performed with the splus package. thirteen arrays with missing data in more than 30% of the spots and arrays with irregular distribution according to m-a plots were excluded. spots in which > 50% of the pixels reached saturation in either channel, flagged spots and spots with intensity ≤ in one channel and ≤ in the other channel were filtered out. if the intensity in one channel was < but that of the other channel was > we arbitrarily assigned an intensity of to the channel with the lowest intensity. the log2cy5/cy ratios were normalized by approximating median values to zero. spots with ratios > or < 1/ were truncated at and 1/ respectively. data normalization was done by median centering log2ratio of all genes for each array. for data normalization we used certain arbitrary cutoff criteria to remove spots with weak signal. weak signal approximating background fluorescence is not reliable and the corresponding log ratios are often distorted resulting in disproportionately high or low values that would bias the statistical results. because the interpretation of spots with low signal is difficult to make we have adopted the policy of excluding them to focus the analysis on genes whose expression pattern is more reliably tracked by the array tool. in regard to spots with a ratio of > or <-1/ most of them are due to extremely weak signal in one channel which generates a disproportionately extreme value. although there is no published "gold standard" for the selection of such cutoffs, in practice the parameters that we used for this study are most commonly accepted by investigators and reasonable because allow retention of most of the data in the array excluding only the most extreme and least reliable information. validation of the data set through analysis of "reference concordance" to select from the data base the most reliable genes, we first assessed reproducibility by testing the level of concordance in reference experiments. reference concordance relies on the expectation that results obtained by repeatedly hybridizing the same test and reference material should perfectly collimate and the degree of deviation from such prediction estimates experimental variance. concordance can be easily measured by periodically re-hybridizing the same test sample with the constant reference sample. we analyzed a matrix of forward and reciprocally-labeled replicate array experiments hybridized periodically every other cdna array slides. reciprocal labeling was applied to measure labeling bias. data generated by reciprocal experiments were mathematically reversed into the same labeling direction for data analysis. genes that were discordant due to labeling bias were identified by student's t test as those with ratios significantly different between the forward and reciprocal experiments after reversal of the reciprocal values and the median ratio difference between the forward and reciprocal experiments was larger than fold. the genes whose variances of the log ratio across all experiments were among the top percentile of variance of all genes were identified as discordant due to hybridization bias. in addition, genes with more than 50% missing values were excluded. overall out of were judged potentially discordant and were excluded from further analysis. the remaining concordant genes were used for subsequent analyses. endogenous reference gene identification we studied of consecutive array experiments remaining after the exclusion of the reference concordance experiments and experiments judged of poor quality due to missing data or irregular distribution by m-a plot analysis. selection of candidate endogenous reference genes followed these steps. first, the seven experiments used for the analysis of reference concordance in which labeling was done identically to the rest of the experimental samples were used to calculate experimental variance. median of variances across the seven replicate arrays was calculated for all genes with average intensity in both channels >. this parameter provides an estimate of the variance due to experimental variation since theoretically no differences in gene expression should be noted by using the same material. based on the assessment of the seven repeat experiments, we defined differential expression as > standard deviations from the mean of the arrays which is equivalent to -fold change. consequently, genes with fold changes < from zero across all arrays were categorized as candidate endogenous reference genes and were ranked according to ascending values of sd of log ratio. among these genes, we further selected candidate endogenous reference genes according to the following criteria: in at least 95% of the arrays the log ratio had to be within sd from zero and the average intensities of both channels across all samples need to be higher than. the intensity parameter was added to ensure that the selected endogenous reference genes were expressed at a relatively high level in most tissues to mitigate excessive fluctuations in log ratios occurring when a low value is applied as denominator. this is important when a reference gene is applied as a denominator in the equation used to normalize the ratio of other tests genes; in such cases robust gene expression decreases the range of ratios resulting from the analysis decreasing, therefore, the experimental variance. sixty-nine genes fit these criteria. the range of mean log ratio was from to and the sd of mean log ratio was from to. analysis of replicate array based on the same group of candidate endogenous reference genes demonstrated a similar distribution of mean log ratio and sd of mean log ratios suggesting that this variance could be attributed to predominantly experimental noise. among these genes we further selected for validation that had high mean fluorescence intensity. however, probe preparation and other technical considerations limited to analysis to only of these genes which included: need hirip gpld rab snrpd fzd col8a and afap. "leave-one-out cross-validations" were used to estimate the error. in each validation experiment one array was left out. the remaining arrays were used to identify endogenous reference genes using the same parameters used for the original analysis. the median log ratio of the endogenous reference gene in the left out array was then compared with the "real" normalization factor consisting of the median log ratio for all genes in that array supposed to approximate a balanced expression of test and reference genes for that array. this procedure was repeated for each array times. only in arrays out of were found errors in endogenous reference gene selection. the expression of each candidate endogenous reference gene identified in this study was then compared with available information about the expression of the same genes in normal human tissues reported by the affymetrix hg-u95a-e probe sets. we found a good correlation between the expression patterns observed by us and that reported by the affymetrix geenchip array. in addition, a cluster of endogenous reference genes suggested by applied biosystems for standardization during qrt-pcr was analyzed by evaluating the sd of their log ratios across the arrays. as data indicated in table although some genes demonstrated mean sd log ratios comparable to that of the endogenous reference genes identified in this study, the corresponding percentage of the arrays in which log ratios were within sd from zero ranged between – which is significantly lower than the frequency in which the same parameter fell for the candidate house keeping genes identified here. in addition, we looked at the variation in our arrays of genes classically used as endogenous reference genes including: ribosomal protein l hrpt, β-actin and tubulin-α in all cases the mean sd of log2ratio was relatively larger than for the endogenous reference genes identified in table with the following respective values: for ribosomal protein l for hrpt, for β-actin and for laminin-α interestingly, however, the mean intensities were relatively higher for all of these genes compared with the ones identified in this study suggesting that these genes are expressed probably at higher concentrations in tissue and, therefore, although relatively variable in expression in different tissues, they have been a useful marker of rna abundance when only rough estimates are required like, for instance, for northern blotting. to test the usefulness of various endogenous reference gene we resorted to the well characterized expression of the melanocytic lineage-specific gene gp100/pmel that is expressed exclusively though heterogeneously in cells of melanocytic lineage. the assumption of this experiment was that in spite of its heterogeneity of expression in samples of melanocytic origin, overall the expression of gp should be higher in meloanocytic compared with non melanocytic samples with a high degree of significance. rho-c has been associated with the metastatization process in melanoma but its specificity for melanocytic lineage is unknown. the differentiation control element dice is found in the 3'-utr of numerous eukaryotic mrnas and there is no solid association between its pattern of expression and specific physio-pathological or developmental conditions. therefore, we compared the expression profile of these three genes in cdna libraries that included melanoma cell lines, melanoma metastases, benign nevi and miscellaneous samples containing a predominance of primary esophageal, renal and colon cancers paired with normal tissues from the same organ and a large number of circulating mononuclear cells. in addition, samples from most other normal and cancerous tissues were included although in smaller number. as endogenous reference genes we chose β-actin and new candidate genes identified by this study. the test genes were separately hybridized to the transcriptome array. each gene labeled with cy was co-hybridized with individual endogenous reference genes labeled with cy we then compared log cy / cy ratios between melanocytic and non-melanocytic tissues. gp was, as expected, expressed more in melanocytic lesions with high degree of significance no matter what gp / endogenous reference gene combination was used. the range of significance, however, varied extensively depending upon the gp / endogenous reference gene combination. this difference was considered representative of the ability of different endogenous reference genes to normalize for tissue-specific gene expression patterns. in details, all endogenous reference genes could segregate melanocytic from non melanocytic lesions with a high degree of significance. there was, however, a big range in the discriminatory capacity among endogenous reference genes with nedd rab and fzd providing the highest predictive value and β-actin being one of the least reliable. the large variation in the results obtained using different endogenous reference genes to normalize gp expression could have been due to a higher stability of the expression of some genes across all samples or to a differential expression of the endogenous reference genes themselves in melanocytic lesion. in the latter case, the better results obtained could be coincidental and not useful in other experimental situations. we, therefore, tested whether pooling results obtained with all endogenous reference genes independently of each predictive value in this controlled experimental situation could yield results as informative about gp pattern of expression as those obtainable with individual endogenous reference genes, particularly those that provided the best prediction. the advantage of this strategy is that it does not depend on previous knowledge of gene expression patterns for the selection of individual endogenous reference genes not applicable in conditions, in which the suitability of a gene for a given experimental situation is, contrary to gp is unknown. thus, we compared the predictive value of the average of gp / endogenous reference gene log ratios obtained with all endogenous reference genes and those obtained using the genes that provided the best five or the best three results. the use of pooled information from various endogenous reference genes appeared to stabilize the non-melanocytic log ratios gp / endogenous reference. in fact, the sd of thelog ratios gp / endogenous reference among non-melanocytic samples derived by pooling endogenous reference gene results were significantly lower than the sd obtainable with any individual endogenous reference gene. the basis for this test was that non-melanocytic lesions uniformly should not express gp and, therefore, the log ratios for any gp / endogenous reference gene combination should be constant resulting in very low sd. sd for non-melanocytic lesions were much lower using pooled endogenous reference genes for the normalization and this was true at extreme levels of significance. similar results were obtained when results from the best and best endogenous reference genes were pooled together. in addition, even if pooling data together did not provide the highest level of discrimination compared to the best results obtained with selected individual endogenous reference genes, the capacity of pooled data sets to discriminate gp expression between melanocytic and non melanocytic samples was still very high. interestingly, different endogenous reference genes provided not only different levels of discrimination but also provided different estimates of gp expression in melanocytic lesions with gp / endogenous reference gene log ratio above or below. this, of course, has no impact on the ability to characterize patterns of expression of different genes in different tissues but rather suggests that the latter group of endogenous reference genes is expressed at higher copy number, therefore, biasing the data toward its own fluorescence channel. simple adjustment in probe concentration could easily solve this problem. we then analyzed the expression of rho-c and dice whose pattern of expression in the transcriptome array cannot be predicted based on available information save for the notion that of the expression of rho-c is related to the metastatic process of melanoma. the pattern of rho-c was characterized with all the endogenous reference genes and representative information is presented in figure most endogenous reference genes yielded a pattern suggestive of a preferential expression of rho-c in melanoma metastases. in addition, averages of rho-c / endogenous reference gene log2ratios demonstrated clearly a specific expression of this gene in melanoma metastases. this pattern was not necessarily specific for melanoma as the other cancerous tissues spotted in the transcriptome array were obtained from primary lesions. thus, it is possible that the preferential expression in melanoma metastases was due to the metastatic process rather than the melanocytic lineage. interestingly, the melanoma cell line mel-a did not constitutively express rho-c as previously observed by clark et al.. in addition, none of the melanoma cell lines mostly derived from melanoma metastases demonstrated expression of rho-c. this information suggests that rho-c may be involved in the natural metastatic process in vivo but is not constitutively expressed in vitro. finally, dice did not demonstrate any specificity of expression and appeared variably expressed in all specimens independently of the endogenous reference gene used for normalization. validation of data by protein analysis and quantitative real-time pcr we tested whether gp / endogenous reference gene log ratios correlated with gp protein expression level as measured by intra-cellular facs analysis. the gp / endogenous reference gene log ratios were derived from the transcriptome arrays while protein expression was based on previous characterizations of the same cell lines. overall, a good correlation was noted for gp values normalized with most endogenous reference genes. however, with the exception of nedd the best correlation was obtained when pooled information was used from the best and candidate endogenous reference genes as defined before. interestingly, rab that scored very high as a predictor of gp transcriptional expression in melanocytic lesions and β-actin provided the worst correlation with protein expression. this data suggest that, although different endogenous reference genes may yield better predictive value than others in large data sets, it is likely that their reliability varies in different conditions and possibly the best results can be obtained by pooling information from several of them. we also tested by quantitative real-time pcr the discriminatory capacity of endogenous reference genes in samples with known patterns of expression of gp we selected previously characterized melanoma cell lines four of which were known to express and two known not to express gp at the transcriptional and protein level. in addition, fine needle aspirate samples from melanoma metastases expressing gp and a series of tumors known not to express gp were tested. absolute expression of gp was measured and although the lesions supposed to be gp expressing appeared more positive than those supposed to be gp negative there was no absolute demarcation among the two groups. a better differentiation between positive and negative lesions could be obtained when the various endogenous reference genes were used for the analysis or the combined values from all of them. discussion although the concept of endogenous reference genes is appealing it may be unrealistic to expect that a gene could be equally expressed in all eukaryotic cells in all physiopathological conditions. thus, the endogenous reference gene concept can only approximate such ideal biological behavior. in fact, numerous studies noted that the expression of the ostensible endogenous reference genes varies according to distinct physio-pathological situations and can be affected by experimental manipulation. for instance, the levels of expression of gapdh, cyclophilin and β-actin fluctuate in different tissues, disease stage and are affected by cell behavior like proliferation. in addition, endogenous reference genes expression varies markedly in cancer compared to normal tissue suggesting that oncogenesis may influence their expression. in this study, therefore, we looked for relative stable endogenous reference gene challenging a cdna clone data base against approximately specimens that included a broad variety of normal and neoplastic tissues as well as cell lines. indeed, few previously unnamed genes approximated the ideal endogenous reference gene by being expressed consistently in more than 95% of the specimens analyzed within a variance that was barely and insignificantly different from what could be expected from intrinsic experimental fluctuation. in fact, the threshold that we selected based on the variance observed by periodically repeating identical experiments was consistent with criteria suggested by others who claimed that genes with fold changes greater than can be classified as differentially expressed at ~95% confidence interval. this is consistent to our selection criterion that contained endogenous reference genes within a fold maximal variation. although these genes have not been proposed before as candidate endogenous reference genes, analysis of available data bases suggested that they are relatively stably expressed in various normal tissues. selection of these candidate endogenous reference genes provided useful information when the expression of a lineage specific gene such as gp was compared between relevant and irrelevant samples. in addition, gp expression normalized according to some of the new endogenous reference genes more closely aligned to lineage specificity than normalization performed using β-actin. this data, however, confirm that there is no such thing as a perfect endogenous reference gene. although some endogenous reference genes demonstrated high power of discrimination their ranking was dictated by previous knowledge of the expected experimental results. however, gene expression analysis is done to determine such specificity and in most cases no previous knowledge of tissue specificity is truly available. one may argue that, in spite of the extreme variability in normalization power demonstrated by various endogenous reference genes, still significant differences could extracted between the melanocytic and non-melanocytic samples no matter what endogenous reference genes were used. this statement is correct in this experimental situation where hundreds of samples could be compared with the transcriptome array. however, in cases when only few samples are analyzed the difference in discriminatory capacity becomes critical. for instance, when randomly selected gp / endogenous reference gene log ratios from melanoma metastases were compared with those from kidney specimens only nedd rab fzd afap, col snrpd and pooled endogenous reference gene-based normalizations yielded significant differences between the two groups while non significant values were obtained for normalizations based on hirip β-actin and gpld the lowest limit of detection in the transcriptome array of specific probes in term of copy numbers is currently under investigation. decreasing concentrations of β-actin alone printed the array spots as internal controls suggest that the detection range is from ng / spot to pg / spot when standard fluorescent intensity range parameters are applied. the higher limit of detection before saturation can be easily adjusted to avoid under estimates of highly expressed genes, therefore, increasing the dynamic range of detection. this is achieved by adjusting the fluorescent intensity right below the saturation for both channel and subsequently calculating the abundance of reference gene expression using serially diluted internal control spots as standard curve. with few exceptions, the expression of most genes can be estimated with relative accuracy using this semi-quantitative approach. since in most cases endogenous reference genes are selected based on previous knowledge of their usefulness for a particular experimental situations, it is likely that the best consistency could be achieved in such circumstances when averages of several endogenous reference gene normalizations are used. this may represent a reasonable solution particularly if a cocktail of endogenous reference genes identically labeled could be used as reference within a single experiment. we are presently testing various combinations of lineage specific genes and pools of endogenous reference genes for hybridization to the transcriptome array and for qrt-pcr. such attempts, however, have been quite disappointing so far as results obtained using mixture of endogenous reference genes do not yield information comparable to that achievable when results obtained with the same endogenous reference genes used individually are pooled together. this is possibly due to imbalanced density of various endogenous reference genes in different samples with predominant effects of some over others. therefore, extensive work in the future will be aimed at equilibrating the interactions among endogenous reference genes in various experimental conditions to test whether mixtures of them could be used in multiplex for normalization of large expression analysis studies.  our observations, together with previous work by others indicate that much caution should be taken when using endogenous reference to normalize the expression of test genes. it is likely that previous analyses based on single endogenous reference genes might have been strongly biased by the individual selections and the interpretation of the results should looked at with caution. in particular, this study suggests that even when the most stably expressed genes in array experiments are used as endogenous reference genes, significant variation in test gene expression estimates may occur and the best normalization is achieved when data from several endogenous reference genes are pooled together to minimize minimal but significant variation among samples. we are presently optimizing strategies for the preparation of endogenous reference gene mixtures that could yield information comparable to that of data pooled from individual endogenous reference gene normalizations. 
the usefulness of genes among them was tested by normalizing the relative gene expression against test genes whose expression pattern was known. the range of accuracy of individual endogenous reference genes was wide whereas consistent information could be obtained when information pooled from different endogenous reference genes was used. endogenous reference genes are commonly used to normalize expression levels of other genes with the assumption that the expression of the former is constant in different tissues and in different physiopathological conditions. in this study, we searched for stably expressed genes in cdna array hybridization experiments encompassing different tissues and cell lines. this study suggests that even when the most stably expressed genes in array experiments are used as endogenous reference, significant variation in test gene expression estimates may occur and the best normalization is achieved when data from several endogenous reference genes are pooled together to minimize minimal but significant variation among samples. we are presently optimizing strategies for the preparation of endogenous reference gene mixtures that could yield information comparable to that of data pooled from individual endogenous reference gene normalizations.
 endogenous reference genes are commonly used to normalize expression levels of other genes with the assumption that the expression of the former is constant in different tissues and in different physiopathological conditions. whether this assumption is correct it is, however, still matter of debate. in this study, we searched for stably expressed genes in cdna array hybridization experiments encompassing different tissues and cell lines.  several genes were identified whose expression was highly stable across all samples studied. the usefulness of genes among them was tested by normalizing the relative gene expression against test genes whose expression pattern was known. the range of accuracy of individual endogenous reference genes was wide whereas consistent information could be obtained when information pooled from different endogenous reference genes was used.  this study suggests that even when the most stably expressed genes in array experiments are used as endogenous reference, significant variation in test gene expression estimates may occur and the best normalization is achieved when data from several endogenous reference genes are pooled together to minimize minimal but significant variation among samples. we are presently optimizing strategies for the preparation of endogenous reference gene mixtures that could yield information comparable to that of data pooled from individual endogenous reference gene normalizations. 