BACKGROUND
expression microarrays, with the capability to measure the mrna expression level of tens of thousands of genes simultaneously, have found broad application in both clinical and basic research  <cit> .
with the generation of large data sets from microarray experiments, statistical methods are needed to extract useful information.
many methods have had specific implementations written for the analysis of gene expression data, such as various forms of clustering, self ordered maps, singular value decomposition and significance analysis  <cit> .
however, these methods all rely on the assumption that there are no gross process errors in the original data.
previous analyses of systematic errors in microarray data have focused on problems at the level of sample preparation, labelling, or hybridization.
this report focuses on steps in the microarray production process prior to hybridization that may ultimately result in errors in the underlying data.
much of the microarray data published at stanford is based on mouse and human arrays produced under controlled and monitored conditions at the brown and botstein laboratories and at the stanford functional genomics facility .
nevertheless, as large datasets based on the stanford human array began to accumulate, a small but significant number of discrepancies were detected that required a serious attempt to track down the original source of error.
due to a controlled process environment, sufficient data was available to accurately track the entire process leading to up to the final expression data.
in this paper, we describe our statistical methods to detect the inconsistencies in microarray data that arise from process errors, and discuss our technique to locate and fix these errors.
we are able to fix those errors that originate at the level of any microtiter plate used during a multi-step microarray production process.
the major process fail points in cdna microarray production at stanford are shown in table  <dig>  it is at these points that misidentifications can occur.
other types of processes resulting in expression data will have their own possible fail points.
regardless of the particular process, for the sake of subsequent error checking it is important to track every instance where samples have moved from one microtiter plate to another, or from microtiter plate to microarray.
our process involves four such reallocations: from an archival 96-well block to a 96-well growth block, then to a 96-well pcr plate, then to four 384-well print plates and finally to  <dig> 10– <dig> thousand element microarrays.
even in highly automated bar-coded and vision controlled systems the possibility exists that plates might become swapped, skipped, misordered, or rotated by  <dig> degrees during one of the process steps.
our own process is fairly well automated, but like most academic and commercial efforts our process involves hand loading of robots.
for example in our case, during the transfer from  <dig> to 384-well format, it is possible to accidentally rotate a 96-well plate or misorder the  <dig> or 384-well plates.
during printing a 384-well plate might be accidentally skipped, swapped or printed in the wrong orientation .
even with the best engineering controls to prevent plate rotations, the potential exists for misidentified plates.
inconsistencies in our data were first detected both by visual inspection of microarray data as well as the appearance of anomalously large first components in singular value decomposition analyses of ovarian tumour data  <cit>  traced to different production batches of arrays.
our algorithm was developed to detect and repair these types of errors, using the similarities in expression levels between sets of spots from different microarrays.
the algorithm was used to check all of the microarray data in our database for which there was a sufficient process record.
the general idea, illustrated in figure  <dig>  is as follows: partition the microarray expression data from a single microarray into sets based on the various microtiter plates the samples have been in throughout their process history.
for example, at stanford we keep our process fairly simple, with a minimum of plate changes, so our spotting material can be said to have existed in either  <dig> well or 384-well format during its entire process history.
thus, we partition the data into sets corresponding to the  <dig> or 384-well plate history.
next, an expression vector for each element of the partition  based on absolute  expression levels is formed and compared to every other expression vector from every other plate on many other array batches.
a rank matrix of correlation coefficients is formed which should be close to unity on the diagonal and far from unity off the diagonal.
non-unitary diagonal elements indicate problems with that plate comparison.
a rank comparison of the best correlations can be made to find swapped plates.
the algorithm can be repeated assuming a rotated plate to check to see if the discrepancy can be attributed to a plate rotation.
it should be noted that in cases where no process error is indicated, the method can still indicate the presence of problem plates, print batches or pcr rounds that should be flagged for particular attention in downstream analyses.
our algorithm  is for arrays produced in the brown and botstein laboratories and the stanford functional genomics facility.
however, the ideas are general and can be applied to many other types of high throughput operations.
in most of the research studies using our human microarray, a common reference  <cit>  is compared against a tumour or tissue specimen.
the common reference is usually labelled with the cy <dig> dye, pseudo-colour green in most visualizations of the data, while the sample specimens are labelled with cy <dig> dye, pseudo-colour red.
subtle corrections due to background subtraction issues and normalization are not important for this analysis.
because a common reference is used in a large number of hybridization experiments at stanford, all of the cy <dig> intensities  across various kinds of experiments are comparable.
we measure the similarities of two sets of spots by taking the correlations between the common reference intensities of these two sets of spots.
for those experiments that do not use common reference, the comparison is made using experiments with similar samples in either the cy <dig> or cy <dig> channel.
RESULTS
finding misidentified data
this example is from the analysis of experiments from an ovarian tumor dataset <cit>  that first led us to develop mufu.
here, it was noticed that similar tissues were not clustering across batch boundaries as expected.
also, an anomalously large first component in a singular value decomposition analysis <cit>  pointed to problems at the array batch level.
after some detective work, visual inspection uncovered anomalies in certain print plates as seen in figure  <dig>  we were able to isolate the problem to distinct sets of 96-well plates that had been swapped during an upstream process step, probably during the transfer from 96-well plates to 384-well plates.
not wishing to go through this sort of process again and again, mufu was developed to more succinctly recapitulate this finding and apply it to all data.
with mufu, a check using a 384-well plate partition of the data shows a discrepancy but no obvious plate rotation or misidentification.
we then repartition and repeat our tests at the 96-well plate level.
the results are shown in table 2a where we show a  <dig> way pair wise comparison.
the first four comparisons are for batches whose print plates are made from the first pcr round.
the middle four compare the first pcr round to the second, and the last four compare batches from the second pcr round.
mismatches are evident in the middle set of comparisons.
from table 2b we see that the distance matrix identifies a match between plate h and plate i and plate n and plate o for all four comparisons of the first pcr round to the second.
in all four comparisons, the two distributions resolve themselves well, as can be seen in figure  <dig>  leading to the conclusion that h and i are swapped, and n and o are swapped, most likely in the plates from the first pcr round.
the ambiguity is broken by sequencing a small sampling of wells, which confirms that the misidentifications are in the print plates from the first round of pcr, and not the second.
in figure  <dig> we show the effect on the ovarian data after the correction has been applied.
also, in figure  <dig> we show how, via visual inspection of the actual spots on an array, one can verify that the fix has properly reorganized the data.
the figure shows spots from the four different affected 96-well plates before and after the fix is applied.
finding a 384-well plate rotation
we compare four arrays a <dig>  a <dig>  a <dig> and a <dig> from a particular print production batch a to four arrays b, c, d and e from four other distinct print production batches.
the total number of  <dig> well microtiter plates in print batch a is  <dig>  the results of the plate rotation test are shown in table 3a.
here we see plate p, identified in both the rank and rotated-rank matrices across print batches is a clear candidate for a plate rotation.
the data from the distance matrix comparisons is shown in table 3b.
in figure  <dig> we show the distributions of the diagonal elements of the distance matrix and the rotated-distance matrix.
these distributions are well resolved and allow us to easily distinguish data due to a plate rotation from noisy data.
finding a misidentified 384-well plate
in this example we compare four arrays a <dig>  a <dig>  a <dig> and a <dig> from a particular print batch a to four other arrays b, c, d and e from four other distinct print batches.
as seen in table 4a no plate is found to be a candidate for plate rotation, however we do find that plate q has a poor self-match comparison.
indeed, when the distance matrix is examined, we see the plate matches instead plate r across the four array comparisons.
we check the three criteria for plate misidentification and summarize these data in table 4b.
in figure  <dig> we show the distributions of the match and mismatch distributions.
these distributions are quite distinct and give us confidence that we can distinguish the proper match from an accidental match.
discussion
out of the  <dig> million features checked in our database using mufu, problems were detected and corrected on  <dig>  million of them.
that we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm.
that so few problems existed overall  reassures us as to the robustness of our process in general.
microarray data, by its sheer volume, presents interesting laboratory information management challenges.
the data arrive at the investigators desk after a significant number of steps.
we have found that the better you track production, quality control, and experimental steps, the better chance you have of uncovering the reasons behind discrepancies that may appear in the data.
often, statistical analyses look only at the data presented as final expression values or ratios, without taking into account relevant quality control data.
in our effort to understand our errors and the source of large systematic discrepancies in our data we have found the mufu algorithm a useful test of certain classes of process errors, and as a check for general problems with specific process steps or microtiter plates.
we use mufu to find, verify and fix problems that can be attributed to an error in plate processing.
we also flag plates for which we can find no specific problem but we see yield inconsistent results.
these may be plates that, at some point in the process, had a problem  that was not detected while the process step was being carried out.
the fact that we can test the data, using our standard quality control hybridizations, for these types of quality issues is reassuring and has helped us gain confidence in our data.
obviously, there are many other classes of error that creep into microarray data.
aside from the gross process errors that are amenable to detection, as we have described here, there is also a large class of more subtle systematic errors that can contribute to the overall systematic error on the expression ratio measurement.
isolating the source of these individual errors is sometimes quite difficult.
properties of the microarray feature such as spot size, shape and uniformity can contribute, but the majority of systematic errors are introduced at the time the experiment is performed.
slide post processing, rna quality, labelling, hybridization and washing all lend the possibility for introducing systematic errors.
improvements in protocols and hybridization apparatus have helped reduce these errors and should continue to do so in the future.
as these sources of error are identified and eliminated, expression microarrays will continue to provide progressively more sensitive measures of gene expression.
CONCLUSIONS
process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis.
we have shown the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays.
we generalized our procedure using a simple heuristic, which found and fixed several problems with the proper assignment of gene identifier with physical microarray feature.
we found thirteen print runs  that had four plates mistracked, six print runs with single  <dig> plate rotations, and one instance of a plate rotation at the  <dig> well plate level.
one skipped plate was detected, as well as one plate printed twice.
out of the  <dig> million features checked in our database using mufu, problems were detected and corrected on  <dig>  million of them.
that we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm.
that so few problems existed overall  reassures us as to the robustness of our process in general.
a list of corrected arrays can be found at .

 expression microarrays, with the capability to measure the mrna expression level of tens of thousands of genes simultaneously, have found broad application in both clinical and basic research. with the generation of large data sets from microarray experiments, statistical methods are needed to extract useful information. many methods have had specific implementations written for the analysis of gene expression data, such as various forms of clustering, self ordered maps, singular value decomposition and significance analysis. however, these methods all rely on the assumption that there are no gross process errors in the original data. previous analyses of systematic errors in microarray data have focused on problems at the level of sample preparation, labelling, or hybridization. this report focuses on steps in the microarray production process prior to hybridization that may ultimately result in errors in the underlying data. much of the microarray data published at stanford is based on mouse and human arrays produced under controlled and monitored conditions at the brown and botstein laboratories and at the stanford functional genomics facility. nevertheless, as large datasets based on the stanford human array began to accumulate, a small but significant number of discrepancies were detected that required a serious attempt to track down the original source of error. due to a controlled process environment, sufficient data was available to accurately track the entire process leading to up to the final expression data. in this paper, we describe our statistical methods to detect the inconsistencies in microarray data that arise from process errors, and discuss our technique to locate and fix these errors. we are able to fix those errors that originate at the level of any microtiter plate used during a multi-step microarray production process. the major process fail points in cdna microarray production at stanford are shown in table it is at these points that misidentifications can occur. other types of processes resulting in expression data will have their own possible fail points. regardless of the particular process, for the sake of subsequent error checking it is important to track every instance where samples have moved from one microtiter plate to another, or from microtiter plate to microarray. our process involves four such reallocations: from an archival 96-well block to a 96-well growth block, then to a 96-well pcr plate, then to four 384-well print plates and finally to 10– thousand element microarrays. even in highly automated bar-coded and vision controlled systems the possibility exists that plates might become swapped, skipped, misordered, or rotated by degrees during one of the process steps. our own process is fairly well automated, but like most academic and commercial efforts our process involves hand loading of robots. for example in our case, during the transfer from to 384-well format, it is possible to accidentally rotate a 96-well plate or misorder the or 384-well plates. during printing a 384-well plate might be accidentally skipped, swapped or printed in the wrong orientation. even with the best engineering controls to prevent plate rotations, the potential exists for misidentified plates. inconsistencies in our data were first detected both by visual inspection of microarray data as well as the appearance of anomalously large first components in singular value decomposition analyses of ovarian tumour data traced to different production batches of arrays. our algorithm was developed to detect and repair these types of errors, using the similarities in expression levels between sets of spots from different microarrays. the algorithm was used to check all of the microarray data in our database for which there was a sufficient process record. the general idea, illustrated in figure is as follows: partition the microarray expression data from a single microarray into sets based on the various microtiter plates the samples have been in throughout their process history. for example, at stanford we keep our process fairly simple, with a minimum of plate changes, so our spotting material can be said to have existed in either well or 384-well format during its entire process history. thus, we partition the data into sets corresponding to the or 384-well plate history. next, an expression vector for each element of the partition based on absolute expression levels is formed and compared to every other expression vector from every other plate on many other array batches. a rank matrix of correlation coefficients is formed which should be close to unity on the diagonal and far from unity off the diagonal. non-unitary diagonal elements indicate problems with that plate comparison. a rank comparison of the best correlations can be made to find swapped plates. the algorithm can be repeated assuming a rotated plate to check to see if the discrepancy can be attributed to a plate rotation. it should be noted that in cases where no process error is indicated, the method can still indicate the presence of problem plates, print batches or pcr rounds that should be flagged for particular attention in downstream analyses. our algorithm is for arrays produced in the brown and botstein laboratories and the stanford functional genomics facility. however, the ideas are general and can be applied to many other types of high throughput operations. in most of the research studies using our human microarray, a common reference is compared against a tumour or tissue specimen. the common reference is usually labelled with the cy dye, pseudo-colour green in most visualizations of the data, while the sample specimens are labelled with cy dye, pseudo-colour red. subtle corrections due to background subtraction issues and normalization are not important for this analysis. because a common reference is used in a large number of hybridization experiments at stanford, all of the cy intensities across various kinds of experiments are comparable. we measure the similarities of two sets of spots by taking the correlations between the common reference intensities of these two sets of spots. for those experiments that do not use common reference, the comparison is made using experiments with similar samples in either the cy or cy channel.  finding misidentified data this example is from the analysis of experiments from an ovarian tumor dataset that first led us to develop mufu. here, it was noticed that similar tissues were not clustering across batch boundaries as expected. also, an anomalously large first component in a singular value decomposition analysis pointed to problems at the array batch level. after some detective work, visual inspection uncovered anomalies in certain print plates as seen in figure we were able to isolate the problem to distinct sets of 96-well plates that had been swapped during an upstream process step, probably during the transfer from 96-well plates to 384-well plates. not wishing to go through this sort of process again and again, mufu was developed to more succinctly recapitulate this finding and apply it to all data. with mufu, a check using a 384-well plate partition of the data shows a discrepancy but no obvious plate rotation or misidentification. we then repartition and repeat our tests at the 96-well plate level. the results are shown in table 2a where we show a way pair wise comparison. the first four comparisons are for batches whose print plates are made from the first pcr round. the middle four compare the first pcr round to the second, and the last four compare batches from the second pcr round. mismatches are evident in the middle set of comparisons. from table 2b we see that the distance matrix identifies a match between plate h and plate i and plate n and plate o for all four comparisons of the first pcr round to the second. in all four comparisons, the two distributions resolve themselves well, as can be seen in figure leading to the conclusion that h and i are swapped, and n and o are swapped, most likely in the plates from the first pcr round. the ambiguity is broken by sequencing a small sampling of wells, which confirms that the misidentifications are in the print plates from the first round of pcr, and not the second. in figure we show the effect on the ovarian data after the correction has been applied. also, in figure we show how, via visual inspection of the actual spots on an array, one can verify that the fix has properly reorganized the data. the figure shows spots from the four different affected 96-well plates before and after the fix is applied. finding a 384-well plate rotation we compare four arrays a a a and a from a particular print production batch a to four arrays b, c, d and e from four other distinct print production batches. the total number of well microtiter plates in print batch a is the results of the plate rotation test are shown in table 3a. here we see plate p, identified in both the rank and rotated-rank matrices across print batches is a clear candidate for a plate rotation. the data from the distance matrix comparisons is shown in table 3b. in figure we show the distributions of the diagonal elements of the distance matrix and the rotated-distance matrix. these distributions are well resolved and allow us to easily distinguish data due to a plate rotation from noisy data. finding a misidentified 384-well plate in this example we compare four arrays a a a and a from a particular print batch a to four other arrays b, c, d and e from four other distinct print batches. as seen in table 4a no plate is found to be a candidate for plate rotation, however we do find that plate q has a poor self-match comparison. indeed, when the distance matrix is examined, we see the plate matches instead plate r across the four array comparisons. we check the three criteria for plate misidentification and summarize these data in table 4b. in figure we show the distributions of the match and mismatch distributions. these distributions are quite distinct and give us confidence that we can distinguish the proper match from an accidental match. discussion out of the million features checked in our database using mufu, problems were detected and corrected on million of them. that we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm. that so few problems existed overall reassures us as to the robustness of our process in general. microarray data, by its sheer volume, presents interesting laboratory information management challenges. the data arrive at the investigators desk after a significant number of steps. we have found that the better you track production, quality control, and experimental steps, the better chance you have of uncovering the reasons behind discrepancies that may appear in the data. often, statistical analyses look only at the data presented as final expression values or ratios, without taking into account relevant quality control data. in our effort to understand our errors and the source of large systematic discrepancies in our data we have found the mufu algorithm a useful test of certain classes of process errors, and as a check for general problems with specific process steps or microtiter plates. we use mufu to find, verify and fix problems that can be attributed to an error in plate processing. we also flag plates for which we can find no specific problem but we see yield inconsistent results. these may be plates that, at some point in the process, had a problem that was not detected while the process step was being carried out. the fact that we can test the data, using our standard quality control hybridizations, for these types of quality issues is reassuring and has helped us gain confidence in our data. obviously, there are many other classes of error that creep into microarray data. aside from the gross process errors that are amenable to detection, as we have described here, there is also a large class of more subtle systematic errors that can contribute to the overall systematic error on the expression ratio measurement. isolating the source of these individual errors is sometimes quite difficult. properties of the microarray feature such as spot size, shape and uniformity can contribute, but the majority of systematic errors are introduced at the time the experiment is performed. slide post processing, rna quality, labelling, hybridization and washing all lend the possibility for introducing systematic errors. improvements in protocols and hybridization apparatus have helped reduce these errors and should continue to do so in the future. as these sources of error are identified and eliminated, expression microarrays will continue to provide progressively more sensitive measures of gene expression.  process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis. we have shown the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays. we generalized our procedure using a simple heuristic, which found and fixed several problems with the proper assignment of gene identifier with physical microarray feature. we found thirteen print runs that had four plates mistracked, six print runs with single plate rotations, and one instance of a plate rotation at the well plate level. one skipped plate was detected, as well as one plate printed twice. out of the million features checked in our database using mufu, problems were detected and corrected on million of them. that we were able to find and correct both previously known and unknown problems gives us confidence in the algorithm. that so few problems existed overall reassures us as to the robustness of our process in general. a list of corrected arrays can be found at. 
due to a controlled process environment, sufficient data was available to accurately track the entire process leading to up to the final expression data. we show the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays. much of the microarray data published at stanford is based on mouse and human arrays produced under controlled and monitored conditions at the brown and botstein laboratories and at the stanford functional genomics facility. to date, the brown and botstein laboratories and the stanford functional genomics facility have together produced large-scale cdna microarrays. nevertheless, as large datasets based on the stanford human array began to accumulate, a small but significant number of discrepancies were detected that required a serious attempt to track down the original source of error. in this paper, we describe our statistical methods to detect the inconsistencies in microarray data that arise from process errors, and discuss our technique to locate and fix these errors. process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis.
 much of the microarray data published at stanford is based on mouse and human arrays produced under controlled and monitored conditions at the brown and botstein laboratories and at the stanford functional genomics facility. nevertheless, as large datasets based on the stanford human array began to accumulate, a small but significant number of discrepancies were detected that required a serious attempt to track down the original source of error. due to a controlled process environment, sufficient data was available to accurately track the entire process leading to up to the final expression data. in this paper, we describe our statistical methods to detect the inconsistencies in microarray data that arise from process errors, and discuss our technique to locate and fix these errors.  to date, the brown and botstein laboratories and the stanford functional genomics facility have together produced large-scale cdna microarrays. by applying the heuristic described here, we have been able to check most of these arrays for misidentified features, and have been able to confidently apply fixes to the data where needed. out of the million features checked in our database, problems were detected and corrected on million of them.  process errors in any genome scale high throughput production regime can lead to subsequent errors in data analysis. we show the value of tracking multi-step high throughput operations by using this knowledge to detect and correct misidentified data on gene expression microarrays. 