Re: The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer.

Receiver operating characteristic (ROC) curves are a powerful and fl exible tool to identify differentially expressed genes in microarray experiments. This use was discussed by Baker ( 1 ) in reference to a study by Pepe et al. ( 2 ), and was described by us ( 3 ), to illustrate a method of selecting cut-off values to separate different groups of cancer patients. Moreover, Baker indicated an innovative strategy to identify cancer patients that was based on grouping genes selected on the basis of the partial area under the ROC curve (pAUC) and suggested that multiple methods could be developed to generate classifi cation rules. In our opinion, this strategy is promising and should not be limited to the framework of the early detection of cancer. In fact, using the more conventional whole AUC for gene selection, bidirectional rules can be obtained, allowing identifi cation of different classes of cancer patients or of subjects differentially exposed to carcinogenic compounds. To assess the latter hypothesis, we developed a classifi cation method that combined cut-off values at the highest observed specifi city, as illustrated in Fig. 1 , using two hypothetical genes. In the fi rst step, the combination of a few genes with high AUC is identifi ed, which allows the best separation between the two groups. In the next steps, the same procedure is repeated recursively on the remaining genes to generate a panel of classifi cation rules. For each class, a score proportional to the AUC is then assigned to each sample. The scores generated at each step are summed with the previous scores to yield, in each rule, the genes with the highest AUC. This method was applied to the data from two online databases ( 4 , 5 ), and the results were validated by complete leaveone-out cross-validation ( 6 ) . The fi rst data set ( 4 ), generated from 34 current smokers and 23 nonsmokers, used Affymetrix chips and included 7329 genes ( http:// pulm.bumc.bu.edu/aged/index.html ). The second data set ( 5 ) used a specialized cDNA array (Lymphochip), designed at the Stanford University School of Medicine, and evaluated the expression of 4096 genes in neoplastic and nonneoplastic cells ( http://llmpp.nih.gov/lymphoma/ data/fi gure1 ). The analysis was restricted to the two largest groups of patients (42 patients with diffuse large-cell lymphomas and 20 patients with other lymphatic malignancies). After excluding missing values, 1332 genes were retained in the analyses. Using the conventional 0.05 P value, adjusted for multiple comparisons by Bonferroni’s correction, 94 hallmark genes were extracted from the fi rst database ( 4 ) and 444 from the second database ( 5 ) . In the fi rst database, three samples (5.3%) were misclassifi ed during cross-validation at the fi rst step (using two genes) whereas the error rate fell to 1.8% (one sample) during steps 3 to 20, corresponding to the inclusion of 12 to 90 genes. In the second database, cross validation misclassifi cations ranged between three (4.8%) and zero samples during the fi rst nine steps (1 to approximately 20 genes) and between one (1.6%) and zero samples during the next steps. After step 18, the number of misclassifi cations increased. The high performance observed in this study (zero to one misclassifi cations during cross validation for each database) indicates that, as suggested by Baker ( 1 ), ROC analysis represents a powerful and reliable tool for supervised analyses of microarray data.