As observed in other high throughput analyses, e.g., microarray expression profiling, where the number of profiled features greatly exceeds that of the assayed samples, concurrent analysis of MALDI-TOF spectral peaks to evaluate null hypotheses for differential urine peptide biomarkers leads to the multiple-testing problem. To address the multiple-hypothesis testing problem, we estimated the FDR in concurrent statistical tests of peptide panels, of the same size as our biomarker panel; multiple permutated "random" training data sets were constructed. The class labels of our training samples in either cohorts of SJIA F and QOM/RD, or cohorts of SJIA F and KD/FI, were permutated 500 times such that each time every sample would be randomly assigned a new class label (SJIA F or QOM/RD in F and QOM/RD discrimination; SJIA F or KD/FI in SJIA F and KD/FI discrimination). For each of the 500 simulated "training" sets, NSC algorithm was applied to rank all the MALDI-TOF spectral peak features based upon their ability to discriminate the binary classes: SJIA F versus QOM/RD; and SJIA F versus KD/ FI, respectively. The NSC-selected top 17-peak features were then designated as the "panel" for LDA analysis. ROC analysis subsequently was used to calculate the AUC for this "falsely discovered panel". The AUC values of the 500 falsely discovered panels were standardized, and the density distribution was plotted in Figs. 2 and 3d. FDR was calculated as the number of AUC values greater than that of our 17-urine-peptide panel divided by the total number of AUC values of the "falsely" discovered panels. As shown in Figs. 2 and 3d , the FDRs of our urine peptide biomarker panel are estimated as 0.2% in SJIA F versus QOM/RD discrimination, and 0.2% in SJIA F versus KD/ FI discrimination respectively. These results support the notion that the discovery of our peptide biomarker panel is Fig. 3 Evaluation of the 17-peptide biomarker panel as a classifier of active SJIA versus inactive (quiescent or remitted) SJIA. a A logistic regression model was used to find a panel-based algorithm that minimizes the total classification error discriminating active systemic SJIA from inactive SJIA. The maximum estimated probabilities for each of the wrongly classified samples, are labeled with arrows. b A modified 2×2 contingency table shows the percentage of classifications that agreed with clinical diagnosis. c The discriminant analysis-derived prediction scores for each sample were used to construct a receiver operating characteristic (ROC) curve; 500 testing data sets, generated by bootstrapping, from the SJIA systemic flare, and inactive SJIA data were used to derive estimates of standard errors and confidence intervals for our ROC analysis. The plotted ROC curve is the vertical average of the 500 bootstrapping runs, and the box and whisker plots show the vertical spread around the average. d Distribution of the standardized ROC AUC values of the 500 falsely discovered panels upon the 500 classlabel permutated data set of the cohort of SJIA F and QOM/RD urine peptidomes. Examining all the 500 falsely discovered biomarker panel ROC AUC values, the number of falsely discovered same-size panels that have ROC AUC values greater than that of the original urine biomarker panel (represented by the red vertical line) dividing the total number of the "falsely discovered" biomarker panels led to the estimation of false discovery rate FDR unlikely to be the outcome of chance in multiple hypothesis testing.