Table 13. Number of errors for each classifier when considering the ‘‘MCI’’ test set (47 samples). ‘‘MCI’’ test set Method ClassViaRegression Bayes Net j48 Ordinal Class Classifier Na¨ıve Bayes Na¨ıve Bayes Simple Na¨ıve Bayes Up. Simple Logistic Logistic LWL LMT NBTree Kstar MultiClass Classifier Multilayer Perceptron Random Committee Decorate Random Forest AdaBoost Part IB1 Ibk SMO PAM Average Agreement % 18 19 17 19 19 17 17 17 14 14 20 14 14 17 15 14.4 16.2 14.6 16.2 16 16 16 16 12 11 16.29 65% 10 17 19 17 17 17 17 17 14 14 20 14 17 11 21 13.7 16.1 15.5 16.2 13 16 16 16 15 12 16.11 66% 6 14 15 17 17 16 16 16 14 14 20 14 14 18 19 16.2 17.9 17.5 18.6 18 19 19 19 16 16 5 16 15 17 17 16 16 16 14 14 20 14 14 18 16 15.7 17.9 16.2 18.1 19 19 20 20 15 16 Alzheimer In Table 12, the same comparison is made but only considering the ‘‘AD’’ test set. Once again, it is possible to visualize the performance of the 5-protein signature, obtaining not only the best average result but also the best individual results, presenting 3 errors on 3 occasions. Finally, Table 13 presents the same analysis for the ‘‘MCI’’ test set. In this case the most remarkable observation is the lack of quality to predict MCI-AD. The improved performance of the largest signatures is related to the fact that the signatures have more genes, and because they were not trained to distinguish between MCI patients, the use of more proteins allows a slightly better performance. Nevertheless, even the best signature for this case (a 10-protein signature) presents a poor performance when compared with the previous results. Discussion In conclusion, it is clear that the experiment performed by Ray et al. provided an extremely useful dataset for the identification of Alzheimer’s disease biomarkers. We have uncovered a robust 5- protein signature with near 97% of accuracy to predict AD against non-demented controls using their data. Our signature has less than one third of the proteins than the one proposed in the original paper, and at least the same level of prediction performance. The next step on this important quest is to set up an independent experimental procedure that now considers samples with mild cognitive impairment (but without AD) in the training set. We do not agree with the methodology of using a training set without MCI to select biomarkers to differentiate between AD and MCI [1]. This has not been done and warrants further investigation. Only in this way we can uncover useful biomarkers to discriminate between AD and MCI. On the positive side, our methods reveal the true predictive potential of testing for Alzheimer’s disease using this panel of signalling proteins. We also believe that our methods show promise and warrant their application in other settings. It is clear that Alzheimer researchers can benefit directly from our identification of more robust biomarkers. The method is revealed to be useful, simple yet very powerful, and warrants its application in other multifactorial diseases. Methods Our methodology consisted of the application of an integrative data analysis method. We used four steps: a) abundance quantization, b) feature selection, c) literature analysis, d) selection of a classifier algorithm which is independent of the feature selection process. These steps were performed without using any of the test datasets. For the first two steps, we used the application of Fayyad and Irani’s discretization algorithm [7] for selection and quantization, which in turn creates an instance of the (alpha-beta)-k-Feature Set problem [8–10]. Fayyad and Irani’s method filtered only 14 out of 120 proteins of the training set (i.e. those proteins for which no threshold was selected were filtered out). After quantization, samples 7, 43 (AD, ‘‘Alzheimer’s Disease’’) and 48 (NDC, ‘‘Nondemented Control’’) of the training set were ‘‘in conflict’’, which means that they have quantized values (for all 14 proteins selected) which are the same although they belong to different classes. These conflicts are then removed, i.e. the three samples of the training set are eliminated and we apply our algorithms to the remaining 80 samples of the training set. Numerical solution of the (alpha-beta)-k-Feature Set problem led to the selection of only 10 proteins, Table 4. For a detailed explanation of the methods and other applications, 16.78 64% 16.77 64% The signature with the best performance on each classifier is highlighted in bold text. doi:10.1371/journal.pone.0003111.t013 In Table 9, a feature of the experiments it is worth commenting: all the signatures drop at least 30% in accuracy when considering the ‘‘MCI’’ dataset. This is understandable since the classifiers have no sample labelled ‘‘MCI’’ in the training set. The best overall result, considering both test sets, is obtained by the 6-protein and 5-protein signatures. They present 18 errors and in both signatures this result is obtained twice when using the LMT and Simple Logistic classifiers (Tables 7 and 8). In Table 10, the standard deviations of the number of errors are almost constant for all signatures, in all datasets. This reinforces our previous claim, the poor performance of the signatures on the ‘‘MCI’’ dataset is related to the fact that the signatures were not trained to identify between AD and MCI. To present the experiment results in another form, we compared the performance of each signature in each test. Table 11 presents the comparison between the signatures when considering all the test sets (‘‘AD’’+‘‘MCI’’) totalling 139 samples. It is remarkable that the 5-protein signature not only has a better average performance, but also presents the best result on 16 of the 24 algorithms used for classification (the number of errors highlighted in bold text indicates the best performance for this particular classifier). PLoS ONE | www.plosone.org 10 September 2008 | Volume 3 | Issue 9 | e3111