18 18 18 17 20 15 15 13 17 18 17 18 17 16 20 20 19 18 18 19 14 15 15 14 18 16 13 14 16 10 18 22 18 21 22 22 18 21 21 17 20 11 12 10 11 12 11 13 11 13 11 18 21 20 18 21 22 21 25 17 20 18 19 22 21 20 21 20 21 22 17 16 17 17 16 14 14 16 15 15 13 29 25 25 27 27 26 24 29 23 22 9 9 8 8 8 8 8 7 9 10 8.4 90.9% 18 17 19 18 16 17 21 20 20 18 20 18 15 18 17 20 18 14 18 16 11 13 11 13 11 12 14 10 10 10 10 7 9 7 8 9 7 8 5 9 10 11 8 12 11 8 8 9 10 9 9.6 15 16 15 15 15 17 15 15 14 16 76 144 121 83 33 51 162 37 136 60 76 144 121 83 33 51 162 37 136 60 Table 1. Number of errors from the 18-genes randomly selected signatures on the ‘‘AD’’ validation test set. Seed Number S18-1 S18-2 S18-3 S18-4 S18-5 S18-6 S18-7 S18-8 S18-9 S18-10 Average Error Average Accuracy Seed Number 16.9 81.6% S18-11 14.5 84.2% S18-12 11.5 87.5% S18-13 20.1 78.2% S18-14 25.7 72.1% S18-15 18.4 80.0% S18-16 11.5 87.5% S18-17 89.6% S18-18 6.2 93.3% S18-19 13.4 85.4% S18-20 4 7 7 6 6 6 7 7 5 7 20 19 22 19 22 22 23 23 23 21 Alzheimer 10 13 13 15 15 15 13 11 14 15 11 11 8 13 9 10 10 13 10 12 Average Error Average Accuracy 18.2 80.2% 78.3% 20.3 77.9% 15.3 83.4% 17.4 81.1% 7.9 91.4% 15.3 83.4% 21.4 76.7% 10.7 88.4% The Random forest algorithm was used as classifier. For each signature 10 runs with different seeds were done. We used the WEKA software implementation, and the algorithm was allowed to generate 150 trees. The best and worst signatures are highlighted in bold text. In two cases we found signatures that classify above 90%, comparable with the results of Ray et al. that report on 91% AD predictability as a result of their proposed methodology. doi:10.1371/journal.pone.0003111.t001 independence of the classifier and the robustness of this set of biomarkers (i.e. 96% accuracy when predicting AD against non- demented control). The 6-protein signature is composed of the abundances of IL- 1a, IL-3, IL-6, EGF, TNa and G-CSF. We remark that IL-6 was not selected by Ray et al. in the preliminary gene selection, and as a consequence it is not part of their 18-protein signature. Recognising that the importance of IL-6 as a biomarker for AD is debatable and that many classifiers do not make use of its abundance to inform decisions, we also present our results of a 5- protein signature that ignores IL-6. Results Base case–analysis of the performance of randomly selected signatures Before reporting our experimental results, it was important to understand the worst possible performance results that a set of k proteins can have when they are selected at random (from the available 120 proteins under study). We showed results of two experiments that aim at quantifying this. We showed the classification performance of 20 signatures with 18 proteins selected at random with a uniform distribution (obviously, we have selected 18 as is the same number of proteins as the signature proposed by Ray et al.). Analogously, we performed the same experiment now constrained to select only six proteins chosen at random (as we will later present comparative results using signatures that only employ 6 and 5 proteins). The two different collections of 20 sets of randomly generated signatures were chosen using an equal probability for each of the 120 proteins in the set (obviously, not allowing repetitions and constrained to have either 18 or 6 different proteins in total). For this experiment, we decided to use a random forests algorithm (RF) as a base classifier (we are using the algorithm implemented in [3] for reproducibility purposes), generating 150 trees. As the chosen classifier also has a stochastic nature, for each signature we ran 10 experiments with different seeds, and the results we found are quite interesting. For these twenty 18-protein signatures the average error over the 92 samples considered on the ‘‘AD’’ test set, is 15.13 meaning an 84% effectiveness, see Table 1. For the 6-protein case, an average error of 30.5% was observed meaning that an expected lower value of 67% effectiveness was found, see Table 2. With these results we can infer that the original selection of the 120 genes is quite remarkable for revealing biomarkers for prediction of clinical AD. Since a random selection with a simple, yet robust, PLoS ONE | www.plosone.org 2 September 2008 | Volume 3 | Issue 9 | e3111