Table 2. Number of errors from the 6-genes randomly selected signatures on the ‘‘AD’’ validation test set. Seed Number 76 144 121 83 33 51 162 37 136 60 Average Error Average Accuracy Seed Number 76 144 121 83 33 51 162 37 136 60 Average Error Average Accuracy S6-1 40 40 38 40 41 39 41 40 42 40 40.1 56.4% S6-11 32 30 29 31 32 30 33 31 32 31 31.1 66.2% S6-2 34 32 37 33 33 33 35 33 36 35 34.1 62.9% S6-12 26 29 25 27 24 25 23 25 29 26 25.9 71.8% S6-3 20 19 18 19 17 19 19 17 19 17 18.4 80.0% S6-13 30 36 30 31 32 34 30 31 31 33 31.8 65.4% S6-4 31 34 33 31 35 28 31 32 34 28 31.7 65.5% S6-14 16 17 17 17 17 18 17 17 19 15 17 81.5% S6-5 31 32 35 33 33 34 36 31 34 27 32.6 64.6% S6-15 37 41 41 44 40 41 37 40 43 41 40.5 56.0% S6-6 32 33 30 34 30 30 34 29 32 31 31.5 65.8% S6-16 39 43 37 35 35 38 33 35 35 36 36.6 60.2% S6-7 29 30 28 27 27 28 28 27 30 29 28.3 69.2% S6-17 33 32 37 30 35 32 35 32 32 31 32.9 64.2% S6-8 32 31 32 27 28 28 27 35 34 32 30.6 66.7% S6-18 34 36 34 35 36 35 36 37 39 38 36 60.9% S6-9 24 23 27 24 27 24 26 24 24 23 24.6 73.3% S6-19 32 35 35 33 32 33 35 35 34 32 33.6 63.5% Alzheimer S6-10 34 33 31 31 29 30 33 32 26 29 30.8 66.5% S6-20 24 24 23 23 23 23 23 22 27 24 23.6 74.3% The Random forest algorithm was used as classifier, for each signature 10 runs with different seeds were done. We used the WEKA software implementation, and the algorithm was allowed to generate 150 trees. The best and worst signatures are highlighted in bold text. This result shows what it is expected, that a 6-signature, when the biomarkers are randomly chosen, is performing significantly worse than the panel of 18 biomarkers selected by Ray et. al. Now the best result (81.5%) is worse than the average result of a random 18-signature (86%). doi:10.1371/journal.pone.0003111.t002 classification method allows us to find ‘‘good’’ 18-protein predictor with only a random selection procedure restricted to these 120 proteins. Table 3, Figure 1 and Figure 2 resume the experiment. It is remarkable that by choosing 18 proteins at random we were able to obtain a very good signature, at least for this classifier, under the conditions explained above. Perhaps the reason of obtaining such good signatures is that a smaller number of proteins, that all signatures have in common, is all that it is needed for predictive molecular signature. Figures 1 and 2 show the relation between the considered signatures with 18 and 6 proteins and the random ones. Computational studies: Results obtained with four different signatures We report all the results obtained using a set of 24 classifiers which have been selected from the Weka software suite [3], aiming at sampling different algorithmic methodologies in current practice. These classifiers are applied having as input the four different signatures with the same training set. To ensure reproducibility of our reported methods, no parameter was modified from the classifier’s default setting from Weka’s downloaded code. In this way we were not biasing the experiment with ad hoc parameter selection and we ensure the complete reproducibility of our claims. We are also aware that better results are possible when adjusting the parameters of each classifier considering only the samples of the training set. Table 3. Random experiments report. 18-gene random signatures 6-gene random signatures Average Error Best Signature (average) Worst Signature (average) Standard Deviation Accuracy Average 15.14 6.2 25.7 5.36 83.5% 30.59 17 40.5 6.21 66.7% The table shows the average results of the 20 random signatures for each size, also including the best and worst results and the standard deviation. The accuracy average is calculated considering the error average over the 92 samples of ‘‘AD’’ validation test set. doi:10.1371/journal.pone.0003111.t003 PLoS ONE | www.plosone.org 3 September 2008 | Volume 3 | Issue 9 | e3111