Che et al. BMC Genomics 2010, 11(Suppl 2):S1 http://www.biomedcentral.com/1471-2164/11/S2/S1 Table 3 Performance comparison among machine learning algorithms Method CART Naive Bayesian BayesianNet J48 Logistic Neural Network SVM 0.813 0.589 0.604 0.858 0.628 0.659 0.565 Sensitivity Specificity F-Measure Accuracy AUC 0.807 0.893 0.914 0.843 0.893 0.849 0.902 0.811 0.710 0.727 0.850 0.738 0.742 0.695 0.810 0.743 0.760 0.850 0.762 0.754 0.735 0.855 0.811 0.839 0.892 0.845 0.833 0.734 F-Measure, accuracy and AUC. Another decision tree approach, CART is ranked to the second best classifier. Ensemble learning algorithms can improve classification accuracy Using the J48 decision tree approach as the baseline for classification, we applied four decision-tree-based ensembles, adaBoost, bagging, MultiBoost, and random forest in classifying four datasets (Salmonella, Staphylo- coccus, Streptococcus and their mixed-up dataset). Again the WEKA package and default parameters were used. Table 4 lists five performance evaluation metrics of each ensemble algorithm on the mixed-up dataset. We found that, in general, all ensemble algorithms could improve classification. For the visualization purpose, we also provide ROC curves and their corresponding AUC values of four ensemble algorithms, as well as the J48 algorithm, on each of four datasets (See Figure 2). Bagging has the highest AUC values based on the evaluation of four datasets. Compared to those of a single J48 algorithm, the AUC values of the bagging algorithm are 5% (0.92 versus 0.87), 6% (0.89 versus 0.83), 4% (0.94 versus 0.90), and 5% (0.94 versus 0.89) higher in the models of Salmonella, Staphylococcus, Streptococcus, and mixed-up three, respectively. It is interesting to see that adaBoost performs the best among all algorithms in the dataset of Salmonella, but no improvements in the datasets of Sta- phylococcus and Streptococcus. Since one genome may contain several or up to doz- ens of GIs [6,15], we may unavoidably face the problem Table 4 Performance comparison between J48 and decision tree based ensembles Method J48 AdaBoost Bagging MultiBoost Random Forest Sensitivity Specificity F-Measure Accuracy AUC 0.858 0.890 0.870 0.880 0.819 0.843 0.910 0.872 0.871 0.889 0.850 0.902 0.873 0.876 0.859 0.850 0.900 0.871 0.876 0.850 0.892 0.932 0.940 0.942 0.908 The ensemble classifiers include bagging, AdaBoost, MultiBoost, random forest. Page 4 of 9 of collecting small datasets for GI classification. The datasets used in our study were from thirty-seven strains of three different genera, with 331 GIs and 337 non-GIs. The relative small datasets for the genera of Staphylo- coccus and Streptococcus led to the unsmooth ROC curves, as shown in Figure 2. As more genomes of the strains in these genera will be sequenced and the train- ing datasets become bigger, the ROC curves will be more accurately reflect our classification algorithms. We further investigated the contribution of each single feature by using “leave-one-feature-out” model, where in each experiment one feature was removed from all fea- ture model. Figure 3 lists the ROC curves and their cor- responding AUC values of the bagging algorithm on each of four datasets (See Additional file 2 for the corre- sponding one of adaBoost). By analyzing each feature using the “leave-one-feature-out” model, We found that the feature “Size” is very informative in the dataset of Salmonella (Figure 3), where we can see that the classi- fication power dropped by 11.3% of the AUC value when the Size feature was removed. The high contribu- tion of the Size feature in the model of Salmonella sug- gests that other GI-associated features do correlate with the Size feature. Another observation from these classifi- cation results is that a single feature can be dropped from the multiple features in many cases, without affect- ing the classification performance dramatically. For instance, the AUC values are 0.924, 0.924, 0.925 and 0.924 for the models without the feature of Integrase, Insertion point, Phage, and Repeat, while the AUV value for all feature model is 0.923. This analysis indicates the contributions of some feature values are redundant, even they are informative by applying the signal-to- noise feature analysis. Overall, when multiple features are considered together, the most informative features are the Size and IVOM features. This is in contrast with the signal-to- noise analysis, where the features of Integrase, Phage and Repeats are most informative. These results bring us an attention that the feature selection should be taken cautiously when applying the single feature analysis. Comparison to other approaches We compared the classification results of our J48-based bagging algorithm with a sequence composition based approach, AlienHunter [15]. We found the overall accu- racy of our approach on the genera of Salmonella, Sta- phylococcus and Streptococcus are 14% (0.87 versus 0.73), 8% (0.79 versus 0.71), 8% (0.86 versus 0.78) higher than that of AlienHunter, respectively. The superior per- formance of our approach to AlienHunter is that our approach uses multiple features, while AlienHunter uses the information of sequence composition only.