Predictor CDK Best element predictor (PredPhospho) Unweighted voting Meta-predictor Best unreduced weighted voting Meta-predictor Best reduced weighted voting Meta-predictor CK2 Best element predictor (NetPhosK_0.5) Unweighted voting Meta-predictor Best unreduced weighted voting Meta-predictor Best reduced weighted voting Meta-predictor PKA Best element predictor (PredPhospho) Unweighted voting Meta-predictor Best unreduced weighted voting Meta-predictor Best reduced weighted voting Meta-predictor PKC Best element predictor (PPSP_balanced) Unweighted voting Meta-predictor Best unreduced weighted voting Meta-predictor Best reduced weighted voting Meta-predictor 0.853 0.853 0.857 0.863 0.871 0.809 0.844 0.867 0.827 0.820 0.837 0.839 0.741 0.733 0.744 0.772 CDK CK2 PKA PKC 0.912 0.878 0.883 0.773 GPS KinasePhos NetPhosK PPSP PredPhospho Scansite Max of the six category Meta-predictor (weighted voting with restricted grid search) 0.832 0.904 0.828 0.791 CDK 0.8761 0.8713 0.7767 0.8721 0.8670 0.7584 GPS 0.8761 0.8956 Predicting performance assessed on MetaPS06 datasets. P-values in Fisher’s Z-transformation test (compared with the MCC of the best element predictor) are shown in parentheses. Table 7. Areas under the ROC curves for the six element predicting programs and the weighted voting meta-predictor with restricted grid search MCC 0.708 0.699 0.711 (0.45) 0.726 (0.24) 0.730 0.617 0.675 0.722 0.642 0.477 0.433 0.500 0.620 0.669 (0.16) 0.675 (0.11) 0.521 (0.11) 0.864 0.893 0.850 0.784 0.730 (0.19) 0.779 (0.027) 0.699 (0.014) 0.558 (0.011) CK2 0.8130 0.7508 0.9307 0.8767 0.7791 0.7734 NetPhosK 0.9307 0.9313 PKA 0.8446 0.8234 0.8749 0.8860 0.8537 0.7656 PKA 0.8896 0.8946 PKC 0.7574 0.7440 0.7581 0.7994 0.7149 0.6397 PPSP 0.7994 0.8247 PAGE 9 OF 11 Nucleic Acids Research, 2008, Vol. 36, No. 4 e22 Table 5. Predicting performance of unweighted voting, best unreduced weighted voting and best reduced weighted voting meta-predictors Table 6. Predicting performance of weighted voting meta-predictors with restricted grid search of parameters Accuracy Sensitivity Speciﬁcity Accuracy MCC Predicting performance assessed on MetaPS06 datasets. Meta- predictors having predicting performance exceeding that of the best element predictor are shown in italic. P-values in Fisher’s Z-transformation test (compared with the MCC of the best element predictor) are shown in parentheses. values than that of the best of the element predictors (P < 0.05, Fisher’s Z-transformation test, see Table 6). Moreover, an ROC-based comparison indicated that the meta-predictors had higher ROC areas than those of any element predicting programs for all four kinase families (Table 7). The minimal and maximal improvements in accuracy and MCC (across the four kinase families) achieved by the meta-predictors over the best predictor for each element predicting program are presented separately in Table 8. The parameters selected in the four ﬁnal weighted voting meta-predictors are shown in Table 9. For each of the four meta-predictors, at least eight non-zero weight parameters were selected. At least one non-zero weights were used for all but two element predictors (NetPhosK_0.7 and PPSP_highspec) in the four ﬁnal meta-predictors, indicating that the good performance achieved by these meta-predictors was due to their ability to harness the combined strengths of multiple element predictors. A web server implementing the four ﬁnal meta- predictors was established and is accessible at http:// MetaPred.umn.edu/MetaPredPS/. Table 8. Minimal and maximal improvements in accuracy and MCC achieved by the weighted voting meta-predictor with restricted grid search over the best predictor of each element predicting program Minimal Maximal Minimal Maximal improvement improvement improvement improvement in accuracy in accuracy in MCC in MCC GPS 0.020 KinasePhos 0.042 NetPhosK 0.022 PPSP 0.025 PredPhospho 0.011 Scansite 0.042 0.077 0.115 0.159 0.041 0.080 0.100 0.035 0.098 0.049 0.043 0.022 0.103 0.166 0.249 0.343 0.081 0.163 0.243 Minimal and maximal improvements in accuracy and MCC were calculated across the four datasets for CDK, CK2, PKA and PKC kinase families. DISCUSSION Combination and reduced voting strategies We found that a weighted voting strategy with parameters selected by a grid search scheme produced satisfactory meta-predictors whose performance exceeds that of all element predictors for all four kinase families. In contrast, the combination strategy and the reduced weighted voting strategy (with weights set by the accuracy or MCC of the element predictors) failed to yield meta-predictors with satisfactory predicting performance for at least one