e22 Nucleic Acids Research, 2008, Vol. 36, No. 4 PAGE 8 OF 11 Table 4. Predicting performance of combinatorial meta-predictors Number of element predictors in combination Element predictors included in best combinatorial meta-predictor Accuracy MCC CDK 2 3 4 5 6 CK2 2 3 4 5 6 PKA 2 3 4 5 6 PKC 2 3 4 5 6 GPS, PredPhospho GPS, NetPhosK_0.3, PredPhospho GPS, NetPhosK_0.3, PPSP_highsens, PredPhospho GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho, Scansite_low Best element predictor (PredPhospho) NetPhosK_0.3, PPSP_balanced GPS, NetPhosK_0.3, PPSP_balanced GPS, NetPhosK_0.3, PPSP_highsens, Scansite_low GPS, KinasePhos_90, NetPhosK_0.3, PPSP_highsens, Scansite_low GPS, KinasePhos_90, NetPhosK_0.3, PPSP_highsens, PredPhospho, Scansite_low Best element predictor (NetPhosK_0.5) NetPhosK_0.3, PredPhospho GPS, NetPhosK_0.3, PPSP_highsens GPS, NetPhosK_0.3, PPSP_highsens, PredPhospho GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho, Scansite_low Best element predictor (PredPhospho) NetPhosK_0.3, PPSP_balanced GPS, NetPhosK_0.3, PPSP_balanced GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho GPS, KinasePhos_bitscore, NetPhosK_0.3, PPSP_highsens, PredPhospho, Scansite_low Best element predictor (PPSP_balanced) 0.859 0.859 0.859 0.856 0.799 0.853 0.857 0.832 0.808 0.778 0.748 0.871 0.827 0.82 0.819 0.81 0.784 0.827 0.76 0.757 0.739 0.717 0.676 0.741 0.717 (0.36) 0.717 (0.36) 0.716 (0.38) 0.708 (0.50) 0.575 0.708 0.700 0.652 0.621 0.565 0.508 0.730 0.638 0.625 0.619 0.599 0.555 0.642 0.489 (0.37) 0.485 (0.41) 0.448 0.405 0.319 0.477 Predicting performance assessed on MetaPS06 datasets. For each l (2 l 6), the composition and predicting performance of the best meta- predictors composed of l element predictors are shown, together with the predicting performance of the best element predictor. Meta-predictors having predicting performance exceeding that of the best element predictor are shown in italic. P-values in Fisher’s Z-transformation test (compared with the MCC of the best element predictor) are shown in parentheses. Simple voting strategies: unweighted voting, unreduced weighted voting and reduced weighted voting We used the unweighted voting, unreduced weighted voting and reduced weighted voting strategies in constructing meta-predictors for the phosphorylation site problem. Unlike a multi-class prediction problem, a score threshold needs to be set for two-class phosphorylation site prediction problems. This score threshold is set as the half of the sum of all weights for the element predictors (see Materials and Methods). As shown in Table 5, the prediction performance of no unweighted voting meta-predictor exceeded that of the best element predictors. For the CDK and PKA kinase families, unreduced weighted voting meta-predictors (using the accuracy values of the element predictors as the weights) achieved slightly improved prediction over than that of the best element predictors. For the CK2 and PKC kinase families, none of the unreduced weighted meta-predictors produced satisfactory predicting perfor- mance. For the CDK, PKA and PKC kinase families, the best reduced weighted voting meta-predictors improved accuracy between 1.0% and 3.1%, and MCC between 1.8% and 4.4%, compared to those of the best element predictors. However, for the CK2 kinase family, no reduced weighted voting meta-predictor oﬀered a satisfac- tory predicting performance (Table 5). Weighted voting with restricted grid search parameter selection The weighted voting strategy with the weights set by the MCC or accuracy values of the element predictors did not render meta-predictors with satisfactory predicting performance for all four kinase families. We thus explored the more general form of the weighted voting strategy, with the weights of element predictors determined from the data. A grid search, with carefully devised restricted search space, allows the search to be executed in a manageable amount of time (Table 2 and Materials and Methods). The grid search of the 16 parameters was conducted on each of the four MetaPS06 datasets with 10-fold cross- validation. As is shown in Table 6, for all four kinase families, the weighted voting meta-predictors obtained by grid search parameter selection exhibits outstanding predicting performance which not only exceeds that of the best element predictors, but also surpasses that of any combinatorial or reduced voting meta-predictors constructed described above. The meta-predictors achieved an increase in accuracy of between 1.1% and 4.3%, and an increase in MCC of between 2.2% and 8.1% compared to the best element predictor for each kinase family. For the CK2, PKA and PKC kinase families, the meta-predictors demonstrated signiﬁcantly higher MCC