PAGE 5 OF 11 predictors are derived from these programs (Table 1). Each prediction program is discussed below: GPS (19,25). GPS uses the 7-mer peptide sequence surrounding the center S/T position (three amino acids on each side of S/T) as its features. It calculates a similarity score between each pair of 7-mer sequences in the training dataset based on the BLOSUM62 substitution matrix, and performs a clustering of the training sequences. The prediction is made based on the average similarity score calculated between the test sequence and all sequences belonging to a given kinase family. KinasePhos (13,24). KinasePhos uses the 9-mer peptide sequence surrounding the center S/T (four amino acids on each side of S/T) as its features, and constructs proﬁle HMM for making predictions about phosphorylation sites. Several options are provided in the prediction program, including three given levels of speciﬁcity: 90%, 95% and 100%, and an option ‘by default HMM bit score’. These four options are regarded as four separate element predictors for the meta-prediction problem, and they are termed KinasePhos_90, KinasePhos_95, KinasePhos_100 and KinasePhos_bitscore, respectively. NetPhosK (17). NetPhosK uses the 15-mer or 17-mer peptide sequence surrounding the center S/T position as features, and it trains ANN models for making prediction about phosphorylation sites. Two options are provided in the prediction server: ‘prediction without ﬁltering’ and ‘prediction with ESS ﬁltering’. We chose the ‘prediction without ﬁltering’ option because the ESS ﬁltering would take prohibitively long time to compute for the scale of prediction task we had. Furthermore, the NetPhosK server allows the setting of a ‘threshold’ value. Although a wide range of threshold values are provided for selection (between 0 and 0.95), we found that threshold values below 0.3 lead to very low speciﬁcity (close to 0), and threshold values above 0.7 give rise to very low sensitivity in the prediction results. Thus we chose to use these three threshold levels: 0.3, 0.5 and 0.7 for making predic- tions, and they are regarded as three separate element predictors, namely, NetPhosK_0.3, NetPhosK_0.5 and NetPhosK_0.7, respectively, in this study. PPSP (20). PPSP deﬁnes features using the 9-mer peptide sequence surrounding the S/T position, and a Bayesian-based model was constructed for making predictions about phosphorylation sites. The PPSP pre- diction server provides three options: high sensitivity, balanced and high speciﬁcity. These three options are regarded as three separate element predictors for our purposes, and they are referred to as PPSP_highsens, PPSP_balanced and PPSP_highspec, respectively. PredPhospho (16). PredPhospho uses peptide sequences of variable lengths (7-mer through 51-mers) centered on the S/T position, as its features; it trains SVM for making its predictions about phosphorylation sites. Nucleic Acids Research, 2008, Vol. 36, No. 4 e22 Scansite (14). Scansite organizes peptide sequence features obtained from an oriented peptide library and phage display experiments into position-speciﬁc scoring matrices (PSSM). For a potential phosphorylation site, the PSSM score is calculated and the prediction is made based on whether the score exceeds a pre-set threshold value. Three options are provided in the Scansite prediction program, each representing a diﬀerent stringency levels: high stringency, medium stringency and low stringency. They are regarded as three separate element predictors for the purposes of the meta-prediction problem, namely, Scansite_high, Scansite_medium and Scansite_low. Obtaining and pre-processing prediction results of element predictors All six online prediction servers take protein sequences as their input. Prediction jobs were submitted to each of the prediction servers using locally developed Perl scripts with the speciﬁed prediction options. The prediction result pages were parsed and processed with another set of Perl scripts. For every potential phosphorylation site (or every occurrence of S/T in the protein sequence), certain numerical scores were produced by the element prediction programs. These scores are of diﬀerent mathematical meanings for diﬀerent prediction programs. For example, the score produced by KinasePhos is the HMM bit score, the score produced by PPSP represents the risk diﬀerence in the Bayesian decision model and the score produced by Scansite is the calculated PSSM score. In addition to the numerical score, each element prediction program also makes a binary determination about whether a site is a phosphorylation site for a given family of kinases. The numerical scores produced by the prediction programs contain richer information than the binary determination, which might lead to improved predicting performance in meta-predictors. However, it is diﬃcult to compare these scores across prediction programs due to their diﬀerent meanings. In this study, we ignored the numerical scores, and only used the binary determinations. Performance measures In the protein subcellular localization prediction problem (7), only a small fraction of samples (about 3%) are categorized into multiple subcellular compartments. For the phosphorylation site prediction problem, however, a substantial proportion of phosphorylation sites (>20%) are known to be phosphorylated by multiple kinases. Therefore it is more proper to treat the phosphorylation site prediction for the four major S/T kinase families as four separate two-class classiﬁcation problems, than consider- ing it as a single four-class classiﬁcation problem, as was done for the protein subcellular localization prediction problem. For two-class classiﬁcation problems, commonly used performance measures for predictors include sensitiv- ity, speciﬁcity, accuracy and Matthew’s correlation coeﬃ- cient (MCC) (7). Sensitivity and speciﬁcity indicate the predictor’s abilities to curb false-negative and false-positive predictions, respectively, and these two measures need to be examined together to obtain an overall evaluation