e22 Nucleic Acids Research, 2008, Vol. 36, No. 4 Table 1. Summary of the 15 element predictors Element predictor Refs. URLs GPS KinasePhos_90 KinasePhos_95 KinasePhos_100 KinasePhos_bitscore NetPhosK_0.3 NetPhosK_0.5 NetPhosK_0.7 PPSP_highsens PPSP_balanced PPSP_highspec PredPhospho Scansite_low Scansite_medium Scansite_high (19,25) http://973-proteinweb.ustc.edu.cn/gps/ gps_web/predict.php (13,24) http://kinasephos.mbc.nctu.edu.tw/ (17) (20) (16) (14) http://www.cbs.dtu.dk/services/ NetPhosK/ http://bioinformatics.lcd-ustc.org/ PPSP/ http://pred.ngri.re.kr/ PredPhospho.htm http://scansite.mit.edu/motifscan_ seq.phtml For data features and classiﬁcation methods, see text. personal communications, and these data were eliminated from the phosphorylation site compilation. We were unable to obtain the manually curated data used in the development of NetPhosK. However, considering that this program was released comparatively early (prior to December 2003), there is good chance that the small number of manually curated phosphorylation sites used in the development of this program were gathered in the Phospho.ELM eﬀort which took place in the following year, and were excluded from the compilation together with other earlier Phospho.ELM data. Scansite is diﬀerent from all other element prediction programs in that data obtained from oriented peptide library and phage display experiments were used in its development. These data are completely independent of the data compilation we made. Because no overlap exists between these two types of data, no data exclusion was necessary. At the end of this step, 3252 S/T phosphorylation sites remained in the compilation. Step 3. Assigning class labels to phosphorylation sites. Next, each phosphorylation site remaining in the compilation was assigned one of ﬁve class labels—four of which corresponding to the four kinase families: CDK, CK2, PKA and PKC, and ‘other’, denoting all other S/T kinase families. The class labels were assigned based on the annotations from the original data sources, according to a kinase name index table (available as Supplementary Table 1). The kinase name index table was created based primarily on KinBase (23), with a small number of manually added entries. Following the assignment of class labels, some redundant entries were emerged. After these redundant entries were removed, 2489 phos- phorylation sites remained in the compilation. They include 294 CDK phosphorylation sites, 229 CK2 phosphorylation sites, 360 PKA phosphorylation sites, PAGE 4 OF 11 348 PKC phosphorylation sites and 1258 phosphorylation sites labeled with the ‘other’ class label. Step 4. Making of four balanced datasets. The task of phosphorylation site prediction consists of four separate two-class classiﬁcation problems, one for each of the four major kinase families. For each of the four classiﬁcation problems, a dataset needs to be prepared that include both ‘positive data’ and ‘negative data’. A good phosphoryla- tion site predictor should be able to diﬀerentiate true phosphorylation sites for a given kinase family from phosphorylation sites of other families as well as from non-phosphorylation sites (occurrences of S/T in proteins that are not substrates of any S/T kinases). Thus, the ‘negative data’ should include two diﬀerent types of data: the true phosphorylation sites of other kinase families, and non-phosphorylation sites. A collection of ‘non-phosphorylation sites’ was compiled: from the same set of proteins where the known phosphorylation sites reside in, 1400 occurrences of S/T were picked for which no phosphorylation has been reported to have taken place. This collection of S/T sites likely includes some true phosphorylation sites that had not yet been identiﬁed. However, in the absence of a better method of making non-phosphorylation sites, this way of preparing the non-phosphorylation site data is an accepted practice in phosphorylation site prediction (15,24). For each of the four datasets, all phosphorylation sites in the data compilation (prepared in previous steps) with the corresponding class labels were included as the ‘positive set’. All phosphorylation sites bearing any other class labels (the three other families, and ‘other’), and the 1400-sample non-phosphorylation sites data were lumped together. All sites in the lumped set overlapping with any sites in the ‘positive set’ were removed, after which, the ‘negative set’ was constructed by picking samples at random from the lumped set. The size of the negative set was set to be 1.5 times that of the positive set, following the established practice in the ﬁeld (16). The ﬁnal, balanced datasets for the classiﬁcation of CDK, CK2, PKA and PKC phosphorylation sites consist of 294 positive and 441 negative samples, 229 positive and 343 negative samples, 360 positive and 540 negative samples and 348 positive and 552 negative samples, respectively. These datasets are available as Supplemen- tary Table 2. Selection of element predictors Because we focused on predicting phosphorylation sites for the four major S/T kinase families—CDK, CK2, PKA and PKC—in this study, non-speciﬁc phos- phorylation site prediction programs, including NetPhos and DISPHOS, were excluded. PHOSITE was excluded because its implementation is not available. PREDIKIN requires kinase sequences as input, thus it was also excluded. Six programs remained in the list of element predicting programs included in this study. A total of 12 element