BACKGROUND
deriving useful knowledge from genomic data depends critically on the ability to accurately assign biological roles to genes and their products.
the only conclusive way of characterizing protein function is by experimental determination, although interpretation may still lead to errors.
even with high-throughput screenings, functional assays remain time consuming, expensive and technically challenging.
additionally, experimental techniques target known aspects of function, which may provide a one-sided picture of otherwise multi-faceted concepts.
currently, the 'gold standard' for high-quality database annotation consists of manual curation by experts based on literature searching and bioinformatics analyses  <cit> .
however, this route is also time consuming and the number of sequences manually annotated clearly lags behind the exponential growth in the number of known sequences  <cit> .
consequently, most entries in uniprot  <cit>  include electronically inferred annotations, often of unknown reliability.
therefore, the development of reliable automatic function prediction tools is a major goal, since it is the only way of keeping database annotations updated as more and more sequence data pours in.
simple function predictors implement annotation transfer from well characterized proteins with high sequence similarity to the target.
however, they achieve rather limited performance  <cit> , principally due to the challenges posed by multi-domain proteins, and by paralogous and analogous genetic relationships.
more sophisticated methods aim at detecting local signatures indicative of function such as sequence motifs, domains, or domain architectures  <cit> .
previous work has also benefitted from considering phylogenetic analyses  <cit> , the hierarchical structure of go  <cit>  and the co-occurrence of its terms in high-quality database annotations  <cit> .
neural networks  <cit>  and support vector machines  <cit>  have been successfully trained to recognize sequence-derived patterns indicative of generic functional roles.
the requirement of sufficient training examples for robust modelling often restricts such methods to making broad functional assignments.
while such predictions do not help design specific experiments, they can be valuable for proteins with distant or no detectable sequence relatives.
other homology-free methods assume that functionally associated genes are co-expressed and their products interact with each other in some capacity.
usually these supervised or unsupervised approaches exploit high-throughput gene expression or protein-protein interaction data largely to assign proteins to known biological processes  <cit> .
additionally, large-scale screening data have also become available, and genomic data integration is a promising avenue with the potential to overcome the intrinsic limitations of techniques utilizing individual information sources.
integrative strategies can provide both increased coverage of the protein universe and more confident functional assignments when supporting evidence can be gathered from different studies  <cit> .
a wide array of machine learning and computational statistics tools have been applied to effectively combine heterogeneous data types, including those mentioned, as well as information about gene co-localization, transcription factor binding, phenotype annotations, disease associations and biomedical literature.
interested readers are referred to  <cit>  and citations therein.
as in other research areas of computational biology, the need to critically test and compare so many approaches has recently prompted the establishment of community-wide experiments.
the mousefunc experiment blindly tested independent integrative methods by providing common m. musculus datasets with limited homology information for both training and internal benchmarking purposes  <cit> .
the cafa challenge provided a more realistic setting to comprehensively undertake blind testing of generic function prediction methods.
the key task was to predict gene ontology   <cit>  terms for  <dig>  target proteins from a diverse range of species including seven eukaryotes and eleven prokaryotes.
prior to the experiment start, uniprotkb/swiss-prot curators had reviewed most of the corresponding entries, but approximately 30% had no go terms for the molecular function  or biological process  categories.
the remaining targets were principally annotated with electronically inferred annotations which were rarely specific .
here we first detail the integrative approach we used for this annotation challenge, which combines into a single framework a wide variety of tools and biological information sources encompassing sequence, gene expression and protein-protein interaction data.
we then benchmark the overall methodology and its components using a novel scoring scheme to measure the information overlap between predicted and reference go terms.
finally, we discuss the lessons we learnt from the first round of cafa.
methods
our approach to predicting function is based around combining a broad range of large scale function annotation methods and data sources.
unless otherwise noted, each component method was calibrated against a benchmark set, with the estimated precision used to determine the "confidence" of each method.
the performance of all methods was independently evaluated on  <dig> cafa targets that received experimental functional annotations between january and july  <dig>
benchmarking and estimation of prediction accuracy
a set of  <dig>  well characterized proteins from uniprotkb/swiss-prot was selected for internal benchmarking and calibration purposes.
we defined the set of reference functional annotations as the reported go terms for the molecular function  and biological process  categories along with all those more general descriptions linked by "is a"  relationships in the ontology.
each method described below was then run independently on this benchmark set and the predicted go terms were collated.
we then independently binned the predictions into equally sized groups based on their raw scores - e.g.
bit scores from psi-blast  <cit>  or raw svm scores from ffpred  <cit>  - and calculated the number of true and false positives - tp and fp respectively - for each group.
from these statistics we derived precision values  that we subsequently fitted to the raw scores, assuming a standard logistic function .
we used these regression models  to estimate the precision of each prediction given a particular raw score.
with the above benchmark in hand, a range of different component methods were evaluated.
we preselected only methods which had significantly better than random performance on the benchmark set, and the final component methods are as follows.
go term prediction using psi-blast
the most basic method employed in this work was a standard psi-blast search against the uniref <dig>  <cit>  database, using  <dig> iterations, and an e-value threshold of 1x10- <dig> for both hit selection and profile inclusion.
then, for each target sequence we report the mf and bp go terms associated with any matches with alignment coverage of at least 85% of its length.
in order to estimate the accuracy of the predictions, we selected the highest bit score of the alignments between the target and those sequences annotated with that term and converted this bit score into a value in  given the logistic regression model obtained during the benchmark.
go term prediction from uniprotkb/swiss-prot text-mining
for targets which already had descriptive text, keywords or comments in uniprotkb/swiss-prot, go terms were assigned using a naïve bayes text-mining approach  <cit> .
to start with, the relative frequencies of occurrence of single words and consecutive pairs of words were computed for all records where a go term appears ), along with the equivalent frequency in records where the go term is absent ) in the data bank.
to avoid zero counts, a pseudocount of  <dig> was used for all probability estimates.
from this raw data, the conditional probability of each go term based on the occurrence of each word group was estimated as follows:
p≈ffword|go+f
by combining these probabilities , naive bayes classification was carried out on the raw text from a particular uniprotkb/swiss-prot entry and likely go terms predicted.
to further extend the scope of the textual analysis, words occurring in the different uniprotkb/swiss-prot record types were recorded, and some simple pre-parsing of feature  records was also carried out.
specifically, in cases where the ft records specified residue numbers or residue number ranges, these residue numbers were placed into bins of width  <dig> and converted into word tokens .
this allowed a simple form of feature-based function prediction to be carried out.
for example, the following uniprotkb/swiss-prot ft record:
topo_dom  <dig>  <dig> extracellular ,
would be parsed into the following five tokens:
ft:topo_dom ft:numval <dig> ft:numval <dig> ft:extracellular ft:potential.
these tokens are treated in exactly the same way as words found in other record types.
go term prediction from amino acid trigram mining
the same naïve bayes classification approach that was used to relate go terms to descriptive statements in the uniprotkb/swiss-prot records was also applied to the amino acid sequences.
in this case, rather than words in description lines, trigrams of amino acids were counted across the whole sequence database.
from this raw data, the conditional probability of each go term based on the occurrence of each trigram was estimated in the form of log likelihood scores.
for all trigrams in the target sequence, log likelihood scores for particular go terms were calculated by summing the individual log likelihood scores for all referenced go terms in the training data.
go term prediction from sequence features
ffpred  <cit>  was also used to make go term predictions for eukaryotic targets.
ffpred starts by predicting a diverse range of sequence features, which include secondary structure elements, disordered regions, signal peptides, glycosylation sites, and several others.
these features are then analysed by a series of support vector machines  to assign go terms from a subset of  <dig>
go term prediction from orthologous groups
in addition to pairwise close homologues, more distant evolutionary relationships were obtained from the eggnog  collection of orthologous groups  <cit> .
for each target that could be assigned to an orthologous group, all go terms found in goa  <cit>  for all members of the same group were assigned with an estimated precision being calculated as the fraction of group members that shared the given go term.
go term prediction from profile-profile comparison
in addition to pairwise close homologues, very distant evolutionary relationships were obtained from the use of profile-profile comparisons.
psi-blast was used to compute position specific scoring matrices  for each entry in uniprotkb/swiss-prot along with every target sequence.
each target pssm was then compared against the set of uniprotkb/swiss-prot pssms according to the following scoring scheme based on the dot products of the two pssm vectors x  and y :
sx,y=∑i=120xitfyi∑i=120xitf+∑i=120yitfxi∑i=120yitf
where xtf and ytf are the respective target frequencies.
a simple affine gap penalty was used with an opening penalty of  <dig> and an extension penalty of  <dig>
to predict go terms, the profile-profile alignments were evaluated using a neural network with  <dig> input units,  <dig> hidden units and a single output unit.
the  <dig> inputs were the profile-profile alignment score, the percentage sequence identity, the percentage alignment coverage of sequence  <dig> and the percentage alignment coverage of sequence  <dig>  the single output represented the presence or absence of common go terms between the two proteins.
the neural network was then trained using the benchmark go data set to distinguish alignments sharing common go terms from those that do not.
training vectors were generated per go term rather than per alignment, such that a pair of profiles having  <dig> common go terms and say  <dig> mismatching go terms would result in  <dig> separate training vectors .
this ensured that the posterior probabilities extracted from the trained neural network would more accurately represent probabilities of go term occurrence.
after training, the neural network output was calibrated to produce precision estimates using the procedure described above.
rather than using a separate data set, the same benchmark set used to train the network was used to calibrate the output.
this was acceptable here because the eventual evaluation of the method would be carried out on "blinded" data not available at the time the neural network training took place.
go term prediction from high-throughput data sources
for human proteins  medium-to-low-confidence predictions of protein function were generated by an svm regression method called functionspace  <cit> .
human proteins are assigned coordinates in an 11-dimensional feature space and go terms assigned from annotated close neighbours in this space.
the  <dig> dimensions represent pairwise sequence similarity, predicted cellular localization, secondary structure similarity, transmembrane topology, disordered segment features, sequence-derived domain architecture, structure-based domain architecture, sequence domain fusion patterns, structural domain fusion patterns, protein-protein interactions and microarray data.
the microarray features were derived from bi-clustering of  <dig> publicly available experimental microarray datasets in order to identify which experiments and genes had maximal predictive value for specific go categories.
in total, over  <dig>  features were combined to train the  <dig> svm regression models and their outputs are integrated using a further second layer regression svm.
integration of methods and post-processing
to produce a final consensus set of predictions, separate prediction files were generated for each of the component methods described above, and these predictions were combined using a network propagation algorithm based on the go graph structure.
we start by defining dij as the difference in depth between two go terms i and j. for this to be defined, a direct path between the two terms must exist that does not include the root node.
for pairs of terms that have no direct connecting path, dij is left undefined, and in programming terms is given a null value to flag this situation.
where the sign of dij is positive, this indicates that term i is deeper  than term j and vice versa for negative values.
as a first step, all predicted terms from each component method are collated into a single set of terms.
this set is not extended further e.g.
by adding higher level or intermediate terms.
for cases where n methods predict the same term, the combined precision for the common term  is estimated as follows:
p'=1-∏i=1n
where α∈ <cit>  is a constant attenuation parameter  and pi the estimated precision for prediction i. the use of this attenuation parameter is to prevent the final estimated precision values from reaching a maximum of  <dig>  due to a single 'vote' from just one of the methods.
for each target, all lower level term scores were propagated up to higher level terms on the same path to root according to the following procedure:
foreach go term i
foreach go term j≠i
if dij ≠ null and dij <  <dig> then
pi'←1-
end if
the final list of terms was produced by first ranking the terms according to the final chained precision estimates calculated as described above.
to exclude mutually incompatible term pairs from the final list, each pair of predicted go terms was checked for co-occurrence in uniprotkb.
for any pairs of terms for which dij ≠ null and which were not seen to co-occur in uniprotkb, the term with lowest estimated precision was deleted from the list.
where a pair of such terms had equal precision values, both were kept.
cogic scores
in the light of the reference experimental data released by the assessors, we analysed our own predictions using a novel scoring scheme that takes into account each term specificity and confidence value.
we propose to calculate different graph-information content  similarity scores  <cit>  for overlapping subsets of go term predictions compared to their reference annotations, and to combine them such that the final measure lies in .
simgic=∑t∈a∩bic∑t∈a∪bic
where ic is the information content of t - an empirical measure of its specificity.
for each term t, we first counted its occurrence n in uniprotkb/swiss-prot functional assignments.
we chose to use only experimentally supported evidence codes; this reduces the size of the reference corpus, but also helps minimize assessment bias towards homology-based annotations  <cit> .
the initial counts were then propagated to the ancestral nodes and ic values computed as ict=-lognn where r is the root of the ontology.
for each target, four overlapping subsets of predicted go terms are calculated based on confidence scores ≥  <dig> ,  <dig> ,  <dig>  and  <dig>  and denoted as p <dig>  p <dig>  p <dig> and p <dig> respectively.
these sets as well as the set r of reference go terms are expanded by adding all ancestral nodes.
the simgic scores s <dig>  s <dig>  s <dig>  and s <dig> are then calculated by comparing r with p <dig>  p <dig>  p <dig> and p <dig> respectively and the final combined simgic  score for a given protein target is given by
cogic=s1+s2+s3+s <dig>
thus ensuring that correct predictions with higher confidence scores are rewarded more.
the cogic score equals  <dig> when r=p1=p2=p3=p <dig>  i.e.
all predicted go terms are validated and their confidence scores are greater than or equal to  <dig> .
its value is  <dig> if r∩p1=r∩p2=r∩p3=r∩p4={r}, where r is the root of the ontology .
baseline predictions of go terms
we compare the performance of our integrative approach with two independent naïve predictors that the assessors had run against the same test set.
the priors algorithm determines the most frequent  <dig>  go terms in uniprotkb/swiss-prot and assigns them to each target using their frequencies as confidence values.
the blast method transfers go terms from the hits of a blast  <cit>  search against a database of experimentally characterized proteins with default parameters; the confidence values is the scaled sequence identity between the target and the most similar hit bringing the annotation.
RESULTS
role of the component methods
the organizers initially released  <dig>  target proteins at the beginning of the experiment.
at the automated function prediction meeting in july  <dig>  method performance was benchmarked against  <dig> proteins that had been experimentally characterized in the intervening months.
this set comprises of  <dig> proteins with annotated molecular function terms , and  <dig> proteins with biological process terms .
we use these data provided by the organizers as a blind test set for the analyses below.
for each method, the columns report the percent of targets that were assigned go terms relative to the entire benchmark , the subset officially evaluated , the evaluated molecular function predictions , and the assessed biological process assignments .
we performed target-by-target numerical evaluation of the remaining predictions using the cogic similarity score and plotted the results in figure  <dig>  the performance of homology-based methods displays an expected pattern both within and across the mf and bp data sets.
orthologous groups generally scores higher than psi-blast and profile-profile alignment, which in contrast provide higher coverage.
in line with previous findings, the shift of the score distributions towards the top end is larger for molecular function predictions rather than the biological process ones  <cit> .
the amino acid trigram mining classifier appears very competitive with the other homology-based approaches on the mf benchmark.
yet, the picture does not take into account the data in table  <dig> and a more detailed comparison  would suggest that amino acid trigram mining only made predictions for "easy" targets in mf.
the scores for ffpred are unsurprisingly low: this machine-learning tool had been trained to predict a restricted set of general functional classes specifically for human proteins with distant or no detectable sequence relatives.
in order to investigate the usefulness of each component method, we initially tried to measure their exclusive scope with regard to the numbers of targets and go terms officially evaluated.
unfortunately, we couldn't achieve conclusive results due to the large overlap among the sets of predicted targets .
furthermore, no method output go term assignments that  were not explicitly predicted or implied by the annotations made by other components, and that  were reference annotations used for assessment or one of their ancestors.
consequently, we were unable to identify any single component method which was pivotal for the success of the overall strategy.
overall performance of the team jones-ucl
we calculated the distributions of cogic scores for the group jones-ucl and compared them with the other data in figure  <dig>  the final integrative and post-processing steps affect the go term lists and their confidence scores in interesting ways.
on the one hand, jones-ucl does not consistently improve over all components - e.g.
orthologues and text mining tend to achieve slightly higher prediction accuracy scores.
on the other hand, it records a cogic score of  <dig> for fewer targets than the component methods.
overall, no single component yielded significantly more accurate predictions than the aggregate method.
next, we compared the performance of our aggregate method with the two baseline predictors priors and blast.
the plots in figure  <dig> clearly show that jones-ucl scores are higher than the others on both ontologies.
we also tested the null hypothesis that the performance of jones-ucl is not statistically different from that of the naïve approaches using two-sided pair-wise wilcoxon rank sum tests and table  <dig> gives the resulting p-values.
at a p-value cut-off of 10- <dig>  our approach significantly outperforms both blast and priors in assigning molecular function and biological process terms.
the performance of pairs of methods was compared separately on the two benchmark subsets mf and bp.
we tested the null hypothesis that the distributions of the cogic scores over the set of common targets were indistinguishable with a paired two-sided wilcoxon sum rank test.
the table reports the number n of common targets and the p-value.
an inverse relationship exists between the specificity of the predicted go terms and their confidence scores, as a result of the back-propagation algorithm used.
however, it is not immediately clear whether our method tends to be more or less specific than the target annotations.
to investigate this, we analysed how the difference in specificity between predicted and reference go terms varies as a function of the predicted confidence scores.
to this end, we considered pairs  of predicted and experimental go terms where  x and y lie on the same path to root, and  there are no descendants of x predicted for the same target.
for such pairs, we calculated icx-ic and in figure  <dig> we plot the average of these values across all targets against the confidence scores assigned to x. the plot shows the expected inverse correlation between our confidence scores and the average semantic distance from experimental annotations.
at lower confidence values, the team jones-ucl assigned more specific go terms than those found in uniprotkb/swiss-prot annotations.
further studies and additional experimental data will be necessary to determine whether these predictions should be regarded as false positives or not.
conversely, the most confident predictions typically consist of more general terms than those experimentally validated.
this trend stems essentially from homology-based annotation transfers: specific functional annotations can only be made between very close homologues.
as evolutionary distance increases, more general terms can be assigned confidently, because distinguishing between paralogues and orthologues becomes harder and harder due to limited data available.
discussion
this study confirms that combining the strengths of different approaches provides increased coverage of the protein sequence space and more accurate function predictions.
to make our integrative strategy available to the community as a reliable and fast fully automated system, we need to address a few interrelated methodological questions.
these relate to the identification of the most valuable component methods and to their effective combination.
the preliminary analysis presented here did not help estimate properly the individual contributions made by each component.
in future studies, we propose to identify pivotal components by contrasting the performance of the current approach with its "leave-one-out" variants.
more refined statistical analyses may also be required to account for the relatedness between separate components .
the final go term predictions and their confidence scores were generated using heuristic methods.
it will be interesting to explore alternatives that do not weight all the components evenly.
this may be implemented by tailoring the attenuation parameters to the specific data sources and tools or - more flexibly - by training high-level classifiers such as svms or naïve bayes approaches.
of course, this choice may affect the way we will be able to select the subset of components that maximizes prediction scope and accuracy.
the proposed cogic score builds on previous efforts to numerically evaluate the semantic similarity between functional annotations.
it represents an initial attempt to provide a normalized measure of information overlap between the predicted and benchmark annotations for a protein target, which explicitly accounts for prediction confidence values.
the critical evaluation of protein function predictions is not a standard procedure yet, but it is drawing more and more attention.
we hope that this work will contribute to further discussions and new ideas in the future.
CONCLUSIONS
methods for gene function predictions are continuously being devised and improved to account for the ever increasing size of public databases and the broad range of data sources at hand.
evaluating their scope and usefulness has always been difficult due to a lack of suitable test sets, evaluation metrics and additional experimental settings.
the cafa experiment has provided a unique opportunity and a real high-throughput use-case scenario where biologists need as much detailed, accurate and extensive functional annotations as possible.
here we have detailed our integrative prediction approach entered at cafa and have provided a complementary assessment of its performance.
our results are encouraging but overall highlight considerable room for improvement in the field.
in line with previous findings the accuracy of molecular function predictions is higher than for biological process annotations.
this applies to simple homology-based predictors and to more complex methodologies like the one described here, though as yet neither can be said to perform adequately.
computational biologists evidently need to invest a great deal of effort to bring these methods up to an acceptable performance level.
as already witnessed in other areas of bioinformatics, community-wide blind experiments will be pivotal in establishing standards for the evaluation of prediction accuracy and in fostering advancements and new ideas.
competing interests
the authors declare that they have no competing interests.
authors' contributions
dtj conceived the study; all authors participated in its design and implementation; all authors drafted the manuscript, revised it critically and read and approved the final version.
supplementary material
additional file 1
boxplots of the cogic score distributions on the benchmarks that were targeted by the amino acid trigram mining classifier.
figure with legend.
click here for file
acknowledgements
we thank anna lobley for providing a set of predicted go terms and p-values for human protein sequences generated using her functionspace approach.
this work was partially supported by the uk biotechnology and biological sciences research council  and by a marie curie intra european fellowship within the 7th european community framework programme .
declarations
this article has been published as part of bmc bioinformatics volume  <dig> supplement  <dig>  2013: proceedings of automated function prediction sig  <dig> featuring the cafa challenge: critical assessment of function annotations.
the full contents of the supplement are available online at url.
http://www.biomedcentral.com/bmcbioinformatics/supplements/14/s <dig>

 deriving useful knowledge from genomic data depends critically on the ability to accurately assign biological roles to genes and their products. the only conclusive way of characterizing protein function is by experimental determination, although interpretation may still lead to errors. even with high-throughput screenings, functional assays remain time consuming, expensive and technically challenging. additionally, experimental techniques target known aspects of function, which may provide a one-sided picture of otherwise multi-faceted concepts. currently, the'gold standard' for high-quality database annotation consists of manual curation by experts based on literature searching and bioinformatics analyses. however, this route is also time consuming and the number of sequences manually annotated clearly lags behind the exponential growth in the number of known sequences. consequently, most entries in uniprot include electronically inferred annotations, often of unknown reliability. therefore, the development of reliable automatic function prediction tools is a major goal, since it is the only way of keeping database annotations updated as more and more sequence data pours in. simple function predictors implement annotation transfer from well characterized proteins with high sequence similarity to the target. however, they achieve rather limited performance, principally due to the challenges posed by multi-domain proteins, and by paralogous and analogous genetic relationships. more sophisticated methods aim at detecting local signatures indicative of function such as sequence motifs, domains, or domain architectures. previous work has also benefitted from considering phylogenetic analyses, the hierarchical structure of go and the co-occurrence of its terms in high-quality database annotations. neural networks and support vector machines have been successfully trained to recognize sequence-derived patterns indicative of generic functional roles. the requirement of sufficient training examples for robust modelling often restricts such methods to making broad functional assignments. while such predictions do not help design specific experiments, they can be valuable for proteins with distant or no detectable sequence relatives. other homology-free methods assume that functionally associated genes are co-expressed and their products interact with each other in some capacity. usually these supervised or unsupervised approaches exploit high-throughput gene expression or protein-protein interaction data largely to assign proteins to known biological processes. additionally, large-scale screening data have also become available, and genomic data integration is a promising avenue with the potential to overcome the intrinsic limitations of techniques utilizing individual information sources. integrative strategies can provide both increased coverage of the protein universe and more confident functional assignments when supporting evidence can be gathered from different studies. a wide array of machine learning and computational statistics tools have been applied to effectively combine heterogeneous data types, including those mentioned, as well as information about gene co-localization, transcription factor binding, phenotype annotations, disease associations and biomedical literature. interested readers are referred to and citations therein. as in other research areas of computational biology, the need to critically test and compare so many approaches has recently prompted the establishment of community-wide experiments. the mousefunc experiment blindly tested independent integrative methods by providing common m. musculus datasets with limited homology information for both training and internal benchmarking purposes. the cafa challenge provided a more realistic setting to comprehensively undertake blind testing of generic function prediction methods. the key task was to predict gene ontology terms for target proteins from a diverse range of species including seven eukaryotes and eleven prokaryotes. prior to the experiment start, uniprotkb/swiss-prot curators had reviewed most of the corresponding entries, but approximately 30% had no go terms for the molecular function or biological process categories. the remaining targets were principally annotated with electronically inferred annotations which were rarely specific. here we first detail the integrative approach we used for this annotation challenge, which combines into a single framework a wide variety of tools and biological information sources encompassing sequence, gene expression and protein-protein interaction data. we then benchmark the overall methodology and its components using a novel scoring scheme to measure the information overlap between predicted and reference go terms. finally, we discuss the lessons we learnt from the first round of cafa. methods our approach to predicting function is based around combining a broad range of large scale function annotation methods and data sources. unless otherwise noted, each component method was calibrated against a benchmark set, with the estimated precision used to determine the "confidence" of each method. the performance of all methods was independently evaluated on cafa targets that received experimental functional annotations between january and july benchmarking and estimation of prediction accuracy a set of well characterized proteins from uniprotkb/swiss-prot was selected for internal benchmarking and calibration purposes. we defined the set of reference functional annotations as the reported go terms for the molecular function and biological process categories along with all those more general descriptions linked by "is a" relationships in the ontology. each method described below was then run independently on this benchmark set and the predicted go terms were collated. we then independently binned the predictions into equally sized groups based on their raw scores - e.g. bit scores from psi-blast or raw svm scores from ffpred - and calculated the number of true and false positives - tp and fp respectively - for each group. from these statistics we derived precision values that we subsequently fitted to the raw scores, assuming a standard logistic function. we used these regression models to estimate the precision of each prediction given a particular raw score. with the above benchmark in hand, a range of different component methods were evaluated. we preselected only methods which had significantly better than random performance on the benchmark set, and the final component methods are as follows. go term prediction using psi-blast the most basic method employed in this work was a standard psi-blast search against the uniref database, using iterations, and an e-value threshold of 1x10- for both hit selection and profile inclusion. then, for each target sequence we report the mf and bp go terms associated with any matches with alignment coverage of at least 85% of its length. in order to estimate the accuracy of the predictions, we selected the highest bit score of the alignments between the target and those sequences annotated with that term and converted this bit score into a value in given the logistic regression model obtained during the benchmark. go term prediction from uniprotkb/swiss-prot text-mining for targets which already had descriptive text, keywords or comments in uniprotkb/swiss-prot, go terms were assigned using a naïve bayes text-mining approach. to start with, the relative frequencies of occurrence of single words and consecutive pairs of words were computed for all records where a go term appears ), along with the equivalent frequency in records where the go term is absent ) in the data bank. to avoid zero counts, a pseudocount of was used for all probability estimates. from this raw data, the conditional probability of each go term based on the occurrence of each word group was estimated as follows: p≈ffword|go+f by combining these probabilities, naive bayes classification was carried out on the raw text from a particular uniprotkb/swiss-prot entry and likely go terms predicted. to further extend the scope of the textual analysis, words occurring in the different uniprotkb/swiss-prot record types were recorded, and some simple pre-parsing of feature records was also carried out. specifically, in cases where the ft records specified residue numbers or residue number ranges, these residue numbers were placed into bins of width and converted into word tokens. this allowed a simple form of feature-based function prediction to be carried out. for example, the following uniprotkb/swiss-prot ft record: topo_dom extracellular, would be parsed into the following five tokens: ft:topo_dom ft:numval ft:numval ft:extracellular ft:potential. these tokens are treated in exactly the same way as words found in other record types. go term prediction from amino acid trigram mining the same naïve bayes classification approach that was used to relate go terms to descriptive statements in the uniprotkb/swiss-prot records was also applied to the amino acid sequences. in this case, rather than words in description lines, trigrams of amino acids were counted across the whole sequence database. from this raw data, the conditional probability of each go term based on the occurrence of each trigram was estimated in the form of log likelihood scores. for all trigrams in the target sequence, log likelihood scores for particular go terms were calculated by summing the individual log likelihood scores for all referenced go terms in the training data. go term prediction from sequence features ffpred was also used to make go term predictions for eukaryotic targets. ffpred starts by predicting a diverse range of sequence features, which include secondary structure elements, disordered regions, signal peptides, glycosylation sites, and several others. these features are then analysed by a series of support vector machines to assign go terms from a subset of go term prediction from orthologous groups in addition to pairwise close homologues, more distant evolutionary relationships were obtained from the eggnog collection of orthologous groups. for each target that could be assigned to an orthologous group, all go terms found in goa for all members of the same group were assigned with an estimated precision being calculated as the fraction of group members that shared the given go term. go term prediction from profile-profile comparison in addition to pairwise close homologues, very distant evolutionary relationships were obtained from the use of profile-profile comparisons. psi-blast was used to compute position specific scoring matrices for each entry in uniprotkb/swiss-prot along with every target sequence. each target pssm was then compared against the set of uniprotkb/swiss-prot pssms according to the following scoring scheme based on the dot products of the two pssm vectors x and y: sx,y=∑i=120xitfyi∑i=120xitf+∑i=120yitfxi∑i=120yitf where xtf and ytf are the respective target frequencies. a simple affine gap penalty was used with an opening penalty of and an extension penalty of to predict go terms, the profile-profile alignments were evaluated using a neural network with input units, hidden units and a single output unit. the inputs were the profile-profile alignment score, the percentage sequence identity, the percentage alignment coverage of sequence and the percentage alignment coverage of sequence the single output represented the presence or absence of common go terms between the two proteins. the neural network was then trained using the benchmark go data set to distinguish alignments sharing common go terms from those that do not. training vectors were generated per go term rather than per alignment, such that a pair of profiles having common go terms and say mismatching go terms would result in separate training vectors. this ensured that the posterior probabilities extracted from the trained neural network would more accurately represent probabilities of go term occurrence. after training, the neural network output was calibrated to produce precision estimates using the procedure described above. rather than using a separate data set, the same benchmark set used to train the network was used to calibrate the output. this was acceptable here because the eventual evaluation of the method would be carried out on "blinded" data not available at the time the neural network training took place. go term prediction from high-throughput data sources for human proteins medium-to-low-confidence predictions of protein function were generated by an svm regression method called functionspace. human proteins are assigned coordinates in an 11-dimensional feature space and go terms assigned from annotated close neighbours in this space. the dimensions represent pairwise sequence similarity, predicted cellular localization, secondary structure similarity, transmembrane topology, disordered segment features, sequence-derived domain architecture, structure-based domain architecture, sequence domain fusion patterns, structural domain fusion patterns, protein-protein interactions and microarray data. the microarray features were derived from bi-clustering of publicly available experimental microarray datasets in order to identify which experiments and genes had maximal predictive value for specific go categories. in total, over features were combined to train the svm regression models and their outputs are integrated using a further second layer regression svm. integration of methods and post-processing to produce a final consensus set of predictions, separate prediction files were generated for each of the component methods described above, and these predictions were combined using a network propagation algorithm based on the go graph structure. we start by defining dij as the difference in depth between two go terms i and j. for this to be defined, a direct path between the two terms must exist that does not include the root node. for pairs of terms that have no direct connecting path, dij is left undefined, and in programming terms is given a null value to flag this situation. where the sign of dij is positive, this indicates that term i is deeper than term j and vice versa for negative values. as a first step, all predicted terms from each component method are collated into a single set of terms. this set is not extended further e.g. by adding higher level or intermediate terms. for cases where n methods predict the same term, the combined precision for the common term is estimated as follows: p'=1-∏i=1n where α∈ is a constant attenuation parameter and pi the estimated precision for prediction i. the use of this attenuation parameter is to prevent the final estimated precision values from reaching a maximum of due to a single'vote' from just one of the methods. for each target, all lower level term scores were propagated up to higher level terms on the same path to root according to the following procedure: foreach go term i    foreach go term j≠i       if dij ≠ null and dij < then          pi'←1-    end if the final list of terms was produced by first ranking the terms according to the final chained precision estimates calculated as described above. to exclude mutually incompatible term pairs from the final list, each pair of predicted go terms was checked for co-occurrence in uniprotkb. for any pairs of terms for which dij ≠ null and which were not seen to co-occur in uniprotkb, the term with lowest estimated precision was deleted from the list. where a pair of such terms had equal precision values, both were kept. cogic scores in the light of the reference experimental data released by the assessors, we analysed our own predictions using a novel scoring scheme that takes into account each term specificity and confidence value. we propose to calculate different graph-information content similarity scores for overlapping subsets of go term predictions compared to their reference annotations, and to combine them such that the final measure lies in. simgic=∑t∈a∩bic∑t∈a∪bic where ic is the information content of t - an empirical measure of its specificity. for each term t, we first counted its occurrence n in uniprotkb/swiss-prot functional assignments. we chose to use only experimentally supported evidence codes; this reduces the size of the reference corpus, but also helps minimize assessment bias towards homology-based annotations. the initial counts were then propagated to the ancestral nodes and ic values computed as ict=-lognn where r is the root of the ontology. for each target, four overlapping subsets of predicted go terms are calculated based on confidence scores ≥,, and and denoted as p p p and p respectively. these sets as well as the set r of reference go terms are expanded by adding all ancestral nodes. the simgic scores s s s and s are then calculated by comparing r with p p p and p respectively and the final combined simgic score for a given protein target is given by cogic=s1+s2+s3+s thus ensuring that correct predictions with higher confidence scores are rewarded more. the cogic score equals when r=p1=p2=p3=p i.e. all predicted go terms are validated and their confidence scores are greater than or equal to. its value is if r∩p1=r∩p2=r∩p3=r∩p4={r}, where r is the root of the ontology. baseline predictions of go terms we compare the performance of our integrative approach with two independent naïve predictors that the assessors had run against the same test set. the priors algorithm determines the most frequent go terms in uniprotkb/swiss-prot and assigns them to each target using their frequencies as confidence values. the blast method transfers go terms from the hits of a blast search against a database of experimentally characterized proteins with default parameters; the confidence values is the scaled sequence identity between the target and the most similar hit bringing the annotation.  role of the component methods the organizers initially released target proteins at the beginning of the experiment. at the automated function prediction meeting in july method performance was benchmarked against proteins that had been experimentally characterized in the intervening months. this set comprises of proteins with annotated molecular function terms, and proteins with biological process terms. we use these data provided by the organizers as a blind test set for the analyses below. for each method, the columns report the percent of targets that were assigned go terms relative to the entire benchmark, the subset officially evaluated, the evaluated molecular function predictions, and the assessed biological process assignments. we performed target-by-target numerical evaluation of the remaining predictions using the cogic similarity score and plotted the results in figure the performance of homology-based methods displays an expected pattern both within and across the mf and bp data sets. orthologous groups generally scores higher than psi-blast and profile-profile alignment, which in contrast provide higher coverage. in line with previous findings, the shift of the score distributions towards the top end is larger for molecular function predictions rather than the biological process ones. the amino acid trigram mining classifier appears very competitive with the other homology-based approaches on the mf benchmark. yet, the picture does not take into account the data in table and a more detailed comparison would suggest that amino acid trigram mining only made predictions for "easy" targets in mf. the scores for ffpred are unsurprisingly low: this machine-learning tool had been trained to predict a restricted set of general functional classes specifically for human proteins with distant or no detectable sequence relatives. in order to investigate the usefulness of each component method, we initially tried to measure their exclusive scope with regard to the numbers of targets and go terms officially evaluated. unfortunately, we couldn't achieve conclusive results due to the large overlap among the sets of predicted targets. furthermore, no method output go term assignments that were not explicitly predicted or implied by the annotations made by other components, and that were reference annotations used for assessment or one of their ancestors. consequently, we were unable to identify any single component method which was pivotal for the success of the overall strategy. overall performance of the team jones-ucl we calculated the distributions of cogic scores for the group jones-ucl and compared them with the other data in figure the final integrative and post-processing steps affect the go term lists and their confidence scores in interesting ways. on the one hand, jones-ucl does not consistently improve over all components - e.g. orthologues and text mining tend to achieve slightly higher prediction accuracy scores. on the other hand, it records a cogic score of for fewer targets than the component methods. overall, no single component yielded significantly more accurate predictions than the aggregate method. next, we compared the performance of our aggregate method with the two baseline predictors priors and blast. the plots in figure clearly show that jones-ucl scores are higher than the others on both ontologies. we also tested the null hypothesis that the performance of jones-ucl is not statistically different from that of the naïve approaches using two-sided pair-wise wilcoxon rank sum tests and table gives the resulting p-values. at a p-value cut-off of 10- our approach significantly outperforms both blast and priors in assigning molecular function and biological process terms. the performance of pairs of methods was compared separately on the two benchmark subsets mf and bp. we tested the null hypothesis that the distributions of the cogic scores over the set of common targets were indistinguishable with a paired two-sided wilcoxon sum rank test. the table reports the number n of common targets and the p-value. an inverse relationship exists between the specificity of the predicted go terms and their confidence scores, as a result of the back-propagation algorithm used. however, it is not immediately clear whether our method tends to be more or less specific than the target annotations. to investigate this, we analysed how the difference in specificity between predicted and reference go terms varies as a function of the predicted confidence scores. to this end, we considered pairs of predicted and experimental go terms where x and y lie on the same path to root, and there are no descendants of x predicted for the same target. for such pairs, we calculated icx-ic and in figure we plot the average of these values across all targets against the confidence scores assigned to x. the plot shows the expected inverse correlation between our confidence scores and the average semantic distance from experimental annotations. at lower confidence values, the team jones-ucl assigned more specific go terms than those found in uniprotkb/swiss-prot annotations. further studies and additional experimental data will be necessary to determine whether these predictions should be regarded as false positives or not. conversely, the most confident predictions typically consist of more general terms than those experimentally validated. this trend stems essentially from homology-based annotation transfers: specific functional annotations can only be made between very close homologues. as evolutionary distance increases, more general terms can be assigned confidently, because distinguishing between paralogues and orthologues becomes harder and harder due to limited data available. discussion this study confirms that combining the strengths of different approaches provides increased coverage of the protein sequence space and more accurate function predictions. to make our integrative strategy available to the community as a reliable and fast fully automated system, we need to address a few interrelated methodological questions. these relate to the identification of the most valuable component methods and to their effective combination. the preliminary analysis presented here did not help estimate properly the individual contributions made by each component. in future studies, we propose to identify pivotal components by contrasting the performance of the current approach with its "leave-one-out" variants. more refined statistical analyses may also be required to account for the relatedness between separate components. the final go term predictions and their confidence scores were generated using heuristic methods. it will be interesting to explore alternatives that do not weight all the components evenly. this may be implemented by tailoring the attenuation parameters to the specific data sources and tools or - more flexibly - by training high-level classifiers such as svms or naïve bayes approaches. of course, this choice may affect the way we will be able to select the subset of components that maximizes prediction scope and accuracy. the proposed cogic score builds on previous efforts to numerically evaluate the semantic similarity between functional annotations. it represents an initial attempt to provide a normalized measure of information overlap between the predicted and benchmark annotations for a protein target, which explicitly accounts for prediction confidence values. the critical evaluation of protein function predictions is not a standard procedure yet, but it is drawing more and more attention. we hope that this work will contribute to further discussions and new ideas in the future.  methods for gene function predictions are continuously being devised and improved to account for the ever increasing size of public databases and the broad range of data sources at hand. evaluating their scope and usefulness has always been difficult due to a lack of suitable test sets, evaluation metrics and additional experimental settings. the cafa experiment has provided a unique opportunity and a real high-throughput use-case scenario where biologists need as much detailed, accurate and extensive functional annotations as possible. here we have detailed our integrative prediction approach entered at cafa and have provided a complementary assessment of its performance. our results are encouraging but overall highlight considerable room for improvement in the field. in line with previous findings the accuracy of molecular function predictions is higher than for biological process annotations. this applies to simple homology-based predictors and to more complex methodologies like the one described here, though as yet neither can be said to perform adequately. computational biologists evidently need to invest a great deal of effort to bring these methods up to an acceptable performance level. as already witnessed in other areas of bioinformatics, community-wide blind experiments will be pivotal in establishing standards for the evaluation of prediction accuracy and in fostering advancements and new ideas. competing interests the authors declare that they have no competing interests. authors' contributions dtj conceived the study; all authors participated in its design and implementation; all authors drafted the manuscript, revised it critically and read and approved the final version. supplementary material additional file 1 boxplots of the cogic score distributions on the benchmarks that were targeted by the amino acid trigram mining classifier. figure with legend. click here for file acknowledgements we thank anna lobley for providing a set of predicted go terms and p-values for human protein sequences generated using her functionspace approach. this work was partially supported by the uk biotechnology and biological sciences research council and by a marie curie intra european fellowship within the 7th european community framework programme. declarations this article has been published as part of bmc bioinformatics volume supplement 2013: proceedings of automated function prediction sig featuring the cafa challenge: critical assessment of function annotations. the full contents of the supplement are available online at url. http://www.biomedcentral.com/bmcbioinformatics/supplements/14/s 
accurate protein function annotation is a severe bottleneck when utilizing the deluge of high-throughput, next generation sequencing data. keeping database annotations up-to-date has become a major scientific challenge that requires the development of reliable automatic predictors of protein function. we generated the final molecular function and biological process assignments by combining the initial predictions in a probabilistic manner, which takes into account the gene ontology hierarchical structure. in line with previous studies, we find that molecular function predictions are more accurate than biological process assignments. our method integrates into a single framework a wide variety of biological information sources, encompassing sequence, gene expression and protein-protein interaction data, as well as annotations in uniprot entries. the methodology transfers functional categories based on the results from complementary homology-based and feature-based analyses. overall, the results indicate that there is considerable room for improvement in the field. the cafa experiment provided a unique opportunity to undertake comprehensive'blind testing' of many diverse approaches for automated function prediction. we demonstrate that our integrative approach provides increased scope and accuracy over both the component methods and the naïve predictors. as already witnessed in other areas, community-wide blind testing experiments will be pivotal in establishing standards for the evaluation of prediction accuracy, in fostering advancements and new ideas, and ultimately in recording progress. 15- july automated function prediction sig featuring the cafa challenge: critical assessment of function annotations vienna, austria it still remains for the community to invest a great deal of effort to make automated function prediction a useful and routine component in the toolbox of life scientists.
 accurate protein function annotation is a severe bottleneck when utilizing the deluge of high-throughput, next generation sequencing data. keeping database annotations up-to-date has become a major scientific challenge that requires the development of reliable automatic predictors of protein function. the cafa experiment provided a unique opportunity to undertake comprehensive'blind testing' of many diverse approaches for automated function prediction. we report on the methodology we used for this challenge and on the lessons we learnt. methods our method integrates into a single framework a wide variety of biological information sources, encompassing sequence, gene expression and protein-protein interaction data, as well as annotations in uniprot entries. the methodology transfers functional categories based on the results from complementary homology-based and feature-based analyses. we generated the final molecular function and biological process assignments by combining the initial predictions in a probabilistic manner, which takes into account the gene ontology hierarchical structure.  we propose a novel scoring function called combined graph-information content similarity score for the comparison of predicted functional categories and benchmark data. we demonstrate that our integrative approach provides increased scope and accuracy over both the component methods and the naïve predictors. in line with previous studies, we find that molecular function predictions are more accurate than biological process assignments.  overall, the results indicate that there is considerable room for improvement in the field. it still remains for the community to invest a great deal of effort to make automated function prediction a useful and routine component in the toolbox of life scientists. as already witnessed in other areas, community-wide blind testing experiments will be pivotal in establishing standards for the evaluation of prediction accuracy, in fostering advancements and new ideas, and ultimately in recording progress. 15- july automated function prediction sig featuring the cafa challenge: critical assessment of function annotations vienna, austria 