BACKGROUND
genome sequencing projects continue to produce unprecedented amounts of novel protein sequence information, and large-scale experimental efforts are underway to determine the function of the newly discovered proteins  <cit> .
for a majority of proteins it is already possible to predict their approximate function with reasonable accuracy based on their evolutionary relationship or sequence similarity to proteins with known functions  <cit> .
for most recently sequenced bacterial genomes about three quarters of open reading frames can be assigned a possible function in this way.
however, a significant number of predicted proteins in each newly sequenced genome have turned out to defy this approach.
these proteins, which in extreme cases may constitute up to 50% of open reading frames, show no similarity to proteins of known function.
this may be due to missing experimental data, or the proteins are evolving too rapidly or are even unique to a small clade of species.
it would be very useful if one could obtain at least a general idea of the function of such proteins based on their amino acid sequence alone.
of course this is an extremely challenging task, and one that will only be of limited usefulness without combining it with additional information , but nonetheless several techniques to address this issue have been proposed recently  <cit> .
these publications show that using machine learning classifiers it is possible to predict the function of well-characterized proteins based on features of their amino acid sequence, without using homology information  <cit> .
however, it is unclear if and how well such classifiers would transfer to proteins of unknown function.
there are many reasons to assume that these 'unknown' proteins are special and differ from well-characterized proteins in significant ways: they may be evolving at a faster pace, they may function in unconventional ways, they may have unusual physico-chemical properties that have made them less accessible to experimentation.
if 'unknown' proteins are not just a random subset of the proteome, but are biased in such a systematic fashion, classifiers trained and tested on proteins of known function may generalize poorly and will be unable to predict the function of the real proteins of interest.
a direct test of the predictive performance on proteins of unknown function is rarely possible, although a recent retrospective study  <cit>  made some first steps in that direction.
thus a critical systematic assessment of the general prospect of successful classifier transfer is of great interest.
here we show that proteins of known and unknown function do indeed differ significantly.
we go on to show that, surprisingly, proteins from different species do also differ, to an even larger extent.
we then demonstrate that classifiers do nonetheless generalize across species boundaries and use this to provide the first critical estimate of predictive performance on proteins of unknown function.
RESULTS
we based our analysis on the completed and annotated proteomes of seven bacterial pathogens which cause sexually transmitted diseases in humans .
these species cover a wide range of phylogenetic relationships, from closely related species  to very divergent ones .
on the other hand, they all share the same general ecological niche, thus minimizing confounding effects of divergent evolutionary adaptation.
prediction of known protein functions
in this paper we are not interested in optimizing a method of predicting protein functions, but rather in evaluating an aspect of function prediction that has been somewhat neglected previously, namely whether classifiers trained on proteins of known functions can be expected to transfer successfully to proteins of unknown function.
even an optimal classifier would be useless if it could not be applied reliably to the real proteins of interest, i.e.
those for which no function is known at present.
however, as a baseline for our study, we first showed that we are able to correctly predict the function of proteins with known function using a support vector machine classifier based on features derived from their amino acid sequence alone .
confirming previous results  <cit>  we found that this is indeed possible, although with varying performance for each class and species .
only in three highly specialized bacterial species  overall performance was hardly better than random, and we will show below how the results of the present work indicate a way to overcome this problem.
the observed median auc is 56% averaged across all species and functional classes, and is higher for some important functional classes such as intermediary metabolism, dna metabolism, and transport and binding proteins.
these results are equivalent to previously reported accuracies  <cit> .
the generally good performance on such small bacterial genomes is encouraging, especially as it does not rely on the use of posttranslational modification and localization predictions, which are very informative features for eukaryotic proteins  <cit> .
discrimination between 'known' and 'unknown' proteins
previous studies in general stopped at this point and assumed that predictive performance would be maintained when the classifier were applied to proteins of unknown function.
we wanted to know if that is a reasonable assumption.
to determine the overall similarity of known and unknown proteins in the feature space used for function prediction, we trained another set of svm classifiers to try to distinguish between these two sets of proteins.
not unexpectedly we found that they do indeed differ significantly .
the possible reason why this is the case lies in the type of unknown proteins.
a set of unknown proteins in a species typically contains:
• unique proteins – proteins without known homologs
• hypothetical proteins – proteins with homologs of unknown function.
no experimental evidence exists for the function or existence of the protein product.
• wrongly predicted proteins – open reading frames that are not actually expressed , but are only the result of genome misannotation.
• special proteins – proteins that have a special feature  making them different from known proteins, but which do have a biological function.
from the above list, one can see that the set of unknown proteins will contain some members that actually will have a function and others that are probably genome annotation artifacts.
the ability to distinguish between known and unknown proteins is most likely due to the difference between unusual unknown proteins  and normal known proteins.
it is expected that homology-less function prediction will be possible for the 'normal' and unknown proteins , while being much more difficult for the special proteins  and meaningless for wrongly predicted proteins .
therefore, it would be interesting to use the clasification information to estimate the fraction of 'predictable' and 'unpredictable' proteins in the set of unknown proteins.
an exact estimate is not possible, because there is no exact definition of a normal protein available, but we can use the performance of the svm classifier to obtain a rough estimate.
the median auc for the discrimination between proteins of known and unknown function is 63%.
if we assume that 'predictable' unknown proteins are indistinguishable from known proteins, we can calculate the lower bound estimate of the fraction of unpredictable proteins to be 26% /50%).
discrimination between proteins from different bacteria
to determine if the effect of protein set dissimilarity will be as deleterious as one might fear, we argued that proteins from different species will also show some level of dissimilarity and, hence, one could use the performance of classifers across species boundaries to estimate the transferability from known to unknown proteins.
using our taxonomically diverse sample of bacterial species we trained a new set of svm classifiers to try to distinguish between each pair of species.
to our great surprise we found that this task is far easier than function prediction or the discrimination of known and unknown proteins .
the median auc across all species pairs is an astonishing 85%.
this means that given any randomly picked pair of proteins from species a and b, we will be able to assign them to their correct species of origin in 85% of cases.
this finding is entirely unexpected, given that the bacteria in our dataset all share the same highly stable ecological niche, the human urogenital tract.
while they naturally differ widely in their exact pathophysiology, they would still be expected to carry out the same general biological processes using very similar molecular machinery.
the fact that the svm classifiers are nonetheless able to find generally valid species-specific "sequence signatures" is of course of great biological interest.
one possible explanation for this high accuracy in discriminating proteins from different species lies in the varying levels of guanine-cytosin  content in each species.
variations in genomic gc content from 25% to 75% have been shown to be common in prokaryotes  <cit> .
variations in gc content in coding sequences will be reflected in differences of amino acid composition, as gc rich codons will be depleted in low-gc species and vice versa.
even if this variation is subtle, it will influence classifier performance.
classifier transfer between species
how then does this high level of dissimilarity between species affect the performance of function prediction classifiers?
figure  <dig> shows that classifiers transfer across species boundaries with surprisingly little loss of accuracy.
classifiers that perform well on their species of origin do almost as well on each of the other species.
high levels of protein set dissimilarity are apparently tolerated without decreasing performance.
a case of special interest is represented by the mycoplasma genomes, where classifiers perform poorly if they are trained on the species itself, and functions are predicted with higher accuracy if the classifier comes from one of the non-mycoplasma species.
mycoplasma species are highly derived organisms with extremely reduced minimal genomes and their proteomes may be specifically adapted, e.g.
the features used for their svm classifiers differ entirely from those of the other species , but the paradoxical cross-species performance is still difficult to explain by this fact alone.
CONCLUSIONS
in conclusion, we find that proteins with known and unknown function differ significantly, but we also find that classifiers transfer very well between different bacterial species which differ even more.
viewed optimistically, this means that there is a distinct possibility that function prediction classifiers will generalize successfully to predict the function of proteins of unknown function.
figure  <dig> summarizes the results and can also be used to estimate the performance of classifiers for unknown proteins.
in most cases, this performance will be almost as good as that on the known proteins.
our findings also indicate that, especially in the case of "unusual" proteomes, such as the mycoplasmal examples, it may be a promising strategy to train classifiers on related but more conventional species to achieve the highest predictive performance.

 genome sequencing projects continue to produce unprecedented amounts of novel protein sequence information, and large-scale experimental efforts are underway to determine the function of the newly discovered proteins. for a majority of proteins it is already possible to predict their approximate function with reasonable accuracy based on their evolutionary relationship or sequence similarity to proteins with known functions. for most recently sequenced bacterial genomes about three quarters of open reading frames can be assigned a possible function in this way. however, a significant number of predicted proteins in each newly sequenced genome have turned out to defy this approach. these proteins, which in extreme cases may constitute up to 50% of open reading frames, show no similarity to proteins of known function. this may be due to missing experimental data, or the proteins are evolving too rapidly or are even unique to a small clade of species. it would be very useful if one could obtain at least a general idea of the function of such proteins based on their amino acid sequence alone. of course this is an extremely challenging task, and one that will only be of limited usefulness without combining it with additional information, but nonetheless several techniques to address this issue have been proposed recently. these publications show that using machine learning classifiers it is possible to predict the function of well-characterized proteins based on features of their amino acid sequence, without using homology information. however, it is unclear if and how well such classifiers would transfer to proteins of unknown function. there are many reasons to assume that these'unknown' proteins are special and differ from well-characterized proteins in significant ways: they may be evolving at a faster pace, they may function in unconventional ways, they may have unusual physico-chemical properties that have made them less accessible to experimentation. if'unknown' proteins are not just a random subset of the proteome, but are biased in such a systematic fashion, classifiers trained and tested on proteins of known function may generalize poorly and will be unable to predict the function of the real proteins of interest. a direct test of the predictive performance on proteins of unknown function is rarely possible, although a recent retrospective study made some first steps in that direction. thus a critical systematic assessment of the general prospect of successful classifier transfer is of great interest. here we show that proteins of known and unknown function do indeed differ significantly. we go on to show that, surprisingly, proteins from different species do also differ, to an even larger extent. we then demonstrate that classifiers do nonetheless generalize across species boundaries and use this to provide the first critical estimate of predictive performance on proteins of unknown function.  we based our analysis on the completed and annotated proteomes of seven bacterial pathogens which cause sexually transmitted diseases in humans. these species cover a wide range of phylogenetic relationships, from closely related species to very divergent ones. on the other hand, they all share the same general ecological niche, thus minimizing confounding effects of divergent evolutionary adaptation. prediction of known protein functions in this paper we are not interested in optimizing a method of predicting protein functions, but rather in evaluating an aspect of function prediction that has been somewhat neglected previously, namely whether classifiers trained on proteins of known functions can be expected to transfer successfully to proteins of unknown function. even an optimal classifier would be useless if it could not be applied reliably to the real proteins of interest, i.e. those for which no function is known at present. however, as a baseline for our study, we first showed that we are able to correctly predict the function of proteins with known function using a support vector machine classifier based on features derived from their amino acid sequence alone. confirming previous results we found that this is indeed possible, although with varying performance for each class and species. only in three highly specialized bacterial species overall performance was hardly better than random, and we will show below how the results of the present work indicate a way to overcome this problem. the observed median auc is 56% averaged across all species and functional classes, and is higher for some important functional classes such as intermediary metabolism, dna metabolism, and transport and binding proteins. these results are equivalent to previously reported accuracies. the generally good performance on such small bacterial genomes is encouraging, especially as it does not rely on the use of posttranslational modification and localization predictions, which are very informative features for eukaryotic proteins. discrimination between'known' and'unknown' proteins previous studies in general stopped at this point and assumed that predictive performance would be maintained when the classifier were applied to proteins of unknown function. we wanted to know if that is a reasonable assumption. to determine the overall similarity of known and unknown proteins in the feature space used for function prediction, we trained another set of svm classifiers to try to distinguish between these two sets of proteins. not unexpectedly we found that they do indeed differ significantly. the possible reason why this is the case lies in the type of unknown proteins. a set of unknown proteins in a species typically contains: • unique proteins – proteins without known homologs • hypothetical proteins – proteins with homologs of unknown function. no experimental evidence exists for the function or existence of the protein product. • wrongly predicted proteins – open reading frames that are not actually expressed, but are only the result of genome misannotation. • special proteins – proteins that have a special feature making them different from known proteins, but which do have a biological function. from the above list, one can see that the set of unknown proteins will contain some members that actually will have a function and others that are probably genome annotation artifacts. the ability to distinguish between known and unknown proteins is most likely due to the difference between unusual unknown proteins and normal known proteins. it is expected that homology-less function prediction will be possible for the'normal' and unknown proteins, while being much more difficult for the special proteins and meaningless for wrongly predicted proteins. therefore, it would be interesting to use the clasification information to estimate the fraction of'predictable' and'unpredictable' proteins in the set of unknown proteins. an exact estimate is not possible, because there is no exact definition of a normal protein available, but we can use the performance of the svm classifier to obtain a rough estimate. the median auc for the discrimination between proteins of known and unknown function is 63%. if we assume that'predictable' unknown proteins are indistinguishable from known proteins, we can calculate the lower bound estimate of the fraction of unpredictable proteins to be 26% /50%). discrimination between proteins from different bacteria to determine if the effect of protein set dissimilarity will be as deleterious as one might fear, we argued that proteins from different species will also show some level of dissimilarity and, hence, one could use the performance of classifers across species boundaries to estimate the transferability from known to unknown proteins. using our taxonomically diverse sample of bacterial species we trained a new set of svm classifiers to try to distinguish between each pair of species. to our great surprise we found that this task is far easier than function prediction or the discrimination of known and unknown proteins. the median auc across all species pairs is an astonishing 85%. this means that given any randomly picked pair of proteins from species a and b, we will be able to assign them to their correct species of origin in 85% of cases. this finding is entirely unexpected, given that the bacteria in our dataset all share the same highly stable ecological niche, the human urogenital tract. while they naturally differ widely in their exact pathophysiology, they would still be expected to carry out the same general biological processes using very similar molecular machinery. the fact that the svm classifiers are nonetheless able to find generally valid species-specific "sequence signatures" is of course of great biological interest. one possible explanation for this high accuracy in discriminating proteins from different species lies in the varying levels of guanine-cytosin content in each species. variations in genomic gc content from 25% to 75% have been shown to be common in prokaryotes. variations in gc content in coding sequences will be reflected in differences of amino acid composition, as gc rich codons will be depleted in low-gc species and vice versa. even if this variation is subtle, it will influence classifier performance. classifier transfer between species how then does this high level of dissimilarity between species affect the performance of function prediction classifiers? figure shows that classifiers transfer across species boundaries with surprisingly little loss of accuracy. classifiers that perform well on their species of origin do almost as well on each of the other species. high levels of protein set dissimilarity are apparently tolerated without decreasing performance. a case of special interest is represented by the mycoplasma genomes, where classifiers perform poorly if they are trained on the species itself, and functions are predicted with higher accuracy if the classifier comes from one of the non-mycoplasma species. mycoplasma species are highly derived organisms with extremely reduced minimal genomes and their proteomes may be specifically adapted, e.g. the features used for their svm classifiers differ entirely from those of the other species, but the paradoxical cross-species performance is still difficult to explain by this fact alone.  in conclusion, we find that proteins with known and unknown function differ significantly, but we also find that classifiers transfer very well between different bacterial species which differ even more. viewed optimistically, this means that there is a distinct possibility that function prediction classifiers will generalize successfully to predict the function of proteins of unknown function. figure summarizes the results and can also be used to estimate the performance of classifiers for unknown proteins. in most cases, this performance will be almost as good as that on the known proteins. our findings also indicate that, especially in the case of "unusual" proteomes, such as the mycoplasmal examples, it may be a promising strategy to train classifiers on related but more conventional species to achieve the highest predictive performance. 
until now, however, it has been unclear if this ability would be transferable to proteins of unknown function, which may show distinct biases compared to experimentally more tractable proteins. here we show that proteins with known and unknown function do indeed differ significantly. we then show that proteins from different bacterial species also differ to an even larger and very surprising extent, but that functional classifiers nonetheless generalize successfully across species boundaries. machine learning classifiers are able to discriminate between proteins belonging to different functional classes. we conclude that there is very good prospect of successfully predicting the function of yet uncharacterized proteins using machine learning classifiers trained on proteins of known function. predicting the function of newly discovered proteins by simply inspecting their amino acid sequence is one of the major challenges of post-genomic computational biology, especially when done without recourse to experimentation or homology information.
 predicting the function of newly discovered proteins by simply inspecting their amino acid sequence is one of the major challenges of post-genomic computational biology, especially when done without recourse to experimentation or homology information. machine learning classifiers are able to discriminate between proteins belonging to different functional classes. until now, however, it has been unclear if this ability would be transferable to proteins of unknown function, which may show distinct biases compared to experimentally more tractable proteins.  here we show that proteins with known and unknown function do indeed differ significantly. we then show that proteins from different bacterial species also differ to an even larger and very surprising extent, but that functional classifiers nonetheless generalize successfully across species boundaries. we also show that in the case of highly specialized proteomes classifiers from a different, but more conventional, species may in fact outperform the endogenous species-specific classifier.  we conclude that there is very good prospect of successfully predicting the function of yet uncharacterized proteins using machine learning classifiers trained on proteins of known function. 