BACKGROUND
advances in structural biology have increased the rate at which new protein structures are being determined, thus creating a need for automated methods for protein domain classification.
the main computational tools for classification in protein structure databases such as cath  <cit>  and scop  <cit>  remain sequence and structural comparisons.
indeed, in the processing of cath, for example, a high degree of structural similarity often warrants the direct inheritance of the classification of the matched known domains.
although it may be possible to classify protein domains purely on the basis of clear sequence and structural similarity, there are many cases that exhibit 'borderline' or low similarity to existing members which require laborious manual classification.
this manual classification usually requires study of the relevant literature, and so classification of these 'borderline' domains may benefit from automated literature analysis.
to address this need, text mining based methods may complement the existing molecular computational approaches, especially in cases where the evidence from such sequence and structural similarities is inconclusive.
support vector machines  are one of the newer machine learning approaches in wide usage today, and they are based on statistical methods to minimise the risk of error and offer solutions for optimal generalisation performance  <cit> .
svms exploit statistical learning theory and are capable of overcoming the problems commonly associated with high dimensionality, such as overfitting.
svms have been demonstrated to perform well in document classification tasks  <cit> .
in such applications, text documents are represented as vectors according to the bag-of-words model, whereby each word represents a dimension in a high-dimensionality space.
these vectors not only have high dimensionality, but they are also sparse at the same time, as each document typically contains a small subset of the very large set of words which are present in the corpus vocabulary.
svms are based on learning a separating hyperplane that divides two sets of vectors such that the risk of misclassification is minimized, and are particularly suitable for this kind of high dimensionality, sparse data.
there are several applications of svms for document classification.
prebind is an information extraction system based on svm technology for the detection of protein-protein interactions in the literature  <cit> .
stapley et al.
<cit>  used svms to infer the sub-cellular location of proteins from text sources.
rice et al.
<cit>  developed a machine learning approach to mine protein function predictions from text.
another svm based document classification algorithm was implemented to assign documents into nine main categories which correspond to chapters of the wormbook.
the system was optimised for the caenorhabditis elegans corpus using rules, but the same classification engine can be applied to other domains  <cit> .
several approaches already exist that incorporate text mining methods and other bioinformatics tools.
indeed, the combination of sensitive sequence similarity searches with functional annotations has been successfully implemented for the functional prediction of proteins, based on experimental knowledge of remote homologues  <cit> .
other attempts combine functional information with a variety of similarity search methods.
sawted  uses textual descriptions from several fields of uniprot records to enhance the detection of remote homologues from psi-blast results  <cit> .
profat combines psi-blast sequence similarity searches with fold recognition and text mining to refine hits according to their function  <cit> .
various methods are available for automatic functional annotation of proteins from text, and several of these have been evaluated in the second task of biocreative  <cit> .
couto et al., have developed an unsupervised method for recognizing biological properties in free text   <cit> .
the system splits text in sentences and considers the evidence content based on the nomenclature of a genomic ontology that structures the properties.
goannotator  <cit>  is able to link gene ontology  terms in uncurated annotations of uniprot entries with evidence text from documents related to such entries.
other methods for automatic identification of go terms in free text include the approach by ruch  <cit>  which is based on pattern matching and text categorisation ranking, and by gaudan et al.
<cit>  which considers not only the presence of words of the go terms occurring in text, but also proximity between words and their specificity based on their information content.
we have developed a novel text based approach for protein classification, which is based in text similarity of documents related to proteins, with a view to support the curation of protein structure databases.
it is assumed that textual similarity between sets of documents relating to proteins mirrors structural and functional relationships and therefore can be used to make protein classification assignments.
such documents usually contain a description of the protein function and protein structure.
in addition, they may mention specific characteristics, related or homologous proteins and their classification.
in general, they represent concepts that report on functional and structural details.
several of the words used in these descriptions are specific to each protein or protein class.
the method exploits the presence of such protein specific words  in document classification.
it is expected that documents related to proteins that belong to the same class, and therefore have similar function and structure, are more similar than documents related to proteins in different classes.
a classification assignment for a query protein can be inferred by assessing text similarity between a set of documents relating to the unclassified protein and sets of documents relating to classified proteins.
for instance, if documents related to an unclassified protein are similar to documents related to a classified protein, then it could be that they belong to the same class, or in this case, the same protein superfamily.
the documents can be abstracts or full text articles, although only abstracts have been used here due to the large variety of access control methods employed on journal full texts.
an svm model was developed to discriminate sentences in the abstracts containing functional, structural and classification information that are relevant to the protein classification task.
the svm model was used to remove text that is irrelevant to the classification task, thus reducing noise and increasing the prevalence of informative terms, allowing for more accurate classification predictions.
the svm model presented is novel and optimised for the discrimination of sentences in text that contain useful information for protein classification.
the text similarity algorithm was firstly optimised using a previously described gold standard dataset  <cit> .
several conditions were tested, such as the inclusion of additional text from related articles and uniprot  <cit>  and protein data bank  <cit>  annotations, as well as the filtering of the abstracts using the svm model.
the optimal conditions were then applied in text comparisons within a much larger collection of documents that relate to proteins already classified in the superfamily  level of the cath database.
although the structural similarity classifier performed best, text similarity was useful in a logistic regression model that combined the two classifiers.
the method is unique in incorporating structural similarity searches with text mining directly.
in a dataset of 'borderline' proteins, which lack clear structural and sequence similarity to classified proteins, the combination of the structural and text similarity classifiers resulted in an improvement in coverage by up to 50% at low error rates , compared to the structural classifier alone.
additionally, it makes an extra  <dig> % of correct classification decisions when only the highest scoring predictions were used to infer classification.
this method is useful for the challenging task of classification of such 'borderline' cases that usually require manual curation involving time-consuming study of the relevant literature.
application to cath database
the cath database is a hierarchical classification of protein domain structures in pdb.
there are four major levels in this hierarchy: class, architecture, topology  and homologous superfamily.
the latter level groups together domains where evidence from sequence, structure and function similarity suggest they have evolved from a common ancestor.
classification is currently guided by structure and sequence similarity measures such as cathedral  <cit> , ssap  <cit> , profile hmms  <cit>  and manual procedures such as literature analysis.
the ssap structure comparison method uses a double dynamic programming algorithm to align two protein structures.
in cath, an existing classification is inherited if a protein domain displays "clear" structural and sequence similarity  with a classified domain.
however, if there is no domain in the database that fulfils these requirements, the classification is performed manually by considering the results of structural and sequence similarity and analysis of relevant literature.
the literature often contains references to the function of the protein and even which evolutionary family it is thought to belong to.
it is this information that can be highly useful for classification and lends itself to a text mining approach.
RESULTS
combined structure and text classifier outperforms structural similarity in protein classification of 'borderline' cases in cath
an all-versus-all text comparison was performed using dc <dig>  as the query set and textcath as the reference set according to the optimal conditions identified in the gold standard enzyme dataset.
performance was assessed using the auc and mcc measures .
the structural similarity classifier ssap performed best, while the classification power of text similarity was lower as judged from the auc metric  and the mcc .
performance among the svm-filtered and intact reference sets was almost identical, however the svm filtered set  had fewer terms .
although the text classifier performance was average, its usefulness was investigated for adding value to the better performing structural classifier in cases with low structural and sequence similarity.
the performance of the structural similarity measured by the ssap algorithm and text similarity  as classifiers for protein classification in the homologous superfamily level in the dc <dig>  dataset of 'borderline' cases in cath using the textcath as reference set.
classification performance was assessed using the auc and mcc measures on the whole set , training , and test  sets.
nagelkerke's r <dig> is a measure of the variance accounted by the variables of the logistic regression models.
model l.r.
stands for model likelihood chi-square which is the difference between null and residual deviance.
logistic regression models were trained on a random subset of comparisons from  <dig> abstracts and tested on the remaining  <dig> abstracts from the 'borderline' cases dataset dc <dig> .
to check if it is possible to develop an optimal combination of the classifiers  by exploiting any orthogonality in our data, we generated a series of logistic regression models.
the inclusion of the text similarity  variable was significant  as judged from the increase in the likelihood ratio which reflects the difference between error not knowing the independent variables and error when the independents are included in the model .
the importance of the explanatory variables was also confirmed by carrying out wald chi-square tests for statistical significance .
in the model that included the structure and text similarity, both explanatory variables contributed significantly to the model effect, but the structural similarity classifier ssap accounted for the largest effect among the independents.
the equation coefficients of the model  are shown in table  <dig>
coefficients and wald z statistics of the logistic regression model ssap+text that includes ssap  and text  independent variables.
coeff = coefficient for the logistic regression; s.e.
= standard error; wald z, p = wald statistic with corresponding probability.
the discriminative power of the models  was evaluated using the auc statistic.
its value is the total fraction of cases where a classification match is ranked higher than a non-match.
values of auc for models that contain structural classifier ssap are already over  <dig>  indicating significant extant predictive capacity from structure alone.
in contrast, the auc for the text similarity classifier alone was only  <dig> , thus indicating lower classification power when textual information is considered in isolation.
nevertheless, this indicates that the text classification pipeline is extracting almost 88% of the structural information when assessed against the opinion of an expert curator.
clearly highly relevant texts are being extracted, which would be useful to a human curator in the intended practical application.
interestingly, inclusion of the text variable to the structural classifier improves the auc value to  <dig>  for the combined ssap and text model .
this clearly indicates that the text classifier not only provides consistent support for the structure-based classification, but also increases coverage.
classifier accuracy was further analysed using the matthews correlation coefficient .
the results indicated that the combined model ssap+text outperformed the ssap structural classifier .
the significance of the improvements in the mcc was evaluated using fisher's z test, which considers the magnitude of the difference and the strength of the correlation.
the improvements in mcc values were statistically significant .
further, the combined ssap and text model r <dig> was improved  compared to ssap , indicating that the model is useful in classification predictions .
these results demonstrated the real improvement in performance by the inclusion of text variable in the combined logistic regression models.
comparative classifier performance in protein classification
in order to compare the performance of the ssap, text and ssap+text classifiers "coverage versus error" plots were used  <cit> .
the plots were constructed by collecting all comparison scores for each classifier and ranking them in reverse order.
moving down the list, the numbers of matches  accumulated thus far were counted and plotted versus the number of non-matches  for each score cutoff .
for each classifier, the fraction of the total number of matches or coverage, was also calculated at certain error rates.
the fraction of the total number of matches is the number of true positives divided by the total number of true matches  in the test set.
the false positive rate is the fraction of non-matches for a score cutoff.
for instance, fpr of 10- <dig> indicates that the classifier makes  <dig> erroneous predictions out of the possible  <dig>  likewise, fpr of 10- <dig>  10- <dig> and 10- <dig> correspond to  <dig>   <dig>   <dig> and  <dig> errors, respectively.
table  <dig> shows the matches detected as an actual number and as a percentage of the total matches, for a range of fpr values.
the results are shown graphically in figure 2b.
coverage is the fraction of true classification matches and is shown as actual numbers and as a percentage of total tp .
scores range between  <dig> and  <dig>   <dig> and  <dig>  and  <dig> and  <dig> for the text, ssap and ssap + text classifiers, respectively.
total comparisons:  <dig>  positive matches:  <dig>
the data in figures 2a and 2b and table  <dig> demonstrate that at low false positive rates, 10- <dig> and 10- <dig>  all classifiers display low coverage.
at a higher rate of false positives , the ssap, text and ssap+text classifiers detect 10%,  <dig> % and  <dig> % of the true matches.
the ssap+text classifier consistently outperforms the structural classifier when used on its own, especially in the range of low error, with more than double the number of matches found for error rates below 10- <dig>  the tp-fp breakeven point is the number of true positives that equals the number of false positives.
in the test set for the ssap+text, the breakeven point is reached after  <dig> matches are detected .
both ssap and text predict more false than true positives for all cutoffs in the 'borderline' dataset.
aside from the ability of each classifier to separate true from false positives across the range of scores, we were also interested in the power of each method to assign correct structural classifications to proteins of the 'borderline' set in cath using the top scoring hit.
only the best scoring domain comparison was considered for each of the  <dig> domains of the dc <dig>  set , in order to simulate classification tasks.
the combined ssap+text classifier increased coverage by  <dig> %, capturing  <dig>  correct classifications relative to  <dig>  by ssap alone.
discussion
our results demonstrate that although much work is required to address fully-automated classification of proteins that lack clear structural or sequence similarity, the combination of text and structure similarity can improve classification performance for 'borderline' cases.
when text and structural similarity scores were compared for their classification power in a set of simulated 'borderline' cases of the cath protein structure database, structural similarity scores performed best in the h  level of cath.
this may be expected for structural databases, where structural relationships primarily define class membership.
although the ssap scores perform better than text similarity in classification as single predictors, the performance improved considerably when structure and text similarity classifiers were combined in a logistic regression model.
on a dataset of 'borderline' cases, a significant improvement in coverage by up to 50% at low error rates was observed.
for equivalent coverage, the combined classifier benefits from a significant reduction in the number of false positives compared to ssap.
for instance, when ssap captures  <dig> true matches, it also includes  <dig> false positives, while the ssap+text classifier includes  <dig>  which is a 65% reduction.
moreover, the combined classifier was able to make an additional  <dig> % of correct classification predictions based only on the top scoring hits, compared to ssap alone.
the combined classifier is therefore a better starting point than ssap on its own, when classifying a 'borderline' protein.
although the error rate is not sufficiently low to enable fully automatic protein classification of 'borderline' cases, the combined classifier can have practical application in the support and evaluation of manual protein domain classification assignments.
more development of the text similarity algorithm using a more elaborate weighting scheme combined with more efficient text retrieval and information extraction is required to improve performance further.
the text similarity algorithm was benchmarked and optimised using a manually curated dataset of enzyme superfamilies, before it was applied on a large dataset of 'borderline' cases in cath.
removal of sentences which were semantically irrelevant to protein classification using an svm model resulted in better performance of the text classifier, while the dimensionality of the feature space of the document reference set of cath was reduced by  <dig> % making calculations speedier.
although all terms were used in this implementation, performance may improve further with feature selection based on thresholding metrics, such as document frequency, information gain, mutual information or chi-square and more involved weighting schemes.
for example, the weights of a specific set of highly informative terms can be boosted.
for the cath database reference set, these terms would include selected names and synonyms of the superfamily and their protein members and function annotations from uniprot.
performance should also benefit by using a more sophisticated term generation scheme that accounts for synonymy.
all words of the abstracts, including title and authors, were used in our method.
typically abstracts contain a brief description of the protein structure.
a more detailed and extensive description of important features may only be mentioned in the main article, so full text is expected to yield better results.
irrelevant sentences can be removed by classifying each sentence of the full text article using the svm filter.
the inclusion of additional abstracts from related articles in the text related to each protein of the reference set improved performance.
abstracts which are related to the main reference usually contain additional terms that are likely to be characteristic of each class.
as full text articles are not always available in public databases, this is an alternative way to muster additional relevant text for each protein and leverage the lack of full text.
a more comprehensive selection of texts by inclusion of abstracts of relevant articles sourced from uniprot and genbank records may provide more comprehensive text sources for the query proteins.
annotations from pdb and uniprot may also be used alternatively or additionally to abstracts in the query proteins.
in this implementation, abstracts were used as text relating to the query protein of the test set because they are more readily available.
moreover, it is common for annotations to contain technical terms , which may introduce bias to the text similarity classifier.
they also lack consistency and reliability because not all proteins are fully annotated yet, while several annotations are propagated based on high homology rather than experimental observations.
structural database entries are protein domains, however domains lack a one-to-one relationship to pubmed abstracts.
in fact, it is assumed that each abstract possesses information in all domains of its related protein.
in reality it is quite frequent for certain domains within a protein to attract greater interest, and text, than others.
inclusion of additional abstracts or full text articles related to the query protein will increase the likelihood that information relevant to all domains of the protein is present in the text.
combination of the text similarity algorithm with other structural similarity algorithms, such as ce  <cit> , dali  <cit>  and msdfold  <cit>  via logistic regression or machine learning based methods, may provide a useful performance benefit.
the same methodology, with little alteration, can also be used to improve fold recognition and structure prediction results.
moreover, the text similarity algorithm can be proven valuable in protein classification tasks where a more accurate function classifier is not available.
for example, it may be useful in enzyme classification.
in the enzyme commission classification system, enzymes are classified according to the chemical reaction they catalyse.
it is likely that text similarity may be a more appropriate classifier than structure or sequence similarity for this database.
finally, we demonstrated the practical application of a text based classifier in protein structure database curation.
the model resulting from its combination with the structural classifier is superior to the structural classifier alone, thus providing an improved way to classify 'borderline' proteins in the cath protein structure database.
although in the context of full automation the improvements might at first sight look relatively moderate, it is important to bear in mind that in the principal application domain, the textual results are intended to be used by manual curators or users of a structure classification server as a guide to manual classification.
putting it another way, we need the textual results to be confirmatory of the structural similarity results rather than being entirely novel.
the fact that our text classification scheme reproduces around 88% of the purely structure-based auc, and in combination increases the auc by a small but significant amount, shows that we are indeed extracting the most relevant texts and that some of these texts are key to making better informed decisions on superfamily membership.
a user of a hybrid system would therefore be provided with a highly relevant shortlist of texts from which he or she can make an informed decision as to the correct superfamily classification of the protein being analysed.
in other words, the fact that we can improve automatic classification is actually far less important that the fact that we are able to select the relevant texts which can be further analysed by the user in semi-automatic classification.
consequently, we are planning to build this text classification functionality into the user interfaces of our existing fold recognition web servers: both the sequence-based genthreader  <cit> , and the structure-based cathedral server  <cit> .
overall, our findings show a useful combination of a structural similarity with a text mining approach and demonstrate the value of the text based approaches in protein classification.
CONCLUSIONS
in summary, a text based classifier was developed and implemented for the classification of proteins in the cath database.
although the structural similarity scores perform better than text in classification of proteins in structure databases, it was proven that the combination of the structure and text classifiers in a logistic regression model provides a more powerful classifier, significantly increasing coverage especially at low error levels compared to using structural similarity alone.
the benefit is particularly useful in cases where structural similarity is not high enough to be conclusive.
we found that, for 'borderline' matches with ssap scores below  <dig>  which are notoriously difficult to classify, it is preferable to use the combined structure and text similarity classifier than ssap alone.
this result should be useful in the development of servers which aim to classify proteins automatically and reliably.

 advances in structural biology have increased the rate at which new protein structures are being determined, thus creating a need for automated methods for protein domain classification. the main computational tools for classification in protein structure databases such as cath and scop remain sequence and structural comparisons. indeed, in the processing of cath, for example, a high degree of structural similarity often warrants the direct inheritance of the classification of the matched known domains. although it may be possible to classify protein domains purely on the basis of clear sequence and structural similarity, there are many cases that exhibit'borderline' or low similarity to existing members which require laborious manual classification. this manual classification usually requires study of the relevant literature, and so classification of these'borderline' domains may benefit from automated literature analysis. to address this need, text mining based methods may complement the existing molecular computational approaches, especially in cases where the evidence from such sequence and structural similarities is inconclusive. support vector machines are one of the newer machine learning approaches in wide usage today, and they are based on statistical methods to minimise the risk of error and offer solutions for optimal generalisation performance. svms exploit statistical learning theory and are capable of overcoming the problems commonly associated with high dimensionality, such as overfitting. svms have been demonstrated to perform well in document classification tasks. in such applications, text documents are represented as vectors according to the bag-of-words model, whereby each word represents a dimension in a high-dimensionality space. these vectors not only have high dimensionality, but they are also sparse at the same time, as each document typically contains a small subset of the very large set of words which are present in the corpus vocabulary. svms are based on learning a separating hyperplane that divides two sets of vectors such that the risk of misclassification is minimized, and are particularly suitable for this kind of high dimensionality, sparse data. there are several applications of svms for document classification. prebind is an information extraction system based on svm technology for the detection of protein-protein interactions in the literature. stapley et al. used svms to infer the sub-cellular location of proteins from text sources. rice et al. developed a machine learning approach to mine protein function predictions from text. another svm based document classification algorithm was implemented to assign documents into nine main categories which correspond to chapters of the wormbook. the system was optimised for the caenorhabditis elegans corpus using rules, but the same classification engine can be applied to other domains. several approaches already exist that incorporate text mining methods and other bioinformatics tools. indeed, the combination of sensitive sequence similarity searches with functional annotations has been successfully implemented for the functional prediction of proteins, based on experimental knowledge of remote homologues. other attempts combine functional information with a variety of similarity search methods. sawted uses textual descriptions from several fields of uniprot records to enhance the detection of remote homologues from psi-blast results. profat combines psi-blast sequence similarity searches with fold recognition and text mining to refine hits according to their function. various methods are available for automatic functional annotation of proteins from text, and several of these have been evaluated in the second task of biocreative. couto et al., have developed an unsupervised method for recognizing biological properties in free text. the system splits text in sentences and considers the evidence content based on the nomenclature of a genomic ontology that structures the properties. goannotator is able to link gene ontology terms in uncurated annotations of uniprot entries with evidence text from documents related to such entries. other methods for automatic identification of go terms in free text include the approach by ruch which is based on pattern matching and text categorisation ranking, and by gaudan et al. which considers not only the presence of words of the go terms occurring in text, but also proximity between words and their specificity based on their information content. we have developed a novel text based approach for protein classification, which is based in text similarity of documents related to proteins, with a view to support the curation of protein structure databases. it is assumed that textual similarity between sets of documents relating to proteins mirrors structural and functional relationships and therefore can be used to make protein classification assignments. such documents usually contain a description of the protein function and protein structure. in addition, they may mention specific characteristics, related or homologous proteins and their classification. in general, they represent concepts that report on functional and structural details. several of the words used in these descriptions are specific to each protein or protein class. the method exploits the presence of such protein specific words in document classification. it is expected that documents related to proteins that belong to the same class, and therefore have similar function and structure, are more similar than documents related to proteins in different classes. a classification assignment for a query protein can be inferred by assessing text similarity between a set of documents relating to the unclassified protein and sets of documents relating to classified proteins. for instance, if documents related to an unclassified protein are similar to documents related to a classified protein, then it could be that they belong to the same class, or in this case, the same protein superfamily. the documents can be abstracts or full text articles, although only abstracts have been used here due to the large variety of access control methods employed on journal full texts. an svm model was developed to discriminate sentences in the abstracts containing functional, structural and classification information that are relevant to the protein classification task. the svm model was used to remove text that is irrelevant to the classification task, thus reducing noise and increasing the prevalence of informative terms, allowing for more accurate classification predictions. the svm model presented is novel and optimised for the discrimination of sentences in text that contain useful information for protein classification. the text similarity algorithm was firstly optimised using a previously described gold standard dataset. several conditions were tested, such as the inclusion of additional text from related articles and uniprot and protein data bank annotations, as well as the filtering of the abstracts using the svm model. the optimal conditions were then applied in text comparisons within a much larger collection of documents that relate to proteins already classified in the superfamily level of the cath database. although the structural similarity classifier performed best, text similarity was useful in a logistic regression model that combined the two classifiers. the method is unique in incorporating structural similarity searches with text mining directly. in a dataset of'borderline' proteins, which lack clear structural and sequence similarity to classified proteins, the combination of the structural and text similarity classifiers resulted in an improvement in coverage by up to 50% at low error rates, compared to the structural classifier alone. additionally, it makes an extra % of correct classification decisions when only the highest scoring predictions were used to infer classification. this method is useful for the challenging task of classification of such'borderline' cases that usually require manual curation involving time-consuming study of the relevant literature. application to cath database the cath database is a hierarchical classification of protein domain structures in pdb. there are four major levels in this hierarchy: class, architecture, topology and homologous superfamily. the latter level groups together domains where evidence from sequence, structure and function similarity suggest they have evolved from a common ancestor. classification is currently guided by structure and sequence similarity measures such as cathedral, ssap, profile hmms and manual procedures such as literature analysis. the ssap structure comparison method uses a double dynamic programming algorithm to align two protein structures. in cath, an existing classification is inherited if a protein domain displays "clear" structural and sequence similarity with a classified domain. however, if there is no domain in the database that fulfils these requirements, the classification is performed manually by considering the results of structural and sequence similarity and analysis of relevant literature. the literature often contains references to the function of the protein and even which evolutionary family it is thought to belong to. it is this information that can be highly useful for classification and lends itself to a text mining approach.  combined structure and text classifier outperforms structural similarity in protein classification of'borderline' cases in cath an all-versus-all text comparison was performed using dc as the query set and textcath as the reference set according to the optimal conditions identified in the gold standard enzyme dataset. performance was assessed using the auc and mcc measures. the structural similarity classifier ssap performed best, while the classification power of text similarity was lower as judged from the auc metric and the mcc. performance among the svm-filtered and intact reference sets was almost identical, however the svm filtered set had fewer terms. although the text classifier performance was average, its usefulness was investigated for adding value to the better performing structural classifier in cases with low structural and sequence similarity. the performance of the structural similarity measured by the ssap algorithm and text similarity as classifiers for protein classification in the homologous superfamily level in the dc dataset of'borderline' cases in cath using the textcath as reference set. classification performance was assessed using the auc and mcc measures on the whole set, training, and test sets. nagelkerke's r is a measure of the variance accounted by the variables of the logistic regression models. model l.r. stands for model likelihood chi-square which is the difference between null and residual deviance. logistic regression models were trained on a random subset of comparisons from abstracts and tested on the remaining abstracts from the'borderline' cases dataset dc. to check if it is possible to develop an optimal combination of the classifiers by exploiting any orthogonality in our data, we generated a series of logistic regression models. the inclusion of the text similarity variable was significant as judged from the increase in the likelihood ratio which reflects the difference between error not knowing the independent variables and error when the independents are included in the model. the importance of the explanatory variables was also confirmed by carrying out wald chi-square tests for statistical significance. in the model that included the structure and text similarity, both explanatory variables contributed significantly to the model effect, but the structural similarity classifier ssap accounted for the largest effect among the independents. the equation coefficients of the model are shown in table coefficients and wald z statistics of the logistic regression model ssap+text that includes ssap and text independent variables. coeff = coefficient for the logistic regression; s.e. = standard error; wald z, p = wald statistic with corresponding probability. the discriminative power of the models was evaluated using the auc statistic. its value is the total fraction of cases where a classification match is ranked higher than a non-match. values of auc for models that contain structural classifier ssap are already over indicating significant extant predictive capacity from structure alone. in contrast, the auc for the text similarity classifier alone was only, thus indicating lower classification power when textual information is considered in isolation. nevertheless, this indicates that the text classification pipeline is extracting almost 88% of the structural information when assessed against the opinion of an expert curator. clearly highly relevant texts are being extracted, which would be useful to a human curator in the intended practical application. interestingly, inclusion of the text variable to the structural classifier improves the auc value to for the combined ssap and text model. this clearly indicates that the text classifier not only provides consistent support for the structure-based classification, but also increases coverage. classifier accuracy was further analysed using the matthews correlation coefficient. the results indicated that the combined model ssap+text outperformed the ssap structural classifier. the significance of the improvements in the mcc was evaluated using fisher's z test, which considers the magnitude of the difference and the strength of the correlation. the improvements in mcc values were statistically significant. further, the combined ssap and text model r was improved compared to ssap, indicating that the model is useful in classification predictions. these results demonstrated the real improvement in performance by the inclusion of text variable in the combined logistic regression models. comparative classifier performance in protein classification in order to compare the performance of the ssap, text and ssap+text classifiers "coverage versus error" plots were used. the plots were constructed by collecting all comparison scores for each classifier and ranking them in reverse order. moving down the list, the numbers of matches accumulated thus far were counted and plotted versus the number of non-matches for each score cutoff. for each classifier, the fraction of the total number of matches or coverage, was also calculated at certain error rates. the fraction of the total number of matches is the number of true positives divided by the total number of true matches in the test set. the false positive rate is the fraction of non-matches for a score cutoff. for instance, fpr of 10- indicates that the classifier makes erroneous predictions out of the possible likewise, fpr of 10- 10- and 10- correspond to and errors, respectively. table shows the matches detected as an actual number and as a percentage of the total matches, for a range of fpr values. the results are shown graphically in figure 2b. coverage is the fraction of true classification matches and is shown as actual numbers and as a percentage of total tp. scores range between and and and and for the text, ssap and ssap + text classifiers, respectively. total comparisons: positive matches: the data in figures 2a and 2b and table demonstrate that at low false positive rates, 10- and 10- all classifiers display low coverage. at a higher rate of false positives, the ssap, text and ssap+text classifiers detect 10%, % and % of the true matches. the ssap+text classifier consistently outperforms the structural classifier when used on its own, especially in the range of low error, with more than double the number of matches found for error rates below 10- the tp-fp breakeven point is the number of true positives that equals the number of false positives. in the test set for the ssap+text, the breakeven point is reached after matches are detected. both ssap and text predict more false than true positives for all cutoffs in the'borderline' dataset. aside from the ability of each classifier to separate true from false positives across the range of scores, we were also interested in the power of each method to assign correct structural classifications to proteins of the'borderline' set in cath using the top scoring hit. only the best scoring domain comparison was considered for each of the domains of the dc set, in order to simulate classification tasks. the combined ssap+text classifier increased coverage by %, capturing correct classifications relative to by ssap alone. discussion our results demonstrate that although much work is required to address fully-automated classification of proteins that lack clear structural or sequence similarity, the combination of text and structure similarity can improve classification performance for'borderline' cases. when text and structural similarity scores were compared for their classification power in a set of simulated'borderline' cases of the cath protein structure database, structural similarity scores performed best in the h level of cath. this may be expected for structural databases, where structural relationships primarily define class membership. although the ssap scores perform better than text similarity in classification as single predictors, the performance improved considerably when structure and text similarity classifiers were combined in a logistic regression model. on a dataset of'borderline' cases, a significant improvement in coverage by up to 50% at low error rates was observed. for equivalent coverage, the combined classifier benefits from a significant reduction in the number of false positives compared to ssap. for instance, when ssap captures true matches, it also includes false positives, while the ssap+text classifier includes which is a 65% reduction. moreover, the combined classifier was able to make an additional % of correct classification predictions based only on the top scoring hits, compared to ssap alone. the combined classifier is therefore a better starting point than ssap on its own, when classifying a'borderline' protein. although the error rate is not sufficiently low to enable fully automatic protein classification of'borderline' cases, the combined classifier can have practical application in the support and evaluation of manual protein domain classification assignments. more development of the text similarity algorithm using a more elaborate weighting scheme combined with more efficient text retrieval and information extraction is required to improve performance further. the text similarity algorithm was benchmarked and optimised using a manually curated dataset of enzyme superfamilies, before it was applied on a large dataset of'borderline' cases in cath. removal of sentences which were semantically irrelevant to protein classification using an svm model resulted in better performance of the text classifier, while the dimensionality of the feature space of the document reference set of cath was reduced by % making calculations speedier. although all terms were used in this implementation, performance may improve further with feature selection based on thresholding metrics, such as document frequency, information gain, mutual information or chi-square and more involved weighting schemes. for example, the weights of a specific set of highly informative terms can be boosted. for the cath database reference set, these terms would include selected names and synonyms of the superfamily and their protein members and function annotations from uniprot. performance should also benefit by using a more sophisticated term generation scheme that accounts for synonymy. all words of the abstracts, including title and authors, were used in our method. typically abstracts contain a brief description of the protein structure. a more detailed and extensive description of important features may only be mentioned in the main article, so full text is expected to yield better results. irrelevant sentences can be removed by classifying each sentence of the full text article using the svm filter. the inclusion of additional abstracts from related articles in the text related to each protein of the reference set improved performance. abstracts which are related to the main reference usually contain additional terms that are likely to be characteristic of each class. as full text articles are not always available in public databases, this is an alternative way to muster additional relevant text for each protein and leverage the lack of full text. a more comprehensive selection of texts by inclusion of abstracts of relevant articles sourced from uniprot and genbank records may provide more comprehensive text sources for the query proteins. annotations from pdb and uniprot may also be used alternatively or additionally to abstracts in the query proteins. in this implementation, abstracts were used as text relating to the query protein of the test set because they are more readily available. moreover, it is common for annotations to contain technical terms, which may introduce bias to the text similarity classifier. they also lack consistency and reliability because not all proteins are fully annotated yet, while several annotations are propagated based on high homology rather than experimental observations. structural database entries are protein domains, however domains lack a one-to-one relationship to pubmed abstracts. in fact, it is assumed that each abstract possesses information in all domains of its related protein. in reality it is quite frequent for certain domains within a protein to attract greater interest, and text, than others. inclusion of additional abstracts or full text articles related to the query protein will increase the likelihood that information relevant to all domains of the protein is present in the text. combination of the text similarity algorithm with other structural similarity algorithms, such as ce, dali and msdfold via logistic regression or machine learning based methods, may provide a useful performance benefit. the same methodology, with little alteration, can also be used to improve fold recognition and structure prediction results. moreover, the text similarity algorithm can be proven valuable in protein classification tasks where a more accurate function classifier is not available. for example, it may be useful in enzyme classification. in the enzyme commission classification system, enzymes are classified according to the chemical reaction they catalyse. it is likely that text similarity may be a more appropriate classifier than structure or sequence similarity for this database. finally, we demonstrated the practical application of a text based classifier in protein structure database curation. the model resulting from its combination with the structural classifier is superior to the structural classifier alone, thus providing an improved way to classify'borderline' proteins in the cath protein structure database. although in the context of full automation the improvements might at first sight look relatively moderate, it is important to bear in mind that in the principal application domain, the textual results are intended to be used by manual curators or users of a structure classification server as a guide to manual classification. putting it another way, we need the textual results to be confirmatory of the structural similarity results rather than being entirely novel. the fact that our text classification scheme reproduces around 88% of the purely structure-based auc, and in combination increases the auc by a small but significant amount, shows that we are indeed extracting the most relevant texts and that some of these texts are key to making better informed decisions on superfamily membership. a user of a hybrid system would therefore be provided with a highly relevant shortlist of texts from which he or she can make an informed decision as to the correct superfamily classification of the protein being analysed. in other words, the fact that we can improve automatic classification is actually far less important that the fact that we are able to select the relevant texts which can be further analysed by the user in semi-automatic classification. consequently, we are planning to build this text classification functionality into the user interfaces of our existing fold recognition web servers: both the sequence-based genthreader, and the structure-based cathedral server. overall, our findings show a useful combination of a structural similarity with a text mining approach and demonstrate the value of the text based approaches in protein classification.  in summary, a text based classifier was developed and implemented for the classification of proteins in the cath database. although the structural similarity scores perform better than text in classification of proteins in structure databases, it was proven that the combination of the structure and text classifiers in a logistic regression model provides a more powerful classifier, significantly increasing coverage especially at low error levels compared to using structural similarity alone. the benefit is particularly useful in cases where structural similarity is not high enough to be conclusive. we found that, for'borderline' matches with ssap scores below which are notoriously difficult to classify, it is preferable to use the combined structure and text similarity classifier than ssap alone. this result should be useful in the development of servers which aim to classify proteins automatically and reliably. 
testing this classification scheme on a dataset of'borderline' protein domains that lack significant sequence or structure similarity to classified proteins showed that although, as expected, the structural similarity classifiers perform better on average, there is a significant benefit in incorporating text similarity in logistic regression models, indicating significant orthogonality in this additional information. an optimal strategy for the text comparisons was identified by using an established gold standard enzyme dataset. finally when only the highest scoring predictions were used to infer classification, an extra % of correct decisions were made by the combined classifier. the classification of protein domains in the cath resource is primarily based on structural comparisons, sequence similarity and manual analysis. the method is based on the assumption that textual similarity between sets of documents relating to proteins reflects biological function similarities and can be exploited to make classification decisions. we have described a simple text based method to classify protein domains that demonstrates an improvement over existing methods. coverage was significantly increased especially at low error rates, which is important for routine classification tasks: % for the combined structure and text classifier compared to 10% for the structural classifier alone, at 10- error rate. filtering of the abstracts using a machine learning approach to discriminate sentences containing functional, structural and classification information that are relevant to the protein classification task improved performance. the method is unique in incorporating structural and text based classifiers directly and is particularly useful in cases where inconclusive evidence from sequence or structure similarity requires laborious manual classification. one of the main bottlenecks in the processing of new entries is the evaluation of'borderline' cases by human curators with reference to the literature, and better tools for helping both expert and non-expert users quickly identify relevant functional information from text are urgently needed. a text based method for protein classification is presented, which complements the existing sequence and structure-based approaches, especially in cases exhibiting low similarity to existing members and requiring manual intervention.
 the classification of protein domains in the cath resource is primarily based on structural comparisons, sequence similarity and manual analysis. one of the main bottlenecks in the processing of new entries is the evaluation of'borderline' cases by human curators with reference to the literature, and better tools for helping both expert and non-expert users quickly identify relevant functional information from text are urgently needed. a text based method for protein classification is presented, which complements the existing sequence and structure-based approaches, especially in cases exhibiting low similarity to existing members and requiring manual intervention. the method is based on the assumption that textual similarity between sets of documents relating to proteins reflects biological function similarities and can be exploited to make classification decisions.  an optimal strategy for the text comparisons was identified by using an established gold standard enzyme dataset. filtering of the abstracts using a machine learning approach to discriminate sentences containing functional, structural and classification information that are relevant to the protein classification task improved performance. testing this classification scheme on a dataset of'borderline' protein domains that lack significant sequence or structure similarity to classified proteins showed that although, as expected, the structural similarity classifiers perform better on average, there is a significant benefit in incorporating text similarity in logistic regression models, indicating significant orthogonality in this additional information. coverage was significantly increased especially at low error rates, which is important for routine classification tasks: % for the combined structure and text classifier compared to 10% for the structural classifier alone, at 10- error rate. finally when only the highest scoring predictions were used to infer classification, an extra % of correct decisions were made by the combined classifier.  we have described a simple text based method to classify protein domains that demonstrates an improvement over existing methods. the method is unique in incorporating structural and text based classifiers directly and is particularly useful in cases where inconclusive evidence from sequence or structure similarity requires laborious manual classification. 