BACKGROUND
one of the main challenges of genome projects is providing reliable functional annotations for gene products.
for extensively studied model organisms, such as the nematode c. elegans, high quality functional annotations are largely gleaned from manual curation of experiments reported in the published literature, with additional annotations obtained from computational or comparative methods, such as protein domain analysis  <cit> .
for organisms with smaller research communities, however, functional annotations may initially derive largely from computational or comparative methods which, in turn, can rely heavily upon the accuracy and completeness of model organism genome curation for providing suitable reference annotations and training sets  <cit> .
thus, the extent to which model organism databases can keep pace with annotating an ever expanding literature will potentially have an impact not only on model organism genome curation but curation of a variety of other genomes as well.
given that manual curation is unlikely to keep up with current publication rates, developing new approaches to extracting biological facts from the published literature is imperative  <cit> .
introduced over ten years ago, the gene ontology  has since become the de facto resource for functional genome annotation using controlled vocabularies  <cit> .
divided into three distinct ontologies that describe biological processes, molecular functions, and cellular components, the go is used by database curators to record key biological features of a gene product in language that is both humanly readable and computationally amenable.
a key feature of the go is that its ontologies are structured as directed acyclic graphs  in which terms have parent-child relationships, with child terms being more specific or specialized than their respective parent.
for example, the go term mitochondrion is a child of intracellular membrane-bounded organelle which is, in turn, a child of intracellular organelle.
annotations made to more specialized child terms may thus be transitively made to the more general parent terms, as well: an annotation to mitochondrion is, transitively, an annotation to intracellular organelle.
a second key feature of the go is the use of evidence codes to support annotations.
evidence codes are used by curators to give an indication of the methodology that researchers use to infer facts about the genes or gene products they are studying.
for example, annotation of a gene product to the biological process term cell division  based upon a mutant phenotype that results in arrested cell division would use the inferred from mutant phenotype  evidence code.
likewise, annotation of a gene product to the cellular component term plasma membrane  based upon immunofluorescence experiments would use the inferred from direct assay  evidence code.
selection of the appropriate go evidence code thus requires information about the experiment or assay used in a publication which is often found only by reading the full text.
manual go curation is, therefore, a labor-intensive process that if thorough can require reading and annotating the full text of hundreds, if not thousands, of publications.
thus, there is a growing need for semi- or fully-automated go curation strategies that will help database curators rapidly and accurately identify key experimental results in the full text of research articles.
natural language processing  applications offer a promising approach to aiding manual go curation.
such applications use various methodologies, including classification of text within documents, to assist in associating gene products to go terms.
while some of these applications have met with considerable success in suggesting possible go annotations, only a few take into account a real world database curation pipeline in which go curators examine articles from a wide variety of journals and use the full text of these articles to first select a go term and then evaluate the experimental methodology to confidently select the appropriate go evidence code, an absolute necessity for making a go annotation  <cit> .
thus, there is a need for text mining tools that can accurately mimic the manual curation process and thus be reliably incorporated into a database curation pipeline.
here, we describe our strategy for using natural language processing, in particular the textpresso text mining system  <cit> , to curate experimentally determined subcellular localization of c. elegans proteins using the cellular component ontology of go.
textpresso, an open source text mining tool, functions as both a simple search engine and a pattern-based, information extraction engine that employs categories of conceptually related words to semantically mark up the full text of papers.
examples of textpresso categories include disease, phenotype, and regulation, which contain words and phrases such as 'bardet-biedl syndrome1', 'cell fate transformation', and 'downregulate', respectively.
textpresso searches may be performed using keywords and/or categories, and the results are presented as a list of sentences containing terms and phrases that match the search criteria , numerically ranked according to the number of terms or phrases in the sentence that match the search string.
at present, there are  <dig> implementations of textpresso worldwide, including textpresso for neuroscience  <cit> , budding yeast, drosophila, arabidopsis, and e. coli, with over  <dig> categories, such as brain area and human disease, fly body parts, and arabidopsis genes, to aid in searching and fact extraction.
we focused on go cellular component annotations for our studies for three main reasons that suggested to us it would serve as a potential first proof of principle for developing a textpresso-based, semi-automated curation pipeline for a heretofore fully manual approach.
the first reason is that cellular component annotations are generally the result of a limited number of experimental strategies, namely microscopy and subcellular fractionation, thus potentially limiting the number of terms we would need to create new curation task-specific categories.
second, because we had noted during extensive manual curation that the information for the conclusion, i.e., subcellular localization, and the type of experimental assay are often stated in the same sentence, our search strategy of finding sentences that include proteins, cellular components and words or phrases referring to the experimental approach, was likely to be successful.
third, cellular component annotations derived from the small-scale experiments we were aiming to curate are typically made using one go evidence code, "inferred from direct assay ", thus simplifying the annotation process.
in addition, there is a need for cellular component curation tools that make use of the full text of research articles, as authors often fail to include the results of such experiments in the abstracts of their papers.
to illustrate, for a random sampling of  <dig> c. elegans proteins  only  <dig> % of possible go cellular component annotations could be made solely from pubmed abstracts compared to full text, with only  <dig> % of those annotations as specific as annotations made from full text.
to investigate the potential usefulness of textpresso for go cellular component curation, we constructed three new textpresso categories, termed cellular components, assay terms, and verbs, containing terms and phrases found in a gold-standard set of sentences describing experimentally determined subcellular localization.
to assess the performance of the new categories, we used them to annotate previously uncurated c. elegans proteins and found that by using textpresso we were able to make  <dig> % of all possible cellular component annotations with  <dig> % accuracy.
further, by comparing the relative efficiencies of manual versus textpresso-based curation, we find that textpresso-based curation has the potential to improve curation efficiency at least eight-fold, and possibly as much as 15-fold, depending upon the individual curator.
by incorporating a textpresso-based cellular component curation pipeline into wormbase, we have moved from fully manual curation to computer-assisted validation for this data type.
RESULTS
development of new textpresso categories
our approach to developing textpresso categories for go cellular component curation is outlined in figure  <dig> and described in detail in the methods.
to identify words and phrases relevant to reports of subcellular localization experiments, we collected ~ <dig>  sentences from papers reporting experimentally determined subcellular localization, and then analyzed lists of words and phrases used in the sentences, as well as the frequency with which the words and phrases occur, to manually select terms that authors use to describe their experimental results .
words and phrases identified by our word usage and frequency analysis were then manually sorted into three categories: cellular components, assay terms, and verbs, and included terms such as: nucleus, cell body, centrosomal; expression, antibody, throughout; and detect, exhibited, revealed, respectively .
while the division into three categories may seem somewhat arbitrary, it is semantically meaningful.
each of the three categories, along with the name of a c. elegans protein, captures the essential aspects of a localization experiment: a protein object, a cellular component, an action verb that reports the result, and an assay term that has a high probability of linking terms from the three other categories to an actual experimental result.
requiring a match to at least one term in each of multiple categories affords higher specificity than if all phrases were used to form just one category.
in addition, employing multiple categories aids in future category development, as it makes it possible to obtain additional terms and phrases for each category by performing searches requiring matches, for example, to only three of the four categories, thereby identifying sentences that might contain new terms or phrases to potentially be included in the omitted category.
testing strategy
to evaluate the performance of the three new textpresso categories, we measured the recall and precision of searches employing the three new categories at the document, sentence, and annotation level.
in essence: did the search find the correct documents?
did the search return the correct sentences?
could a curator make the correct annotations from those sentences?
in each case, we compared the results of textpresso-based curation to a gold standard of fully manual curation.
for each test, we performed a textpresso search for sentences that contained matches to at least one term in each of the categories, plus a match to one of  <dig> previously uncurated c. elegans proteins chosen at random.
as the c. elegans research community strives to adhere to standard nomenclature practices  we felt that this was a reasonable first approach to testing the effectiveness of the new categories.
we also hoped that by analyzing a diverse set of proteins, we could more accurately assess the performance of the new categories on the existing corpus of c. elegans literature.
to evaluate the results of our test searches, we used standard metrics of information retrieval systems: recall, precision, and f-measure  <cit> .
recall is defined as the ratio of correct answers given by the system to the total number of possible correct answers in the text; recall thus reflects the completeness or coverage of the system.
precision, on the other hand, is defined as the ratio of the number of correct answers given by a system to the total number of answers given by the system; precision thus reflects the relevance of the search results.
in addition, the f-measure, or f-score, is reported as an indication of the accuracy of the test.
test results: documents
for the  <dig> proteins in our test set, textpresso searches using the protein name and the three new categories returned  <dig> primary research articles  with the maximum number of papers returned being  <dig> for the lag- <dig> protein and the minimum being zero, for the agr- <dig> and pak- <dig> proteins.
as determined manually, the  <dig> proteins in our test set were associated with  <dig> papers containing subcellular localization data in the wormbase bibliography.
therefore, to assess recall, we divided the number of correct papers returned by textpresso,  <dig>  by the total number of papers,  <dig>  containing subcellular localization data for all of the genes in our test set as determined manually.
at the document level, our recall rate is thus  <dig> %.
to determine the rate of precision, we divided the number of correct papers returned,  <dig>  by the total number of papers returned,  <dig>  to determine that the precision of our searches was  <dig> %.
thus, for document retrieval we achieved an f-score of  <dig> % .
test results: sentences
to determine the precision and recall rates of returned sentences, we took the following approach: each sentence in every document of our test set  was examined manually in order to select all true positive sentences.
for these purposes, we defined true positives as those sentences that describe experimentally determined subcellular localization.
the true positive sentence set thus includes all sentences from which a curator could make go annotations, as well as those sentences that, while describing subcellular localization, would not automatically result in a go annotation.
for example, sentences describing a protein's subcellular localization in a wild-type background are suitable for go annotation, while sentences describing localization in a mutant background, although often reported with similar, if not identical language, would not typically be annotated for go.
reducing or eliminating the latter type of sentences  from our search results will be a focus of future tool development .
by manual inspection, we identified  <dig> true positive sentences in the  <dig> articles we analyzed .
of the  <dig> true positive sentences, only  <dig> were returned by textpresso, giving a recall rate, at the sentence level, of  <dig> %.
the precision rate, however, was considerably higher, at  <dig> % .
the f-score for sentence returns was thus  <dig> % .
recall and precision: go annotations
one of the main motivations for developing our new textpresso categories was to improve the efficiency with which we assign go cellular component annotations.
therefore, we also evaluated the recall and precision of the go annotations made from our search returns.
for the purposes of this analysis, we scored each annotation just once regardless of the number of times the information supporting that annotation was stated in the paper, as this reflects the normal curation process in which curators do not need to make a separate annotation for repeated statements of the same experimental result.
from the results of our textpresso searches we were able to make  <dig> of  <dig> possible go annotations made by manual curation.
thus, our annotation recall was  <dig> %.
of the  <dig> annotations we made from textpresso sentences,  <dig> were made to the exact same term as from manual annotations, with only one annotation made to a more general term that is a direct parent term of the manually chosen term in the cellular component ontology.
thus, the precision of annotations made from our textpresso searches was  <dig> % and the resulting f-score for go annotations was  <dig> % .
search failures: analysis of false negative and false positive sentences
to more thoroughly evaluate the results of our searches, and to improve upon the content of our first-draft categories, we undertook an analysis of the false negative and false positive sentences identified by our test searches.
for false negative sentences, we focused on a subset of the sentences, namely those that, when missed, resulted in complete loss of an annotation from a paper.
in theory, failure to make an annotation from textpresso sentences could be due to several reasons including authors' use of non-standard protein nomenclature, distribution of relevant terms over several sentences as opposed to all terms being present in a single sentence, incompleteness of our categories, and sentences that reported subcellular localization results without using terms from all of our search categories.
in practice, careful evaluation of our false negative sentences revealed that all of these factors, often in combination, contributed to failed searches.
we thus classified each false negative sentence according to one or more reasons for which it failed to be returned in our searches .
of the  <dig> false negative sentences that we examined, we found that  <dig> of them  were not returned solely because of authors' use of irregular or non-standard c. elegans nomenclature, with non-standard nomenclature a contributing factor in an additional  <dig> % of false negative sentences that were missed due to use of non-standard nomenclature  plus: missing category terms ; information spread out over several sentences ; information expressed using terms from fewer than the three categories our search required ; and a combination of non-standard nomenclature, missing terms, and information spread out over multiple sentences .
amongst these false negative sentences, there were several different nomenclature variations that deviated from the accepted standard.
in one paper, pak- <dig>  the c. elegans ortholog of mammalian p21-activated kinase, was referred to as cepak, while in another, experiments describing localization of maa- <dig>  a membrane-associated acyl-coa binding protein, referred to localization of wtmaa- <dig> using the 'wt' prefix to denote a wild-type versus mutant protein.
in another example, the agr- <dig> protein was referred to throughout a publication by the name of its mammalian ortholog, agrin.
these results indicate that even though the c. elegans community strives to use standardized nomenclature, reliably identifying gene and protein names in the c. elegans corpus will require cataloging and then incorporating protein synonyms in our searches, as well as exploring additional approaches to entity recognition .
for nine sentences , we missed the annotation because the relevant information was distributed amongst several sentences.
in all of these cases, the protein product was at least one, if not the sole, piece of information missing from the sentence.
distribution of information over several sentences was also a contributing factor in an additional  <dig> % of our false negatives , indicating that reliably caprturing information spread out over multiple sentences will be an important area of future investigation.
in only three cases , did we miss sentences solely due to the incompleteness of our categories.
overall, however, missing category terms were a contributing factor in an additional  <dig> % of false negative sentences .
these results suggest that whereas we can, and will, continue to improve the content of the categories, simply adding more terms and phrases to the categories will not, by itself, dramatically improve the number of annotations we can make using textpresso, as we will still need to address the issues of use non-standard protein nomenclature, information spread out over multiple sentences, and results expressed using terms from fewer than the three categories we required.
in five sentences , we found that although curators could have made an annotation directly from those sentences, they were not returned because the localization information was expressed using words from fewer than three categories.
in most of these cases, the missing term was from the experimental category, as authors simply stated that the protein was localized to a particular cellular component, for example "...daf- <dig> also appeared in the cytoplasm of the hypodermis...".
since our test searches required a match to the protein in question plus at least one term from each of the three categories, these sentences were missed.
lastly, in four cases  we missed an annotation for technical reasons.
in these cases, either the relevant information was presented in table format or the protein name was unidentifiable after the pdf-to-text conversion.
to understand the basis for the  <dig> false positive sentences, we again analyzed the content of each of the sentences.
the results are presented in table  <dig>  for  <dig> of the  <dig> sentences,  <dig> %, we determined that the text returned by the searches was not a single sentence but rather, due to technical issues with the pdf-to-text conversion, a run-on of one or more sentences from the text or, in some cases, the entire content of a table.
in five cases , the sentence was returned due to the inclusion of the term 'transmembrane' in our cellular component category.
in these cases, we found that while 'transmembrane' is used to describe a protein domain it is not consistently used to describe experimentally determined subcellular localization and we have since removed it from the cellular component category.
in the remaining  <dig> cases , the false positive sentences were returned due to linguistic ambiguity associated with certain terms included in our categories.
examples include the term 'signal,' which can be used to describe a fluorescent signal, but also a cell-cell signal involved in intercellular communication, 'processes,' which can refer to cellular projections but also to biological events, such as cell division or aging, and 'soma' which can refer to a cell body apart from cellular projections or to somatic, as opposed to germline, tissue.
in none of these cases, however, did the return of a false positive sentence result in an erroneous annotation.
evaluating curation efficiency: textpresso-based vs. fully manual curation
our primary motivation for developing textpresso categories for cellular component curation was to improve the efficiency with which we make go cellular component annotations.
we therefore investigated the relative efficiencies of textpresso versus manual curation by comparing the amount of time it took curators to curate subcellular localization data from a test corpus using each of the methods.
three curators manually examined one of three sets of twenty randomly selected papers  and recorded the amount of time it took to locate and extract subcellular localization information.
curators then traded paper sets and performed the same task, only this time using sentences returned by textpresso.
the results of this curation efficiency test are presented in figure  <dig>  the time required for each curator to manually evaluate and record subcellular localization information from their test set of  <dig> papers was  <dig>   <dig>  and  <dig> minutes each, respectively.
using textpresso, however, the time required to make annotations was approximately  <dig>   <dig>  and  <dig> minutes, respectively.
thus, textpresso-based cellular component curation has the potential to improve curatorial efficiency by at least a factor of  <dig>  and possibly as much as  <dig>  given differences in individual curatorial speed.
therefore, even though our new textpresso categories are not yet able to recover every annotation from the literature, we believe that textpresso-based curation can still greatly improve the efficiency with which information is extracted from the literature and thus, affords a significant improvement to our go curation pipeline.
CONCLUSIONS
efficient curation of biological facts from the published literature is essential for providing database users with the most complete and up-to-date information possible.
currently, the vast majority of experimental results entered into model organism databases such as wormbase are entered manually by curators who need to identify appropriate papers, read the full text, evaluate the information, and enter annotations using curation tools.
exploring ways to improve efficiency at one or more of these steps may greatly enhance the rate at which new information is culled from the literature.
we have developed a semi-automated curation strategy for annotation of c. elegans proteins to gene ontology cellular component terms.
this approach employs the textpresso information retrieval system which marks up the full-text of research articles using terms and phrases from conceptually related categories.
using categories constructed specifically for cellular component curation to annotate previously uncurated proteins, we found that we were able to retrieve  <dig> % of all possible annotations in our test set with a precision of  <dig> %, yielding an f-score of  <dig> %.
this result indicates that a textpresso-based curation pipeline for go cellular component curation is capable of retrieving a significant proportion of subcellular localization data from the literature with high precision, thus making it feasible to incorporate such an approach into our go curation pipeline.
indeed, having created and tested these categories for go cellular component curation, we have now implemented a curation pipeline whereby each week papers newly entered into our curation database are automatically searched using c. elegans protein names and the three categories.
resulting sentences are displayed in a web-based curation form that allows users to enter go annotations directly into the curation database using the associated publications as evidence .
when an annotation is entered into the database, the cellular component category term found in the returned sentence and the gene ontology term selected for annotation are recorded in a relationship index, so that for all future occurrences of a given cellular component category term, a list of suggested gene ontology terms, based upon previous curation, is available to curators.
thus, cellular component curation at wormbase is moving from manual extraction and annotation to computer-assisted validation.
combined with the improved efficiency that textpresso-based curation affords, this new curation pipeline has allowed us to keep current with cellular component annotations using substantially less curator time.
as this system affords considerable time savings even for a relatively small corpus , it may well provide major time savings for larger literatures.
continued improvements to both textpresso and our curation pipeline may further improve curation efficiency.
a thorough analysis of the false negative and false positive sentences highlighted some general considerations for future development.
the false negative sentences in our test set were missed largely due to use of non-standard protein nomenclature and to distribution of relevant information over several sentences.
to address the issue of protein name recognition, we can catalog and include all available synonyms in our searches, as well as make use of true positive sentences from our training and test sets  to employ machine-learning approaches to entity recognition  <cit> .
likewise, we can use cases where relevant information is distributed over multiple sentences to begin to develop approaches to extracting such information using textpresso.
with regard to protein names and entity recognition, however, the importance of standardized nomenclature and consistent use of this nomenclature in the published literature cannot be overstated.
we will also investigate the usefulness of search returns in which one or more categories are systematically eliminated to determine the extent to which such searches can help identify new gene product names as well as potentially new terms for the three cellular component categories.
a benefit, but also potential drawback, of our approach is that it relied primarily on expert curator judgement for choosing terms and assigning them to appropriate categories.
while this approach ensured that our category terms were highly relevant for experimentally determined localization there is, nonetheless, potential for bias in term selection.
although we note that few go annotations were missed solely due to missing category terms, we recognize that future category development would likely benefit from incorporating additional statistical methods as well as efficient, systematic testing that would allow curators to readily compare, for example, the results of searches using a curated set of verbs versus a set comprised of normalized terms that includes all possible tenses for a given verb.
category construction will undoubtedly be an iterative process and we anticipate that further category development and testing will continually improve our results.
although the number of false positive documents and sentences in our test set was not a hindrance to curation, improvements to further reduce the false positive rate may be possible.
textpresso searches rely upon accurate conversion of pdf documents to text, but long lists of information in tables or inadvertent run-on sentences can lead to false positive returns.
prospects for improving these technical issues include obtaining papers in xml format and improving pdf-to-text conversion.
in addition, in analyzing the false positive sentences, we found that nearly 60% of these sentences came from outside the results section of the papers, i.e.
the introduction, discussion, or materials and methods.
if curators could limit searches to the results section of relevant papers, the false positive rate could be greatly reduced.
we have recently implemented a section-tagging feature in textpresso and find that it is helpful not only for reducing the number of false positives, but also for helping curators confirm that a given sentence returned by textpresso indeed derives from experimental data .
lastly, we will investigate the use of exclusion terms in our searches to help lower the rate of returned sentences that describe localization in a mutant, as opposed to wild-type, background.
beyond c. elegans and cellular components
recently, the gene ontology consortium has undertaken a reference genome project in which the genomes of twelve representative, or reference, organisms are being comprehensively annotated  <cit> .
one of the primary motivations for undertaking the reference genome project was to facilitate annotation of newly sequenced genomes using experimentally supported go annotations from the reference genomes.
tools such as textpresso that improve efficiency of manual go curation may thus benefit not only the reference genome communities, but additional communities that rely on reference or model organism curation for reliable functional annotation of their genomes.
as cellular component curation is just one aspect of comprehensive genome annotation, we plan to extend the use of textpresso-based curation to other data types, including molecular functions such as enzymatic activity and nucleic acid binding, to continue to improve the depth and efficiency with which genomes are annotated.

 one of the main challenges of genome projects is providing reliable functional annotations for gene products. for extensively studied model organisms, such as the nematode c. elegans, high quality functional annotations are largely gleaned from manual curation of experiments reported in the published literature, with additional annotations obtained from computational or comparative methods, such as protein domain analysis. for organisms with smaller research communities, however, functional annotations may initially derive largely from computational or comparative methods which, in turn, can rely heavily upon the accuracy and completeness of model organism genome curation for providing suitable reference annotations and training sets. thus, the extent to which model organism databases can keep pace with annotating an ever expanding literature will potentially have an impact not only on model organism genome curation but curation of a variety of other genomes as well. given that manual curation is unlikely to keep up with current publication rates, developing new approaches to extracting biological facts from the published literature is imperative. introduced over ten years ago, the gene ontology has since become the de facto resource for functional genome annotation using controlled vocabularies. divided into three distinct ontologies that describe biological processes, molecular functions, and cellular components, the go is used by database curators to record key biological features of a gene product in language that is both humanly readable and computationally amenable. a key feature of the go is that its ontologies are structured as directed acyclic graphs in which terms have parent-child relationships, with child terms being more specific or specialized than their respective parent. for example, the go term mitochondrion is a child of intracellular membrane-bounded organelle which is, in turn, a child of intracellular organelle. annotations made to more specialized child terms may thus be transitively made to the more general parent terms, as well: an annotation to mitochondrion is, transitively, an annotation to intracellular organelle. a second key feature of the go is the use of evidence codes to support annotations. evidence codes are used by curators to give an indication of the methodology that researchers use to infer facts about the genes or gene products they are studying. for example, annotation of a gene product to the biological process term cell division based upon a mutant phenotype that results in arrested cell division would use the inferred from mutant phenotype evidence code. likewise, annotation of a gene product to the cellular component term plasma membrane based upon immunofluorescence experiments would use the inferred from direct assay evidence code. selection of the appropriate go evidence code thus requires information about the experiment or assay used in a publication which is often found only by reading the full text. manual go curation is, therefore, a labor-intensive process that if thorough can require reading and annotating the full text of hundreds, if not thousands, of publications. thus, there is a growing need for semi- or fully-automated go curation strategies that will help database curators rapidly and accurately identify key experimental results in the full text of research articles. natural language processing applications offer a promising approach to aiding manual go curation. such applications use various methodologies, including classification of text within documents, to assist in associating gene products to go terms. while some of these applications have met with considerable success in suggesting possible go annotations, only a few take into account a real world database curation pipeline in which go curators examine articles from a wide variety of journals and use the full text of these articles to first select a go term and then evaluate the experimental methodology to confidently select the appropriate go evidence code, an absolute necessity for making a go annotation. thus, there is a need for text mining tools that can accurately mimic the manual curation process and thus be reliably incorporated into a database curation pipeline. here, we describe our strategy for using natural language processing, in particular the textpresso text mining system, to curate experimentally determined subcellular localization of c. elegans proteins using the cellular component ontology of go. textpresso, an open source text mining tool, functions as both a simple search engine and a pattern-based, information extraction engine that employs categories of conceptually related words to semantically mark up the full text of papers. examples of textpresso categories include disease, phenotype, and regulation, which contain words and phrases such as'bardet-biedl syndrome1','cell fate transformation', and'downregulate', respectively. textpresso searches may be performed using keywords and/or categories, and the results are presented as a list of sentences containing terms and phrases that match the search criteria, numerically ranked according to the number of terms or phrases in the sentence that match the search string. at present, there are implementations of textpresso worldwide, including textpresso for neuroscience, budding yeast, drosophila, arabidopsis, and e. coli, with over categories, such as brain area and human disease, fly body parts, and arabidopsis genes, to aid in searching and fact extraction. we focused on go cellular component annotations for our studies for three main reasons that suggested to us it would serve as a potential first proof of principle for developing a textpresso-based, semi-automated curation pipeline for a heretofore fully manual approach. the first reason is that cellular component annotations are generally the result of a limited number of experimental strategies, namely microscopy and subcellular fractionation, thus potentially limiting the number of terms we would need to create new curation task-specific categories. second, because we had noted during extensive manual curation that the information for the conclusion, i.e., subcellular localization, and the type of experimental assay are often stated in the same sentence, our search strategy of finding sentences that include proteins, cellular components and words or phrases referring to the experimental approach, was likely to be successful. third, cellular component annotations derived from the small-scale experiments we were aiming to curate are typically made using one go evidence code, "inferred from direct assay ", thus simplifying the annotation process. in addition, there is a need for cellular component curation tools that make use of the full text of research articles, as authors often fail to include the results of such experiments in the abstracts of their papers. to illustrate, for a random sampling of c. elegans proteins only % of possible go cellular component annotations could be made solely from pubmed abstracts compared to full text, with only % of those annotations as specific as annotations made from full text. to investigate the potential usefulness of textpresso for go cellular component curation, we constructed three new textpresso categories, termed cellular components, assay terms, and verbs, containing terms and phrases found in a gold-standard set of sentences describing experimentally determined subcellular localization. to assess the performance of the new categories, we used them to annotate previously uncurated c. elegans proteins and found that by using textpresso we were able to make % of all possible cellular component annotations with % accuracy. further, by comparing the relative efficiencies of manual versus textpresso-based curation, we find that textpresso-based curation has the potential to improve curation efficiency at least eight-fold, and possibly as much as 15-fold, depending upon the individual curator. by incorporating a textpresso-based cellular component curation pipeline into wormbase, we have moved from fully manual curation to computer-assisted validation for this data type.  development of new textpresso categories our approach to developing textpresso categories for go cellular component curation is outlined in figure and described in detail in the methods. to identify words and phrases relevant to reports of subcellular localization experiments, we collected ~ sentences from papers reporting experimentally determined subcellular localization, and then analyzed lists of words and phrases used in the sentences, as well as the frequency with which the words and phrases occur, to manually select terms that authors use to describe their experimental results. words and phrases identified by our word usage and frequency analysis were then manually sorted into three categories: cellular components, assay terms, and verbs, and included terms such as: nucleus, cell body, centrosomal; expression, antibody, throughout; and detect, exhibited, revealed, respectively. while the division into three categories may seem somewhat arbitrary, it is semantically meaningful. each of the three categories, along with the name of a c. elegans protein, captures the essential aspects of a localization experiment: a protein object, a cellular component, an action verb that reports the result, and an assay term that has a high probability of linking terms from the three other categories to an actual experimental result. requiring a match to at least one term in each of multiple categories affords higher specificity than if all phrases were used to form just one category. in addition, employing multiple categories aids in future category development, as it makes it possible to obtain additional terms and phrases for each category by performing searches requiring matches, for example, to only three of the four categories, thereby identifying sentences that might contain new terms or phrases to potentially be included in the omitted category. testing strategy to evaluate the performance of the three new textpresso categories, we measured the recall and precision of searches employing the three new categories at the document, sentence, and annotation level. in essence: did the search find the correct documents? did the search return the correct sentences? could a curator make the correct annotations from those sentences? in each case, we compared the results of textpresso-based curation to a gold standard of fully manual curation. for each test, we performed a textpresso search for sentences that contained matches to at least one term in each of the categories, plus a match to one of previously uncurated c. elegans proteins chosen at random. as the c. elegans research community strives to adhere to standard nomenclature practices we felt that this was a reasonable first approach to testing the effectiveness of the new categories. we also hoped that by analyzing a diverse set of proteins, we could more accurately assess the performance of the new categories on the existing corpus of c. elegans literature. to evaluate the results of our test searches, we used standard metrics of information retrieval systems: recall, precision, and f-measure. recall is defined as the ratio of correct answers given by the system to the total number of possible correct answers in the text; recall thus reflects the completeness or coverage of the system. precision, on the other hand, is defined as the ratio of the number of correct answers given by a system to the total number of answers given by the system; precision thus reflects the relevance of the search results. in addition, the f-measure, or f-score, is reported as an indication of the accuracy of the test. test results: documents for the proteins in our test set, textpresso searches using the protein name and the three new categories returned primary research articles with the maximum number of papers returned being for the lag- protein and the minimum being zero, for the agr- and pak- proteins. as determined manually, the proteins in our test set were associated with papers containing subcellular localization data in the wormbase bibliography. therefore, to assess recall, we divided the number of correct papers returned by textpresso, by the total number of papers, containing subcellular localization data for all of the genes in our test set as determined manually. at the document level, our recall rate is thus %. to determine the rate of precision, we divided the number of correct papers returned, by the total number of papers returned, to determine that the precision of our searches was %. thus, for document retrieval we achieved an f-score of %. test results: sentences to determine the precision and recall rates of returned sentences, we took the following approach: each sentence in every document of our test set was examined manually in order to select all true positive sentences. for these purposes, we defined true positives as those sentences that describe experimentally determined subcellular localization. the true positive sentence set thus includes all sentences from which a curator could make go annotations, as well as those sentences that, while describing subcellular localization, would not automatically result in a go annotation. for example, sentences describing a protein's subcellular localization in a wild-type background are suitable for go annotation, while sentences describing localization in a mutant background, although often reported with similar, if not identical language, would not typically be annotated for go. reducing or eliminating the latter type of sentences from our search results will be a focus of future tool development. by manual inspection, we identified true positive sentences in the articles we analyzed. of the true positive sentences, only were returned by textpresso, giving a recall rate, at the sentence level, of %. the precision rate, however, was considerably higher, at %. the f-score for sentence returns was thus %. recall and precision: go annotations one of the main motivations for developing our new textpresso categories was to improve the efficiency with which we assign go cellular component annotations. therefore, we also evaluated the recall and precision of the go annotations made from our search returns. for the purposes of this analysis, we scored each annotation just once regardless of the number of times the information supporting that annotation was stated in the paper, as this reflects the normal curation process in which curators do not need to make a separate annotation for repeated statements of the same experimental result. from the results of our textpresso searches we were able to make of possible go annotations made by manual curation. thus, our annotation recall was %. of the annotations we made from textpresso sentences, were made to the exact same term as from manual annotations, with only one annotation made to a more general term that is a direct parent term of the manually chosen term in the cellular component ontology. thus, the precision of annotations made from our textpresso searches was % and the resulting f-score for go annotations was %. search failures: analysis of false negative and false positive sentences to more thoroughly evaluate the results of our searches, and to improve upon the content of our first-draft categories, we undertook an analysis of the false negative and false positive sentences identified by our test searches. for false negative sentences, we focused on a subset of the sentences, namely those that, when missed, resulted in complete loss of an annotation from a paper. in theory, failure to make an annotation from textpresso sentences could be due to several reasons including authors' use of non-standard protein nomenclature, distribution of relevant terms over several sentences as opposed to all terms being present in a single sentence, incompleteness of our categories, and sentences that reported subcellular localization results without using terms from all of our search categories. in practice, careful evaluation of our false negative sentences revealed that all of these factors, often in combination, contributed to failed searches. we thus classified each false negative sentence according to one or more reasons for which it failed to be returned in our searches. of the false negative sentences that we examined, we found that of them were not returned solely because of authors' use of irregular or non-standard c. elegans nomenclature, with non-standard nomenclature a contributing factor in an additional % of false negative sentences that were missed due to use of non-standard nomenclature plus: missing category terms; information spread out over several sentences; information expressed using terms from fewer than the three categories our search required; and a combination of non-standard nomenclature, missing terms, and information spread out over multiple sentences. amongst these false negative sentences, there were several different nomenclature variations that deviated from the accepted standard. in one paper, pak- the c. elegans ortholog of mammalian p21-activated kinase, was referred to as cepak, while in another, experiments describing localization of maa- a membrane-associated acyl-coa binding protein, referred to localization of wtmaa- using the'wt' prefix to denote a wild-type versus mutant protein. in another example, the agr- protein was referred to throughout a publication by the name of its mammalian ortholog, agrin. these results indicate that even though the c. elegans community strives to use standardized nomenclature, reliably identifying gene and protein names in the c. elegans corpus will require cataloging and then incorporating protein synonyms in our searches, as well as exploring additional approaches to entity recognition. for nine sentences, we missed the annotation because the relevant information was distributed amongst several sentences. in all of these cases, the protein product was at least one, if not the sole, piece of information missing from the sentence. distribution of information over several sentences was also a contributing factor in an additional % of our false negatives, indicating that reliably caprturing information spread out over multiple sentences will be an important area of future investigation. in only three cases, did we miss sentences solely due to the incompleteness of our categories. overall, however, missing category terms were a contributing factor in an additional % of false negative sentences. these results suggest that whereas we can, and will, continue to improve the content of the categories, simply adding more terms and phrases to the categories will not, by itself, dramatically improve the number of annotations we can make using textpresso, as we will still need to address the issues of use non-standard protein nomenclature, information spread out over multiple sentences, and results expressed using terms from fewer than the three categories we required. in five sentences, we found that although curators could have made an annotation directly from those sentences, they were not returned because the localization information was expressed using words from fewer than three categories. in most of these cases, the missing term was from the experimental category, as authors simply stated that the protein was localized to a particular cellular component, for example "...daf- also appeared in the cytoplasm of the hypodermis...". since our test searches required a match to the protein in question plus at least one term from each of the three categories, these sentences were missed. lastly, in four cases we missed an annotation for technical reasons. in these cases, either the relevant information was presented in table format or the protein name was unidentifiable after the pdf-to-text conversion. to understand the basis for the false positive sentences, we again analyzed the content of each of the sentences. the results are presented in table for of the sentences, %, we determined that the text returned by the searches was not a single sentence but rather, due to technical issues with the pdf-to-text conversion, a run-on of one or more sentences from the text or, in some cases, the entire content of a table. in five cases, the sentence was returned due to the inclusion of the term'transmembrane' in our cellular component category. in these cases, we found that while'transmembrane' is used to describe a protein domain it is not consistently used to describe experimentally determined subcellular localization and we have since removed it from the cellular component category. in the remaining cases, the false positive sentences were returned due to linguistic ambiguity associated with certain terms included in our categories. examples include the term'signal,' which can be used to describe a fluorescent signal, but also a cell-cell signal involved in intercellular communication,'processes,' which can refer to cellular projections but also to biological events, such as cell division or aging, and'soma' which can refer to a cell body apart from cellular projections or to somatic, as opposed to germline, tissue. in none of these cases, however, did the return of a false positive sentence result in an erroneous annotation. evaluating curation efficiency: textpresso-based vs. fully manual curation our primary motivation for developing textpresso categories for cellular component curation was to improve the efficiency with which we make go cellular component annotations. we therefore investigated the relative efficiencies of textpresso versus manual curation by comparing the amount of time it took curators to curate subcellular localization data from a test corpus using each of the methods. three curators manually examined one of three sets of twenty randomly selected papers and recorded the amount of time it took to locate and extract subcellular localization information. curators then traded paper sets and performed the same task, only this time using sentences returned by textpresso. the results of this curation efficiency test are presented in figure the time required for each curator to manually evaluate and record subcellular localization information from their test set of papers was and minutes each, respectively. using textpresso, however, the time required to make annotations was approximately and minutes, respectively. thus, textpresso-based cellular component curation has the potential to improve curatorial efficiency by at least a factor of and possibly as much as given differences in individual curatorial speed. therefore, even though our new textpresso categories are not yet able to recover every annotation from the literature, we believe that textpresso-based curation can still greatly improve the efficiency with which information is extracted from the literature and thus, affords a significant improvement to our go curation pipeline.  efficient curation of biological facts from the published literature is essential for providing database users with the most complete and up-to-date information possible. currently, the vast majority of experimental results entered into model organism databases such as wormbase are entered manually by curators who need to identify appropriate papers, read the full text, evaluate the information, and enter annotations using curation tools. exploring ways to improve efficiency at one or more of these steps may greatly enhance the rate at which new information is culled from the literature. we have developed a semi-automated curation strategy for annotation of c. elegans proteins to gene ontology cellular component terms. this approach employs the textpresso information retrieval system which marks up the full-text of research articles using terms and phrases from conceptually related categories. using categories constructed specifically for cellular component curation to annotate previously uncurated proteins, we found that we were able to retrieve % of all possible annotations in our test set with a precision of %, yielding an f-score of %. this result indicates that a textpresso-based curation pipeline for go cellular component curation is capable of retrieving a significant proportion of subcellular localization data from the literature with high precision, thus making it feasible to incorporate such an approach into our go curation pipeline. indeed, having created and tested these categories for go cellular component curation, we have now implemented a curation pipeline whereby each week papers newly entered into our curation database are automatically searched using c. elegans protein names and the three categories. resulting sentences are displayed in a web-based curation form that allows users to enter go annotations directly into the curation database using the associated publications as evidence. when an annotation is entered into the database, the cellular component category term found in the returned sentence and the gene ontology term selected for annotation are recorded in a relationship index, so that for all future occurrences of a given cellular component category term, a list of suggested gene ontology terms, based upon previous curation, is available to curators. thus, cellular component curation at wormbase is moving from manual extraction and annotation to computer-assisted validation. combined with the improved efficiency that textpresso-based curation affords, this new curation pipeline has allowed us to keep current with cellular component annotations using substantially less curator time. as this system affords considerable time savings even for a relatively small corpus, it may well provide major time savings for larger literatures. continued improvements to both textpresso and our curation pipeline may further improve curation efficiency. a thorough analysis of the false negative and false positive sentences highlighted some general considerations for future development. the false negative sentences in our test set were missed largely due to use of non-standard protein nomenclature and to distribution of relevant information over several sentences. to address the issue of protein name recognition, we can catalog and include all available synonyms in our searches, as well as make use of true positive sentences from our training and test sets to employ machine-learning approaches to entity recognition. likewise, we can use cases where relevant information is distributed over multiple sentences to begin to develop approaches to extracting such information using textpresso. with regard to protein names and entity recognition, however, the importance of standardized nomenclature and consistent use of this nomenclature in the published literature cannot be overstated. we will also investigate the usefulness of search returns in which one or more categories are systematically eliminated to determine the extent to which such searches can help identify new gene product names as well as potentially new terms for the three cellular component categories. a benefit, but also potential drawback, of our approach is that it relied primarily on expert curator judgement for choosing terms and assigning them to appropriate categories. while this approach ensured that our category terms were highly relevant for experimentally determined localization there is, nonetheless, potential for bias in term selection. although we note that few go annotations were missed solely due to missing category terms, we recognize that future category development would likely benefit from incorporating additional statistical methods as well as efficient, systematic testing that would allow curators to readily compare, for example, the results of searches using a curated set of verbs versus a set comprised of normalized terms that includes all possible tenses for a given verb. category construction will undoubtedly be an iterative process and we anticipate that further category development and testing will continually improve our results. although the number of false positive documents and sentences in our test set was not a hindrance to curation, improvements to further reduce the false positive rate may be possible. textpresso searches rely upon accurate conversion of pdf documents to text, but long lists of information in tables or inadvertent run-on sentences can lead to false positive returns. prospects for improving these technical issues include obtaining papers in xml format and improving pdf-to-text conversion. in addition, in analyzing the false positive sentences, we found that nearly 60% of these sentences came from outside the results section of the papers, i.e. the introduction, discussion, or materials and methods. if curators could limit searches to the results section of relevant papers, the false positive rate could be greatly reduced. we have recently implemented a section-tagging feature in textpresso and find that it is helpful not only for reducing the number of false positives, but also for helping curators confirm that a given sentence returned by textpresso indeed derives from experimental data. lastly, we will investigate the use of exclusion terms in our searches to help lower the rate of returned sentences that describe localization in a mutant, as opposed to wild-type, background. beyond c. elegans and cellular components recently, the gene ontology consortium has undertaken a reference genome project in which the genomes of twelve representative, or reference, organisms are being comprehensively annotated. one of the primary motivations for undertaking the reference genome project was to facilitate annotation of newly sequenced genomes using experimentally supported go annotations from the reference genomes. tools such as textpresso that improve efficiency of manual go curation may thus benefit not only the reference genome communities, but additional communities that rely on reference or model organism curation for reliable functional annotation of their genomes. as cellular component curation is just one aspect of comprehensive genome annotation, we plan to extend the use of textpresso-based curation to other data types, including molecular functions such as enzymatic activity and nucleic acid binding, to continue to improve the depth and efficiency with which genomes are annotated. 
we compared the results of manual curation to that of textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated c. elegans protein, and found that textpresso searches identified curatable papers with recall and precision rates of % and %, respectively, when compared to manual curation. from returned sentences, curators were able to make % of all possible experimentally supported go cellular component annotations with % precision. using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories containing words and phrases associated with reports of experimentally determined subcellular localization. incorporating a textpresso-based cellular component curation pipeline at wormbase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. measuring the relative efficiencies of textpresso-based versus manual curation we find that textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed. manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts. continued development of curation task-specific textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation. we employ the textpresso category-based information retrieval and extraction system, developed by wormbase to explore how textpresso might improve the efficiency with which we manually curate c. elegans proteins to the gene ontology's cellular component ontology.
 manual curation of experimental data from the biomedical literature is an expensive and time-consuming endeavor. nevertheless, most biological knowledge bases still rely heavily on manual curation for data extraction and entry. text mining software that can semi- or fully automate information retrieval from the literature would thus provide a significant boost to manual curation efforts.  we employ the textpresso category-based information retrieval and extraction system, developed by wormbase to explore how textpresso might improve the efficiency with which we manually curate c. elegans proteins to the gene ontology's cellular component ontology. using a training set of sentences that describe results of localization experiments in the published literature, we generated three new curation task-specific categories containing words and phrases associated with reports of experimentally determined subcellular localization. we compared the results of manual curation to that of textpresso queries that searched the full text of articles for sentences containing terms from each of the three new categories plus the name of a previously uncurated c. elegans protein, and found that textpresso searches identified curatable papers with recall and precision rates of % and %, respectively, when compared to manual curation. within those documents, textpresso identified relevant sentences with recall and precision rates of % and %. from returned sentences, curators were able to make % of all possible experimentally supported go cellular component annotations with % precision. measuring the relative efficiencies of textpresso-based versus manual curation we find that textpresso has the potential to increase curation efficiency by at least 8-fold, and perhaps as much as 15-fold, given differences in individual curatorial speed.  textpresso is an effective tool for improving the efficiency of manual, experimentally based curation. incorporating a textpresso-based cellular component curation pipeline at wormbase has allowed us to transition from strictly manual curation of this data type to a more efficient pipeline of computer-assisted validation. continued development of curation task-specific textpresso categories will provide an invaluable resource for genomics databases that rely heavily on manual curation. 