BACKGROUND
in  <dig>  pubmed surpassed  <dig> million total articles in its index and is growing at a rate approaching  <dig> million new articles per year.
tools that make effective use of this knowledge base are increasingly vital to biomedical research  <cit> .
search interfaces like pubmed and google scholar help to find individual documents, yet no single document contains all of the knowledge about a particular biological concept.
the knowledge is distributed throughout the text of many different articles with each new contribution simply adding to the stack.
in considering the task of comprehensively capturing and utilizing society's biological knowledge, it is clear that document-centered approaches are insufficient.
recognizing this problem and its clear pertinence to genome-scale biology, the research community began defining ontologies to capture and structure functional genomic knowledge even before the first human genome was fully sequenced  <cit> .
ontologies, such as the gene ontology , provide a mechanism to efficiently bring together individual atomic facts  that may be scattered across many different texts.
such integration has enabled the production of new tools for interacting with and computing with massive bodies of knowledge.
in particular, ontology-based computational analysis now plays a crucial role in the interpretation of the results of high-throughput, genomic studies  <cit> .
while structuring knowledge using ontologies has proven highly beneficial, it presents some substantial challenges.
the task of manually representing knowledge with an ontology is difficult, time-consuming and generally not rewarded by the scientific community.
current paradigms drive scientists to publish their findings as text in traditional journals that must subsequently be sifted through by separate teams of database curators who identify and extract new ontology-based facts.
this process results in a significant bottleneck that is costly both in terms of the resources required and the likelihood that the system is not capturing all of the knowledge that is produced  <cit> .
in addition, there remains a secondary challenge of presenting this knowledge to biologists of all levels in a manner that they can rapidly understand.
faced with these challenges in manual biocuration, database curators have begun to investigate wikis as a potential solution  <cit> .
wikis represent a third approach to knowledge management that straddles the line between document-centric traditional publishing and ontology-driven knowledge base development.
in contrast to typical literature collections, wikis, like ontologies, are concept-centric.
when new facts are added, they are placed directly into the context of existing articles.
for each gene, for example, a single wiki article can summarize knowledge spread over a large and growing corpus of traditional publications related to that gene.
this concept-centricity renders wikis an excellent medium to capture rapidly evolving biological knowledge.
the other distinguishing attribute of wikis is their potential to make use of community intelligence on a massive scale.
wikipedia is well known for harnessing the intellects of literally millions of people to assemble the world's largest encyclopedia.
as a result of both their concept-centric structure and their enticing potential to facilitate mass collaboration, wikis have emerged in a variety of areas of biology.
we have wikis that capture information about genes  <cit> , proteins  <cit> , protein structures  <cit> , snps  <cit> , pathways  <cit> , specific organisms  <cit>  and many other biological entities.
in some cases, wikis are already successfully tapping into the community's collective intellect to produce useful biological knowledge repositories.
one prominent example is the gene wiki  <cit> .
the gene wiki is a growing collection of wikipedia articles, each of which is focused specifically on a human gene.
as of january  <dig>   <dig>  it contained articles on  <dig>  human genes.
these articles collectively amount to over  <dig> megabytes of text and more than  <dig>  million words.
in addition, they contain direct citations to more than  <dig>  distinct articles in pubmed.
to emphasize the collaborative scale of the gene wiki project, in  <dig>  these articles were edited by more than  <dig>  distinct editors and were viewed more than  <dig> million times.
the gene wiki successfully harnesses community intelligence and escapes the "infinite pile" of document-centric approaches by maintaining a single, dynamic article for each gene.
however, most of the captured knowledge is unstructured text and therefore it does not provide the structured gene annotations needed to effectively compute with the knowledge it contains.
wikipedia and the gene wiki are simply not designed to capture ontology-based facts.
hence, while the wiki-model can successfully summarize the collective knowledge of the community, the challenge of fully structuring the information remains.
computational tools for finding ontology terms in text, such as the national center for biomedical ontology's  annotator  <cit>  and the national library of medicine's metamap  <cit> , can help to address the challenge of structuring information presented in natural language.
in this article we describe an approach for mining ontology-based annotations for human genes from the text of the gene wiki.
specifically we used the ncbo annotator to identify structured gene annotations based on the disease ontology   <cit>  and the gene ontology   <cit> .
we evaluated the predicted annotations through comparison to known annotations and manual expert review.
in addition, we assessed the impact of these predicted annotations on gene set enrichment analyses.
RESULTS
system design
as illustrated in figure  <dig>  we applied the ncbo annotator to identify occurrences of do and go terms in gene wiki articles.
the text of each gene wiki article was first filtered to eliminate references and auto-generated text, and then sent to the annotator for concept detection.
since each article is specifically about a particular human gene, we made the assumption that occurrences of concepts within the text of a gene-centric article were also about the gene and considered such occurrences candidate annotations for the gene.
for example, we identified the go term 'embryonic development ' in the text of the article on the dax <dig> gene: "dax <dig> controls the activity of certain genes in the cells that form these tissues during embryonic development".
from this occurrence, our system proposed the structured annotation 'dax <dig> participates in the biological process of embryonic development'.
following the same pattern, we found a potential annotation to the do term 'congenital adrenal hypoplasia'  in the sentence: "mutations in this gene result in both x-linked congenital adrenal hypoplasia and hypogonadotropic hypogonadism".
overall, this workflow resulted in  <dig>  candidate do annotations  and  <dig>  candidate go annotations  from the collaboratively authored text of  <dig>  gene wiki articles.
we next characterized these candidate annotations through comprehensive comparison to pre-existing reference gene annotation databases and through manual inspection by experts in go and do annotation.
comparisons to reference annotations
each candidate annotation was compared to reference annotations for the relevant gene.
matches to the reference could either be exact matches, matches to a more specific term in the same lineage as a reference annotation  or matches to a more general term.
for example, our system suggested that the thrombin gene be annotated with 'hemostasis' ; since thrombin was already annotated with 'blood coagulation'  which is a narrower term than 'hemostasis' and in the same lineage, the predicted annotation was classified as a match to a more general term.
disease ontology
we compared the candidate do annotations to a pre-existing gene-disease annotation database mined from ncbi's generifs  <cit> .
while online mendelian inheritance in man  is probably the most widely recognized gene-disease database, we chose the generif-backed database for comparison because a) it used the do for annotations thus enabling direct comparison , b) at the time it was created, it contained the majority of the gene-disease associations in omim and significantly extended this set, and c) it reported a very high precision rate  - comparable to a manually curated resource  <cit> .
the downside of using this database for comparison was that it had not been updated since  <dig> and hence it was undoubtedly missing more recent information.
in all,  <dig>  of the  <dig>  candidate annotations exactly matched an annotation from the do reference annotations,  <dig>  matched a more general term in the same lineage as a reference annotation,  <dig>  matched a more specific term, and  <dig>  had no match .
we refer to the annotations with no match as 'novel candidates' as these represent potential new annotations.
gene ontology
we next compared the candidate annotations to reference annotations from the go annotation database   <cit> .
the goa database is the accepted standard public reference for go-based annotation of human gene products.
when this analysis was conducted, it provided annotations for  <dig>  distinct human genes.
of the  <dig>  mined go annotations,  <dig>   matched an annotation in the goa database exactly,  <dig>  matched more specific terms than goa annotations,  <dig>   matched more general terms, and  <dig>   did not match any.
the go is divided into three distinct topical branches: biological process , molecular function  and cellular component .
in all, 54% of the  <dig>  candidate annotations used bp terms, 14% used mf, and the remaining 32% used cc.
in the remainder of this article, we focus on the  <dig>  candidate bp annotations because they were the most plentiful in the output and because they are generally the most difficult annotations to determine automatically using other methods  <cit> .
of the  <dig>  candidate bp annotations,  <dig>  were direct matches to gene-function annotations in the goa reference,  <dig>  matched narrower terms than existing annotations,  <dig>   matched more general terms than existing annotations, and the remaining  <dig>   did not match any annotations in the goa database .
manual evaluations
for candidate annotations with no matches in the reference databases, we extracted a random sample and submitted it to expert curators to manually evaluate the quality of the predictions.
for the do evaluations, the sample contained  <dig> candidate annotations or approximately 10% of the  <dig>  candidate annotations that either had no match in the reference set or matched a child of a reference annotation.
for the go we originally selected  <dig> novel candidate annotations for the evaluation but later had to remove  <dig> from consideration after discovering that, due to an error in processing, these were actually parents of reference annotations.
the sample sizes were the largest that could be processed in a reasonable amount of time based on the curator resources at our disposal.
for the evaluation, the curators assigned each candidate annotation in the sample to one of eight categories as follows.
category 1: yes, this would lead to a new annotation
1a: perfect match - the candidate annotation is exactly as it would be from a curator
1b: not specific enough - the candidate annotation is correct but a more specific term should be used instead
1c: too specific - the candidate annotation is close to correct, but is too specific given the evidence at hand
category 2: maybe, but insufficient evidence:
2a: evaluator could not find enough supporting evidence in the literature after about  <dig> minutes of looking
2b: there is disagreement in the literature about the truth of this annotation
category 3: no, this candidate annotation is incorrect:
3a: incorrect concept recognition  which is defined as the transfer of genetic information to a bacterium from a bacteriophage or between bacterial or yeast cells mediated by a phage vector - a completely different concept from signal transduction as intended in the sentence.)
3b: incorrect sentence context - the sentence is a negation or otherwise does not support the predicted annotation for the given gene
3c: this sentence seems factually false
disease ontology
to assess the quality of the candidate do annotations with no match in the reference set discussed above, we reviewed the  <dig> randomly selected novel candidate annotations manually according to the criteria outlined above .
the reviewers  were experts in do-based gene annotation and are active participants in the development of the do.
out of the  <dig> candidates evaluated,  <dig>  were classified in category  <dig>  with  <dig>  assigned to category 1a .
figure  <dig> provides a breakdown of the results of the manual evaluation.
nearly all of the errors fall into category 3b .
for example, the ncbo annotator correctly identified the disease term 'neuroblastoma' in the following sentence: "dock3-mediated rac <dig> activation promotes reorganisation of the cytoskeleton in sh-sy5y neuroblastoma cells and primary cortical neurones as well as morphological changes in fibroblasts"; however, the assumption that the occurrence indicates an association with the gene dock <dig> does not hold because the sentence is referring to a neuroblastoma cell line rather than to the human disease.
gene ontology
we then followed the same protocol to assess the quality of candidate biological process annotations with no match in the goa reference set .
a professional curator familiar with go annotation  manually inspected a random sample of  <dig> candidate annotations with no match in the reference set and classified each with the same  <dig> categories used for the do evaluations.
the performance was substantially worse for the novel go annotations than for the do  - in particular only  <dig>  of the  <dig> candidates that were evaluated were assigned to category 1a .
aside from the low number of 'exact match' results, candidate go annotations generated many more uncertain results  as well as a very large fraction of errors due to incorrect sentence context .
it is worth noting that not one of errors detected in either the do or the go annotations were classified as 3c .
this provides evidence that the text of the gene wiki  is consistently correct.
overall precision of annotation mining system
we integrated the results from both the manual assessments and the comparison to reference data sets to provide an estimate of the overall likelihood that a predicted annotation is biologically valid.
we refer to this likelihood as 'precision' where precision  is equal to the ratio of true positive predictions to the sum of true and false positive predictions.
in order to calculate precision, we need to divide all predictions into two classes 'true' and 'false'.
for this analysis, the true positive set contained the direct matches to reference annotations, the matches to parents of reference annotations , and the estimated number of category  <dig>  novel annotations.
the estimated counts for the novel annotations were derived by multiplying the precision rates observed in the manually evaluated sample by the total number of novel candidate annotations.
to account for the novel predicted annotations that were classified into category 2a , we provide two estimates of precision: one that includes the 2a results as true positives and one that includes them as false positives.
in this way we produce an estimated lower and upper bound on the system's actual precision.
the estimated upper bound for the overall precision of the annotation protocol was thus calculated as:
exact+moregeneral+e1a+e1b+e1c+e2aeall*all
and the estimated lower bound as:
exact+moregeneral+e1a+e1b+e1ceall*all
where exact, more general, child, and none correspond to agreements between candidate annotations and a reference set; e1a, e1b, e1c and e2a refer to evaluation categories 1a, 1b, 1c and 2a, and eall refers to the total number of novel candidates evaluated.
using equations  <dig> and  <dig>  we estimated a range for precision of 90-93% for the do annotations and 48-64% for the go annotations.
in retrospect, it may be more appropriate to remove the category 1c  annotations from the 'biologically valid' grouping here, but, since there were no occurrences of this category in either the do or the go evaluations, this change would not impact the results of the present analysis.
potential applications in enrichment analysis
given these estimates of precision, we next checked to see if the annotations produced here could be used immediately in applications relevant to biological discovery.
specifically we assessed the use of the new annotations in the context of gene-set enrichment analyses.
gene-set enrichment analyses provide a knowledge-based statistical assessment of the important concepts related to a set of genes  <cit> .
since tools for performing enrichment analysis are noise tolerant , but cannot function without annotations, the use of automatically derived annotations as an extension to curated annotations can provide increased power and flexibility in terms of which concepts can be detected.
for example, if a sufficient body of relevant text can be identified for each gene in a study set, enrichment analysis can now be conducted using any of the ontologies present in the ncbo bioportal using the ncbo annotator  <cit> .
the crux of this kind of analysis is the identification of a sufficient quantity of relevant text.
since each gene wiki article is exclusively about one gene, as opposed to a typical article indexed in pubmed that may mention many genes and processes in very specific contexts, the articles form a particularly useful corpus.
while the gene wiki alone does not yet have enough content to warrant the use of annotations derived solely from its text in knowledge-based analyses, we hypothesized that it could be used as an extension to other sources of gene-centric text to improve the results of text-driven enrichment analyses.
evaluation of mined go annotations in gene-set enrichment analysis
to assess the potential value of the gene-wiki derived go annotations, we measured their impact on a controlled gene set enrichment experiment based on the pattern introduced by lependu et al.
<cit> .
as an example, consider the go term for 'muscle contraction' .
this go annotation was associated with  <dig> genes in the goa database.
after blinding ourselves to the origin of this  <dig> gene list, we performed text mining on the titles and abstracts of the publications associated with those  <dig> genes.
we expected to find the term 'muscle contraction' to be enriched relative to the background occurrence in all publications.
however, when using article titles and abstracts alone, we found no such statistical enrichment .
in contrast, after adding gene wiki text to the corpus of publication titles and abstracts, the term 'muscle contraction' was highly enriched .
to confirm that this result was not an artifact, we performed a simulation in which, for each of  <dig>  iterations, we selected  <dig> genes randomly from the set of genes with any go annotations from the goa database and performed the identical analysis.
this provided a way to estimate the probability that we would observe this improvement from the addition of the gene wiki text for random gene sets.
the higher this probability, the lower our confidence that the additional annotations provided by the gene wiki provided a real signal of value.
in only  <dig> of the  <dig>  simulation runs did the gene wiki-enhanced annotation set produce a significant p value  when the pubmed-only set did not.
in fact, for the random gene sets the gene wiki derived annotations were slightly more likely to make the p values worse  though in most cases there was no impact at all.
this simulation demonstrated that, as should be expected from results presented above, the annotations mined from the gene wiki are both non-random and are clearly correlated with annotations shared in curated databases.
in addition, it provided empirical evidence that fisher's exact test is appropriate in this situation .
we extended this analysis to all  <dig> go terms used in human gene annotations and found a consistent improvement in the enrichment scores .
using abstracts and titles alone, this protocol resulted in a significant enrichment score for  <dig> gene sets .
enhancing the data from pubmed mining with the gene wiki text resulted in an increase to  <dig> significant tests .
the relatively low rate of annotation rediscovery for go terms based on the text from abstracts related to associated genes is not surprising.
other groups have reported that only about 10% of the curated go annotations can be found in the text of the abstract of the paper cited as evidence for the annotation  <cit> .
what we show here simply demonstrates that the text from the gene wiki can extend the reach of systems that rely solely on the text of pubmed abstracts for annotation mining.
while the results are preliminary, a similar text-driven enrichment analysis that used the do rather than the go also showed improvement when annotations mined from the gene wiki were included alongside annotations mined from pubmed abstracts .
discussion
our results demonstrate that the gene wiki is an important repository of knowledge about the human genome that is different from and complementary to other biological knowledge sources.
its articles provide a growing source of gene-specific text that pulls together relevant bits of information from the published literature into a form that is both useful for human consumption and highly amenable to natural language processing.
importantly, the mass collaborative approach to assembling these wiki articles scales with the explosive growth of the biomedical literature.
annotations mined from gene wiki text both recapitulated and extended knowledge in existing databases.
disease ontology versus gene ontology for mining gene annotations
we speculate that the differences in the level of precision estimated for the predicted annotations using the go  and the do  were primarily the result of two key factors: the differing scopes of the two ontologies and differences in the way that the two communities view annotations.
the extremely broad scope of the go means that it has many fairly general terms, like 'transduction', that can result in errors due to polysemy.
while fairly general terms do exist in the do, such as 'dependence' , they are far less frequent.
aside from increased numbers of mismatched concepts from the go, the criteria used by curators for establishing that a go annotation is fit for inclusion in the goa database are more stringent than the criteria for establishing a gene-disease association.
this is evident when comparing the category 2a  results for the two ontologies.
as shown in figures  <dig> and  <dig>  the reviewers felt that only 4% of the evaluated do-based candidates needed additional investigation while 26% of the go-based candidates were judged to require more evidence.
in many cases, the 2a scores for the go evaluations resulted when the main evidence for the new annotation was from research conducted in a model organism.
in order for a go curator to accept evidence from another organism, further analysis of sequence and phylogenies must be conducted and such analysis was beyond the scope of this evaluation.
in the case of the do annotations, curators accept evidence from model organisms as sufficient for forming an association.
potential modifications
it is worth noting that the basic protocol described here is not tied to the annotator, it could be used with any concept detection system.
the annotator was used in this analysis because it has a fast convenient api, access to a large number of ontologies including the do, and has been shown to have similar performance to metamap - a longstanding, commonly used tool to for biomedical concept recognition  <cit> .
improvements to the annotator workflow, such as negation detection, are ongoing and will benefit the protocol described here.
for critical assessments of other text-mining tools and applications in biology, see the biocreative competitions  <cit> .
in particular see  <cit>  for a discussion of challenges in working with the go.
in our evaluation of the results, we only had access to a single qualified go curator and two do curators, hence we had to constrain the size of the sample processed and could not calculate inter-annotator agreement .
while we suggest that the number of manually reviewed candidate annotations was sufficient to provide rough estimates on the precision of this protocol, additional evaluations would certainly be valuable.
the scarcity of qualified curators, and the even more apparent scarcity of their time, provides additional motivation for continuing this area of research.
applications
we expect that predicted annotations from the gene wiki will have several applications.
first, professional biocurators could use gene wiki-derived annotations as a useful starting point for their curation efforts.
most obviously, these candidate annotations could be processed according to current curatorial standards  to approve, refine, or reject them as formal annotations  <cit> .
on an even more basic level, curators could simply prioritize pubmed articles that were used in inline gene wiki citations for formal review.
in this scenario, the gene wiki would be used as a crowdsourced method to identify the most relevant scientific literature  <cit> , an increasingly difficult problem based on the rapid growth of pubmed.
second, these candidate annotations could be used directly by end users in statistical analyses that are tolerant to noisy data.
for example, gene set enrichment analysis is among the most popular analysis strategies for genomic studies, and the underlying statistical test is, by definition, noise tolerant.
a recent application called the rich annotation summarizer  performs gene set enrichment analysis using any of the ontologies in the ncbo bioportal by applying the annotator to extract relevant annotations from medline abstracts and the ncbo resource index  <cit> .
annotations derived from the gene wiki could fit directly into these and related systems.
CONCLUSIONS
ontology-based gene annotation forms a crucial component of many tasks in bioinformatics, but accumulating these annotations is costly.
by combining the mass collaboratively generated text of gene-specific articles in the gene wiki with readily accessible natural language processing technology, we introduced a new and scalable system for generating gene annotations.
as with any application of currently available natural language processing, this system is not error free.
go annotations in particular proved difficult to produce at high precision.
looking forward, we can expect that improvements in information extraction technology and the continued expansion of the gene-centric text in the gene wiki will combine to produce an increasingly valuable process for harnessing the ever-expanding body of functional knowledge about the human genome.

 in pubmed surpassed million total articles in its index and is growing at a rate approaching million new articles per year. tools that make effective use of this knowledge base are increasingly vital to biomedical research. search interfaces like pubmed and google scholar help to find individual documents, yet no single document contains all of the knowledge about a particular biological concept. the knowledge is distributed throughout the text of many different articles with each new contribution simply adding to the stack. in considering the task of comprehensively capturing and utilizing society's biological knowledge, it is clear that document-centered approaches are insufficient. recognizing this problem and its clear pertinence to genome-scale biology, the research community began defining ontologies to capture and structure functional genomic knowledge even before the first human genome was fully sequenced. ontologies, such as the gene ontology, provide a mechanism to efficiently bring together individual atomic facts that may be scattered across many different texts. such integration has enabled the production of new tools for interacting with and computing with massive bodies of knowledge. in particular, ontology-based computational analysis now plays a crucial role in the interpretation of the results of high-throughput, genomic studies. while structuring knowledge using ontologies has proven highly beneficial, it presents some substantial challenges. the task of manually representing knowledge with an ontology is difficult, time-consuming and generally not rewarded by the scientific community. current paradigms drive scientists to publish their findings as text in traditional journals that must subsequently be sifted through by separate teams of database curators who identify and extract new ontology-based facts. this process results in a significant bottleneck that is costly both in terms of the resources required and the likelihood that the system is not capturing all of the knowledge that is produced. in addition, there remains a secondary challenge of presenting this knowledge to biologists of all levels in a manner that they can rapidly understand. faced with these challenges in manual biocuration, database curators have begun to investigate wikis as a potential solution. wikis represent a third approach to knowledge management that straddles the line between document-centric traditional publishing and ontology-driven knowledge base development. in contrast to typical literature collections, wikis, like ontologies, are concept-centric. when new facts are added, they are placed directly into the context of existing articles. for each gene, for example, a single wiki article can summarize knowledge spread over a large and growing corpus of traditional publications related to that gene. this concept-centricity renders wikis an excellent medium to capture rapidly evolving biological knowledge. the other distinguishing attribute of wikis is their potential to make use of community intelligence on a massive scale. wikipedia is well known for harnessing the intellects of literally millions of people to assemble the world's largest encyclopedia. as a result of both their concept-centric structure and their enticing potential to facilitate mass collaboration, wikis have emerged in a variety of areas of biology. we have wikis that capture information about genes, proteins, protein structures, snps, pathways, specific organisms and many other biological entities. in some cases, wikis are already successfully tapping into the community's collective intellect to produce useful biological knowledge repositories. one prominent example is the gene wiki. the gene wiki is a growing collection of wikipedia articles, each of which is focused specifically on a human gene. as of january it contained articles on human genes. these articles collectively amount to over megabytes of text and more than million words. in addition, they contain direct citations to more than distinct articles in pubmed. to emphasize the collaborative scale of the gene wiki project, in these articles were edited by more than distinct editors and were viewed more than million times. the gene wiki successfully harnesses community intelligence and escapes the "infinite pile" of document-centric approaches by maintaining a single, dynamic article for each gene. however, most of the captured knowledge is unstructured text and therefore it does not provide the structured gene annotations needed to effectively compute with the knowledge it contains. wikipedia and the gene wiki are simply not designed to capture ontology-based facts. hence, while the wiki-model can successfully summarize the collective knowledge of the community, the challenge of fully structuring the information remains. computational tools for finding ontology terms in text, such as the national center for biomedical ontology's annotator and the national library of medicine's metamap, can help to address the challenge of structuring information presented in natural language. in this article we describe an approach for mining ontology-based annotations for human genes from the text of the gene wiki. specifically we used the ncbo annotator to identify structured gene annotations based on the disease ontology and the gene ontology. we evaluated the predicted annotations through comparison to known annotations and manual expert review. in addition, we assessed the impact of these predicted annotations on gene set enrichment analyses.  system design as illustrated in figure we applied the ncbo annotator to identify occurrences of do and go terms in gene wiki articles. the text of each gene wiki article was first filtered to eliminate references and auto-generated text, and then sent to the annotator for concept detection. since each article is specifically about a particular human gene, we made the assumption that occurrences of concepts within the text of a gene-centric article were also about the gene and considered such occurrences candidate annotations for the gene. for example, we identified the go term'embryonic development' in the text of the article on the dax gene: "dax controls the activity of certain genes in the cells that form these tissues during embryonic development". from this occurrence, our system proposed the structured annotation'dax participates in the biological process of embryonic development'. following the same pattern, we found a potential annotation to the do term'congenital adrenal hypoplasia' in the sentence: "mutations in this gene result in both x-linked congenital adrenal hypoplasia and hypogonadotropic hypogonadism". overall, this workflow resulted in candidate do annotations and candidate go annotations from the collaboratively authored text of gene wiki articles. we next characterized these candidate annotations through comprehensive comparison to pre-existing reference gene annotation databases and through manual inspection by experts in go and do annotation. comparisons to reference annotations each candidate annotation was compared to reference annotations for the relevant gene. matches to the reference could either be exact matches, matches to a more specific term in the same lineage as a reference annotation or matches to a more general term. for example, our system suggested that the thrombin gene be annotated with'hemostasis'; since thrombin was already annotated with'blood coagulation' which is a narrower term than'hemostasis' and in the same lineage, the predicted annotation was classified as a match to a more general term. disease ontology we compared the candidate do annotations to a pre-existing gene-disease annotation database mined from ncbi's generifs. while online mendelian inheritance in man is probably the most widely recognized gene-disease database, we chose the generif-backed database for comparison because a) it used the do for annotations thus enabling direct comparison, b) at the time it was created, it contained the majority of the gene-disease associations in omim and significantly extended this set, and c) it reported a very high precision rate - comparable to a manually curated resource. the downside of using this database for comparison was that it had not been updated since and hence it was undoubtedly missing more recent information. in all, of the candidate annotations exactly matched an annotation from the do reference annotations, matched a more general term in the same lineage as a reference annotation, matched a more specific term, and had no match. we refer to the annotations with no match as'novel candidates' as these represent potential new annotations. gene ontology we next compared the candidate annotations to reference annotations from the go annotation database. the goa database is the accepted standard public reference for go-based annotation of human gene products. when this analysis was conducted, it provided annotations for distinct human genes. of the mined go annotations, matched an annotation in the goa database exactly, matched more specific terms than goa annotations, matched more general terms, and did not match any. the go is divided into three distinct topical branches: biological process, molecular function and cellular component. in all, 54% of the candidate annotations used bp terms, 14% used mf, and the remaining 32% used cc. in the remainder of this article, we focus on the candidate bp annotations because they were the most plentiful in the output and because they are generally the most difficult annotations to determine automatically using other methods. of the candidate bp annotations, were direct matches to gene-function annotations in the goa reference, matched narrower terms than existing annotations, matched more general terms than existing annotations, and the remaining did not match any annotations in the goa database. manual evaluations for candidate annotations with no matches in the reference databases, we extracted a random sample and submitted it to expert curators to manually evaluate the quality of the predictions. for the do evaluations, the sample contained candidate annotations or approximately 10% of the candidate annotations that either had no match in the reference set or matched a child of a reference annotation. for the go we originally selected novel candidate annotations for the evaluation but later had to remove from consideration after discovering that, due to an error in processing, these were actually parents of reference annotations. the sample sizes were the largest that could be processed in a reasonable amount of time based on the curator resources at our disposal. for the evaluation, the curators assigned each candidate annotation in the sample to one of eight categories as follows. category 1: yes, this would lead to a new annotation 1a: perfect match - the candidate annotation is exactly as it would be from a curator 1b: not specific enough - the candidate annotation is correct but a more specific term should be used instead 1c: too specific - the candidate annotation is close to correct, but is too specific given the evidence at hand category 2: maybe, but insufficient evidence: 2a: evaluator could not find enough supporting evidence in the literature after about minutes of looking 2b: there is disagreement in the literature about the truth of this annotation category 3: no, this candidate annotation is incorrect: 3a: incorrect concept recognition which is defined as the transfer of genetic information to a bacterium from a bacteriophage or between bacterial or yeast cells mediated by a phage vector - a completely different concept from signal transduction as intended in the sentence.) 3b: incorrect sentence context - the sentence is a negation or otherwise does not support the predicted annotation for the given gene 3c: this sentence seems factually false disease ontology to assess the quality of the candidate do annotations with no match in the reference set discussed above, we reviewed the randomly selected novel candidate annotations manually according to the criteria outlined above. the reviewers were experts in do-based gene annotation and are active participants in the development of the do. out of the candidates evaluated, were classified in category with assigned to category 1a. figure provides a breakdown of the results of the manual evaluation. nearly all of the errors fall into category 3b. for example, the ncbo annotator correctly identified the disease term'neuroblastoma' in the following sentence: "dock3-mediated rac activation promotes reorganisation of the cytoskeleton in sh-sy5y neuroblastoma cells and primary cortical neurones as well as morphological changes in fibroblasts"; however, the assumption that the occurrence indicates an association with the gene dock does not hold because the sentence is referring to a neuroblastoma cell line rather than to the human disease. gene ontology we then followed the same protocol to assess the quality of candidate biological process annotations with no match in the goa reference set. a professional curator familiar with go annotation manually inspected a random sample of candidate annotations with no match in the reference set and classified each with the same categories used for the do evaluations. the performance was substantially worse for the novel go annotations than for the do - in particular only of the candidates that were evaluated were assigned to category 1a. aside from the low number of'exact match' results, candidate go annotations generated many more uncertain results as well as a very large fraction of errors due to incorrect sentence context. it is worth noting that not one of errors detected in either the do or the go annotations were classified as 3c. this provides evidence that the text of the gene wiki is consistently correct. overall precision of annotation mining system we integrated the results from both the manual assessments and the comparison to reference data sets to provide an estimate of the overall likelihood that a predicted annotation is biologically valid. we refer to this likelihood as'precision' where precision is equal to the ratio of true positive predictions to the sum of true and false positive predictions. in order to calculate precision, we need to divide all predictions into two classes'true' and'false'. for this analysis, the true positive set contained the direct matches to reference annotations, the matches to parents of reference annotations, and the estimated number of category novel annotations. the estimated counts for the novel annotations were derived by multiplying the precision rates observed in the manually evaluated sample by the total number of novel candidate annotations. to account for the novel predicted annotations that were classified into category 2a, we provide two estimates of precision: one that includes the 2a results as true positives and one that includes them as false positives. in this way we produce an estimated lower and upper bound on the system's actual precision. the estimated upper bound for the overall precision of the annotation protocol was thus calculated as: exact+moregeneral+e1a+e1b+e1c+e2aeall*all and the estimated lower bound as: exact+moregeneral+e1a+e1b+e1ceall*all where exact, more general, child, and none correspond to agreements between candidate annotations and a reference set; e1a, e1b, e1c and e2a refer to evaluation categories 1a, 1b, 1c and 2a, and eall refers to the total number of novel candidates evaluated. using equations and we estimated a range for precision of 90-93% for the do annotations and 48-64% for the go annotations. in retrospect, it may be more appropriate to remove the category 1c annotations from the'biologically valid' grouping here, but, since there were no occurrences of this category in either the do or the go evaluations, this change would not impact the results of the present analysis. potential applications in enrichment analysis given these estimates of precision, we next checked to see if the annotations produced here could be used immediately in applications relevant to biological discovery. specifically we assessed the use of the new annotations in the context of gene-set enrichment analyses. gene-set enrichment analyses provide a knowledge-based statistical assessment of the important concepts related to a set of genes. since tools for performing enrichment analysis are noise tolerant, but cannot function without annotations, the use of automatically derived annotations as an extension to curated annotations can provide increased power and flexibility in terms of which concepts can be detected. for example, if a sufficient body of relevant text can be identified for each gene in a study set, enrichment analysis can now be conducted using any of the ontologies present in the ncbo bioportal using the ncbo annotator. the crux of this kind of analysis is the identification of a sufficient quantity of relevant text. since each gene wiki article is exclusively about one gene, as opposed to a typical article indexed in pubmed that may mention many genes and processes in very specific contexts, the articles form a particularly useful corpus. while the gene wiki alone does not yet have enough content to warrant the use of annotations derived solely from its text in knowledge-based analyses, we hypothesized that it could be used as an extension to other sources of gene-centric text to improve the results of text-driven enrichment analyses. evaluation of mined go annotations in gene-set enrichment analysis to assess the potential value of the gene-wiki derived go annotations, we measured their impact on a controlled gene set enrichment experiment based on the pattern introduced by lependu et al.. as an example, consider the go term for'muscle contraction'. this go annotation was associated with genes in the goa database. after blinding ourselves to the origin of this gene list, we performed text mining on the titles and abstracts of the publications associated with those genes. we expected to find the term'muscle contraction' to be enriched relative to the background occurrence in all publications. however, when using article titles and abstracts alone, we found no such statistical enrichment. in contrast, after adding gene wiki text to the corpus of publication titles and abstracts, the term'muscle contraction' was highly enriched. to confirm that this result was not an artifact, we performed a simulation in which, for each of iterations, we selected genes randomly from the set of genes with any go annotations from the goa database and performed the identical analysis. this provided a way to estimate the probability that we would observe this improvement from the addition of the gene wiki text for random gene sets. the higher this probability, the lower our confidence that the additional annotations provided by the gene wiki provided a real signal of value. in only of the simulation runs did the gene wiki-enhanced annotation set produce a significant p value when the pubmed-only set did not. in fact, for the random gene sets the gene wiki derived annotations were slightly more likely to make the p values worse though in most cases there was no impact at all. this simulation demonstrated that, as should be expected from results presented above, the annotations mined from the gene wiki are both non-random and are clearly correlated with annotations shared in curated databases. in addition, it provided empirical evidence that fisher's exact test is appropriate in this situation. we extended this analysis to all go terms used in human gene annotations and found a consistent improvement in the enrichment scores. using abstracts and titles alone, this protocol resulted in a significant enrichment score for gene sets. enhancing the data from pubmed mining with the gene wiki text resulted in an increase to significant tests. the relatively low rate of annotation rediscovery for go terms based on the text from abstracts related to associated genes is not surprising. other groups have reported that only about 10% of the curated go annotations can be found in the text of the abstract of the paper cited as evidence for the annotation. what we show here simply demonstrates that the text from the gene wiki can extend the reach of systems that rely solely on the text of pubmed abstracts for annotation mining. while the results are preliminary, a similar text-driven enrichment analysis that used the do rather than the go also showed improvement when annotations mined from the gene wiki were included alongside annotations mined from pubmed abstracts. discussion our results demonstrate that the gene wiki is an important repository of knowledge about the human genome that is different from and complementary to other biological knowledge sources. its articles provide a growing source of gene-specific text that pulls together relevant bits of information from the published literature into a form that is both useful for human consumption and highly amenable to natural language processing. importantly, the mass collaborative approach to assembling these wiki articles scales with the explosive growth of the biomedical literature. annotations mined from gene wiki text both recapitulated and extended knowledge in existing databases. disease ontology versus gene ontology for mining gene annotations we speculate that the differences in the level of precision estimated for the predicted annotations using the go and the do were primarily the result of two key factors: the differing scopes of the two ontologies and differences in the way that the two communities view annotations. the extremely broad scope of the go means that it has many fairly general terms, like'transduction', that can result in errors due to polysemy. while fairly general terms do exist in the do, such as'dependence', they are far less frequent. aside from increased numbers of mismatched concepts from the go, the criteria used by curators for establishing that a go annotation is fit for inclusion in the goa database are more stringent than the criteria for establishing a gene-disease association. this is evident when comparing the category 2a results for the two ontologies. as shown in figures and the reviewers felt that only 4% of the evaluated do-based candidates needed additional investigation while 26% of the go-based candidates were judged to require more evidence. in many cases, the 2a scores for the go evaluations resulted when the main evidence for the new annotation was from research conducted in a model organism. in order for a go curator to accept evidence from another organism, further analysis of sequence and phylogenies must be conducted and such analysis was beyond the scope of this evaluation. in the case of the do annotations, curators accept evidence from model organisms as sufficient for forming an association. potential modifications it is worth noting that the basic protocol described here is not tied to the annotator, it could be used with any concept detection system. the annotator was used in this analysis because it has a fast convenient api, access to a large number of ontologies including the do, and has been shown to have similar performance to metamap - a longstanding, commonly used tool to for biomedical concept recognition. improvements to the annotator workflow, such as negation detection, are ongoing and will benefit the protocol described here. for critical assessments of other text-mining tools and applications in biology, see the biocreative competitions. in particular see for a discussion of challenges in working with the go. in our evaluation of the results, we only had access to a single qualified go curator and two do curators, hence we had to constrain the size of the sample processed and could not calculate inter-annotator agreement. while we suggest that the number of manually reviewed candidate annotations was sufficient to provide rough estimates on the precision of this protocol, additional evaluations would certainly be valuable. the scarcity of qualified curators, and the even more apparent scarcity of their time, provides additional motivation for continuing this area of research. applications we expect that predicted annotations from the gene wiki will have several applications. first, professional biocurators could use gene wiki-derived annotations as a useful starting point for their curation efforts. most obviously, these candidate annotations could be processed according to current curatorial standards to approve, refine, or reject them as formal annotations. on an even more basic level, curators could simply prioritize pubmed articles that were used in inline gene wiki citations for formal review. in this scenario, the gene wiki would be used as a crowdsourced method to identify the most relevant scientific literature, an increasingly difficult problem based on the rapid growth of pubmed. second, these candidate annotations could be used directly by end users in statistical analyses that are tolerant to noisy data. for example, gene set enrichment analysis is among the most popular analysis strategies for genomic studies, and the underlying statistical test is, by definition, noise tolerant. a recent application called the rich annotation summarizer performs gene set enrichment analysis using any of the ontologies in the ncbo bioportal by applying the annotator to extract relevant annotations from medline abstracts and the ncbo resource index. annotations derived from the gene wiki could fit directly into these and related systems.  ontology-based gene annotation forms a crucial component of many tasks in bioinformatics, but accumulating these annotations is costly. by combining the mass collaboratively generated text of gene-specific articles in the gene wiki with readily accessible natural language processing technology, we introduced a new and scalable system for generating gene annotations. as with any application of currently available natural language processing, this system is not error free. go annotations in particular proved difficult to produce at high precision. looking forward, we can expect that improvements in information extraction technology and the continued expansion of the gene-centric text in the gene wiki will combine to produce an increasingly valuable process for harnessing the ever-expanding body of functional knowledge about the human genome. 
we specifically explore the generation of structured annotations using the gene ontology and the human disease ontology. our system produced candidate gene annotations using the disease ontology and candidate annotations using the gene ontology from the text of the gene wiki. based on manual evaluations and comparisons to reference annotation sets, we estimate a precision of 90-93% for the disease ontology annotations and 48-64% for the gene ontology annotations. the goal of this study is to define and assess a computational strategy for translating the text of gene wiki articles into ontology-based gene annotations. the gene wiki is comprised of more than review articles, each describing one human gene. the gene wiki is a rapidly growing corpus of text focused on human gene function. we further demonstrate that this data set can systematically improve the results from gene set enrichment analyses. ontology-based gene annotations are important tools for organizing and analyzing genome-scale biological data. the gene wiki makes use of wikipedia as a low-cost, mass-collaborative platform for assembling text-based gene annotations. here, we demonstrate that the gene wiki can be a powerful resource for generating ontology-based gene annotations. these annotations can be used immediately to improve workflows for building curated gene annotation databases and knowledge-based statistical analyses.
 ontology-based gene annotations are important tools for organizing and analyzing genome-scale biological data. collecting these annotations is a valuable but costly endeavor. the gene wiki makes use of wikipedia as a low-cost, mass-collaborative platform for assembling text-based gene annotations. the gene wiki is comprised of more than review articles, each describing one human gene. the goal of this study is to define and assess a computational strategy for translating the text of gene wiki articles into ontology-based gene annotations. we specifically explore the generation of structured annotations using the gene ontology and the human disease ontology.  our system produced candidate gene annotations using the disease ontology and candidate annotations using the gene ontology from the text of the gene wiki. based on manual evaluations and comparisons to reference annotation sets, we estimate a precision of 90-93% for the disease ontology annotations and 48-64% for the gene ontology annotations. we further demonstrate that this data set can systematically improve the results from gene set enrichment analyses.  the gene wiki is a rapidly growing corpus of text focused on human gene function. here, we demonstrate that the gene wiki can be a powerful resource for generating ontology-based gene annotations. these annotations can be used immediately to improve workflows for building curated gene annotation databases and knowledge-based statistical analyses. 