BACKGROUND
the automatic identification of gene and protein names in the medline® database of literature citations is a challenging named entity recognition  task.
biomedical ner has been an active research area for some time.
systems capable of high performance on this task are desirable because ner precedes other tasks including information extraction and text mining.
the apparent simplicity of the gene/protein ner task conceals its inherent complexity stemming from an often unconventional and ambiguous genetic nomenclature.
we have previously developed abgene, a gene/protein name tagger trained on medline abstracts using a combination of statistical and rule-based strategies.
due to the difficulty of manually evaluating abgene results, we needed to create a tagged corpus for evaluating the performance of abgene applied to full text articles.
the genia corpus version  <dig>  contains a total of  <dig>  biological terms annotated by two domain experts  <cit> .
however, it was not suitable for our purposes because we ran abgene on unrestricted text, and the genia corpus is restricted to text retrieved using the search terms human, blood cell and transcription factor.
additionally, the entities in genia are allowed to be generic, whereas abgene was designed to extract specific gene/protein names only.
one fundamental problem in corpus annotation is the definition of what constitutes an entity to be tagged.
for example, the muc- <dig> named entity task to identify organizations, persons and locations in text necessitated the lengthy muc- <dig> named entity task definition, which specify the rules for annotating each entity  <cit> .
the following excerpts from the muc- <dig> named entity task definition exemplify the complexity of the annotation process:
a.
<dig>  entity-expressions that modify non-entities
entity names used as modifiers in complex nps that are not proper names are to be tagged when it is clear to the annotator from context or the annotator's knowledge of the world that the name is that of an organization, person, or location.
a.
<dig>  entity-strings embedded in entity-expressions
in some cases, multi-word strings that are proper names will contain entity name substrings; such strings are not decomposable; therefore, the substrings are not to be tagged.
a.
<dig> .
<dig> the definite article in an alias
when a definite article is commonly associated with an alias, it also must be tagged.
<enamex type="person">the godfather</enamex>
however, the scoring program ignores a certain list of premodifiers as specified in section  <dig>  which may make the scoring in some of these cases more lenient than this rule implies.
the scorer does *not* ignore those premodifiers within quotation marks such as inside the tags in a.
<dig> .
<dig> above.
the developers of the genia corpus followed a less exacting annotation strategy that did not allow determiners, ordinals nor cardinals to appear in tagged entities, but left qualifiers, including adjectives, as somewhat arbitrary judgement calls  <cit> .
for genetag annotation, we chose a wide definition of a gene/protein entity, but added a constraint that requires the tagged entity to refer to a specific entity, hereafter called the "specificity constraint."
the specificity constraint allows for entities like tat dna sequence but not dna sequence.
no distinctions were made between genes, proteins, rna, domains, complexes, sequences, fusion proteins, etc.
a finer-grained definition is possible, for example, proteins, genes and rna can be distinguished as separate entities using machine learning with 78–84% accuracy  <cit> .
however, most biomedical ner systems do not make these distinctions.
also, hatzivassiloglou et al.
found that their machine learning algorithms did not perform well against a human baseline model, suggesting that either the humans were correct, and the decreased performance was due to classification difficulty, or the machine-learned programs were penalized for being more consistent than humans.
because humans agreed only about 77% of the time on protein, gene and rna labels, the inclusion of these distinctions in a gold standard would be an additional source of significant ambiguity.
our decision to include domains, complexes, subunits and promoters  was based on gene names in genbank.
for example, the name in  suggests that a particular subunit is considered to be a gene entity because it is associated with a genbank sequence.
similarly,  shows that promoters can be considered to be gene-sequence-related entities.
our specificity constraint requires the presence of sf3b <dig> in  and transaldolase in .
thus, subunit  <dig> alone in  and promoter region alone in  would not be true positives.
1) sf3b <dig>  splicing factor 3b, subunit 4
2) mus musculus transaldolase gene, promoter region
some exceptions to the specificity constraint were incorporated into the annotation scheme due to their appearance in genbank:
3) bhlh transcription factor mrna
4) xenopus laevis similar to pou domain gene
by defining a gene based on gene names in genbank, but requiring only a partial match, we have addressed the fact that gene names in text are often not exact matches to their official names.
this is an advantage of manually annotating a corpus instead of relying on lists of official gene names for a gold standard – each entity in each context can be expertly evaluated and revised if necessary.
the above examples illustrate some of the motivation behind the compilation of a list acceptable alternative gene/protein names.
in , many systems would probably not extract the entire entity, and would be penalized.
a more flexible evaluation would be possible if it were recognized that "sf3b4" and "sf3b <dig>  splicing factor 3b," are acceptable alternatives to the full form.
it would also allow systems to delete the organism name in , as well as the fact that it refers to the promoter region.
the acceptable alternatives are always subject to the specificity constraint so that the important parts of gene/protein names are preserved.
in addition to the specificity constraint described here, we applied semantic constraints to define gene/protein entities.
semantic constraints are rules stating that the tagged entity must contain its true meaning in the sentence context.
these constraints were geared towards multiword entities, especially ones that include numbers, letters and acronyms.
for example, the name in  requires rabies because rig implies that the gene mentioned in this sentence refers to the rabies immunoglobulin, and not just any immunoglobulin.
in , the word receptor is necessary to differentiate igg receptor from igg, an important semantic distinction.
in , the number  <dig> is needed to accurately describe a specific type of tumor necrosis factor, although tumor necrosis factor alone might be adequate in a different context.
5) rabies immunoglobulin
6) igg receptor
7) tumor necrosis factor 1
genetag consists of 20k sentences that have been run through abgene  <cit>  and manually annotated with gene and protein names  by experts in biochemistry, genetics and molecular biology.
it is a heterogeneous set of sentences that contains many true positive gene names, and also many non-gene entities that are morphologically similar to gene names.
there are approximately 24k instances of gene/protein names in the 20k sentences.
15k of the sentences were used in biocreative- <dig> task 1a  <cit> .
previous biomedical ner systems were difficult to compare because there were few large gene-tagged corpora available.
although genetag was not originally intended to be widely distributed, in releasing the corpus to the larger biomedical ner community through the biocreative evaluation, we hoped to stimulate interest in this area and provide a means to evaluate multiple systems on unrestricted biomedical text.
genetag annotation guidelines were designed to define true positive gene/protein names in terms of their specificity and semantics.
each sentence in genetag is annotated with acceptable alternatives to the gene/protein names it contains, allowing for partial matching with semantic constraints, a more meaningful measure of the performance of an ner system than unrestricted partial matching.
this paper provides some background on the corpus including 1) sentence selection, 2) definition of a gene/protein name, 3) tokenization and partial matching and 4) tagging consistency.
implementation
we used naïve bayesian learning to predict the likelihood that a medline document contains a gene/protein name  <cit> .
the classifier was trained on a set of medline documents containing known gene/protein names against the rest of medline.
high-scoring documents almost always contain gene/protein names, and low-scoring documents often contain no gene/protein names.
we found that we could apply our classifier to sentences as well as documents.
since we required the corpus to contain both true gene/protein names and non-gene-related entities, we randomly selected 10k high- and 10k low-scoring medline sentences as the basis for our corpus.
the bayesian classifier is implemented in c++.
the 20k sentences were run through abgene, and the resulting gene/protein tags were manually transformed by three domain experts in genetics, biochemistry and molecular biology.
annotation was done using a web interface, using check boxes under words to indicate gene/protein names for the gold standard, and manual entry of partial match alternative names and indices .
a flexible evaluation program was implemented in perl.
RESULTS
genetag contains 20k sentences tagged with gene/protein names .
an additional file of acceptable alternatives to the tagged gene/protein names is available.
the gold standard combined with the acceptable alternatives allows for flexible scoring using meaningful partial matching.
the flexible evaluation program checks first for the gold standard version of the name, then checks for each of the available alternatives to the gold standard version of the name.
word indices ensure that specificity and semantic constraints are met for sentences that contain overlapping, repeated and/or nested names.
discussion
tokenization is problematic in gene/protein name ner because punctuation cannot be globally removed to make processing straightforward.
gene and protein names often contain hyphens, parentheses, brackets, and other types of punctuation, thus using penn treebank style tokenization  <cit> , where most punctuation is split from neighboring words, is not ideal.
the penn treebank contains "subtleties" for hyphens and dashes, similarly, our original tokenization was based on rules that usually apply to gene and protein names: 1) do not split on hyphens , 2) do not split on single quotation marks , 3) do not split on commas -atpase), and 4) do not split on parentheses and brackets -lyn fusion).
abgene applies these tokenization rules to terms that resemble gene names , and not to other phrases , resulting in inconsistent tokenization.
some parentheses are surrounded by spaces, others are not, dependent on whether the phrase resembled a gene name or not.
the corpus was later re-tokenized automatically  to be closer to the penn treebank style, with the exception of hyphenations and single quotation marks.
although the second tokenization is more consistent than the original tokenization, it introduces some awkward spaces into gene names, for example in ,  and .
8) human alpha  <dig>  2-mannosidase
9)  -1beta
10) d. melanogaster surf- <dig> / rpl7a
exact matching to a gold standard corpus is too stringent for evaluation purposes since there is not always one and only one correct answer for each entity.
however, unrestricted partial matching is suboptimal because it allows unsuitable fragments with insertions and deletions to count as true positives.
canonical form matching is not possible for many organisms given current resources.
in genetag we allowed partial matching, but the matches were subject to the specificity and semantic constraints described earlier for defining a gene/protein name.
partial matching with semantic constraints allows for acceptable gene/protein name alternatives like p <dig>  p <dig> genes, p <dig> protein, but rejects fragments like all p <dig> genes, p <dig> patients and -1beta.
our flexible matching strategy entails that the tokenization of the gold standard corpus and test set be identical since it relies on the location of each word in the sentence.
a positional approach is necessary because often one sentence contains many gene/protein names, including overlapping, repeated and/or nested names.
to illustrate this problem, suppose an ner system identified the following gene in sentence fragment : tgf-beta <dig>  it would be impossible to know which of the two instances of tgf-beta <dig> was actually extracted by the system without sentence indices.
in , a similar ambiguity exists if an ner program were to extract the gene list: tnf, toll-like receptor.
does tnf refer to human tumor necrosis factor or tnf receptor 1?
did the system extract part of toll-like receptor  <dig> or toll-like receptor 4?
the exact location of gene/protein names in a sentence is essential for debugging ner programs based on contextual clues, for example, in  the first instance of tgf-beta <dig> is preceded by stimulating, and the second instance has a more generic context.
the requirement that the gold standard and test sets have the same tokenization is disadvantageous.
however, positional information is necessary for further natural language processing , since it affects both syntactic structure and semantic interpretation.
11) inflammation has been inferred to play a major role in stimulating tgf-beta <dig> production since high concentrations of tgf-beta <dig> have been found in the lungs of patients with various diffuse inflammatory lung diseases.
12) rip3-deficient cells showed normal sensitivity to a variety of apoptotic stimuli and were indistinguishable from wild-type cells in their ability to activate nf-kappa b signaling in response to the following: human tumor necrosis factor , which selectively engages mouse tnf receptor  <dig> ; cross-linking of the b- or t-cell antigen receptors; peptidoglycan, which activates toll-like receptor  <dig> ; and lipopolysaccharide , which stimulates toll-like receptor  <dig>
genetag was annotated manually, thus the tags assigned were judgment calls by human annotators.
annotation guidelines were established, however, many grey areas soon appeared for which no generalized rules were formulated.
in particular, names containing conjunctions were difficult to tag.
in , it seems excessive to require an ner program to recognize the entire fragment, however,  <dig> alone is not a valid gene name.
13) src homology  <dig> and 3
due to the specificity constraint, we were unable to formulate a syntactic rule for handling conjunctions systematically.
for example, the underlined terms satisfying the specificity constraint in examples ,  and  occur arbitrarily to the left or right of the word "or."
accepted alternatives for  were: icp <dig> .
<dig>  icp <dig> .
<dig> promoters, mutant icp <dig> .
<dig> promoters, wild-type or mutant icp <dig> .
<dig>  and mutant icp <dig> .
<dig>  accepted alternatives for  were: rab1b, rab1b, - <dig>  rab1b, - <dig>  - <dig>  and rab1b, - <dig>  - <dig>  - <dig>  accepted alternatives for  were: beta, or gamma pkc, gamma pkc, and pkc.
14) wild-type or mutant icp <dig> .
<dig> promoters
15) rab1b, - <dig>  - <dig>  - <dig>  or -11a
16) alpha, beta, or gamma pkc
adjoining gene/protein names also presented a challenge during annotation, since gene/protein name boundaries are not immediately obvious in these instances.
it is an intricate task to assign the exact boundaries in  and , even for domain experts, and in  it is unclear whether e <dig>  rad <dig> and ubc <dig> are stand-alone synonyms or parts of a complex or fusion protein.
similarly, it is difficult even for experts to determine how many separate entities are denoted in .
often the sentence context, and sometimes the entire abstract context, is inadequate for the correct determination to be made, so other resources  must be consulted.
this time-consuming lookup step necessitates a trade-off between tagging consistency and annotation time.
17) stress-activated protein kinase-jun n-terminal kinase
18) tumor necrosis factor  receptor-associated factor
19) e <dig> rad <dig>
20) laz3/bcl <dig> btb/poz
even in unambiguous cases, tagging inconsistencies can appear due to human error.
in particular, the partial matching alternatives are sensitive to inconsistencies because the names and indices were input manually into a text box on an annotation web page .
CONCLUSIONS
we have described the genetag corpus of tagged gene/protein names in medline text which was used in biocreative task 1a.
the corpus was designed to contain both true and false positive gene/protein names in a variety of contexts.
gene/protein names are defined widely, but are subject to specificity and semantic constraints.
the annotation guidelines were designed with the goal of allowing flexible matching to the gold standard, while retaining the true meaning of the tagged entities.
arbitrary partial matches not corresponding to a complete and meaningful entity fail to meet the annotation guidelines and are scored as false positives and/or false negatives.
a more detailed definition of a gene/protein name, as well as additional annotation rules, could improve interannotator agreement and help solve some of the tagging inconsistencies.
subtle tokenization issues exist in the corpus, and the requirement that the gold standard and test sets have the same tokenization is disadvantageous .
however, a positional approach is necessary to disambiguate sentences which contain adjoined, repeated and/or nested gene names, and for future nlp applications.
a more robust approach would use character-based rather than word-based indices to allow for a wider diversity of tokenization.
authors' contributions
lt outlined the annotation guidelines, annotated text, wrote the evaluation programs and drafted the manuscript.
nx created the web interface for corpus annotation.
lht and wm participated in designing the annotation guidelines and annotated text.
wjw conceived of the project, and participated in its design and coordination.
all authors read and approved the final manuscript.
figures and tables

 the automatic identification of gene and protein names in the medline® database of literature citations is a challenging named entity recognition task. biomedical ner has been an active research area for some time. systems capable of high performance on this task are desirable because ner precedes other tasks including information extraction and text mining. the apparent simplicity of the gene/protein ner task conceals its inherent complexity stemming from an often unconventional and ambiguous genetic nomenclature. we have previously developed abgene, a gene/protein name tagger trained on medline abstracts using a combination of statistical and rule-based strategies. due to the difficulty of manually evaluating abgene results, we needed to create a tagged corpus for evaluating the performance of abgene applied to full text articles. the genia corpus version contains a total of biological terms annotated by two domain experts. however, it was not suitable for our purposes because we ran abgene on unrestricted text, and the genia corpus is restricted to text retrieved using the search terms human, blood cell and transcription factor. additionally, the entities in genia are allowed to be generic, whereas abgene was designed to extract specific gene/protein names only. one fundamental problem in corpus annotation is the definition of what constitutes an entity to be tagged. for example, the muc- named entity task to identify organizations, persons and locations in text necessitated the lengthy muc- named entity task definition, which specify the rules for annotating each entity. the following excerpts from the muc- named entity task definition exemplify the complexity of the annotation process: a. entity-expressions that modify non-entities entity names used as modifiers in complex nps that are not proper names are to be tagged when it is clear to the annotator from context or the annotator's knowledge of the world that the name is that of an organization, person, or location. a. entity-strings embedded in entity-expressions in some cases, multi-word strings that are proper names will contain entity name substrings; such strings are not decomposable; therefore, the substrings are not to be tagged. a.. the definite article in an alias when a definite article is commonly associated with an alias, it also must be tagged. <enamex type="person">the godfather</enamex> however, the scoring program ignores a certain list of premodifiers as specified in section which may make the scoring in some of these cases more lenient than this rule implies. the scorer does *not* ignore those premodifiers within quotation marks such as inside the tags in a.. above. the developers of the genia corpus followed a less exacting annotation strategy that did not allow determiners, ordinals nor cardinals to appear in tagged entities, but left qualifiers, including adjectives, as somewhat arbitrary judgement calls. for genetag annotation, we chose a wide definition of a gene/protein entity, but added a constraint that requires the tagged entity to refer to a specific entity, hereafter called the "specificity constraint." the specificity constraint allows for entities like tat dna sequence but not dna sequence. no distinctions were made between genes, proteins, rna, domains, complexes, sequences, fusion proteins, etc. a finer-grained definition is possible, for example, proteins, genes and rna can be distinguished as separate entities using machine learning with 78–84% accuracy. however, most biomedical ner systems do not make these distinctions. also, hatzivassiloglou et al. found that their machine learning algorithms did not perform well against a human baseline model, suggesting that either the humans were correct, and the decreased performance was due to classification difficulty, or the machine-learned programs were penalized for being more consistent than humans. because humans agreed only about 77% of the time on protein, gene and rna labels, the inclusion of these distinctions in a gold standard would be an additional source of significant ambiguity. our decision to include domains, complexes, subunits and promoters was based on gene names in genbank. for example, the name in suggests that a particular subunit is considered to be a gene entity because it is associated with a genbank sequence. similarly, shows that promoters can be considered to be gene-sequence-related entities. our specificity constraint requires the presence of sf3b in and transaldolase in. thus, subunit alone in and promoter region alone in would not be true positives. 1) sf3b splicing factor 3b, subunit 4 2) mus musculus transaldolase gene, promoter region some exceptions to the specificity constraint were incorporated into the annotation scheme due to their appearance in genbank: 3) bhlh transcription factor mrna 4) xenopus laevis similar to pou domain gene by defining a gene based on gene names in genbank, but requiring only a partial match, we have addressed the fact that gene names in text are often not exact matches to their official names. this is an advantage of manually annotating a corpus instead of relying on lists of official gene names for a gold standard – each entity in each context can be expertly evaluated and revised if necessary. the above examples illustrate some of the motivation behind the compilation of a list acceptable alternative gene/protein names. in, many systems would probably not extract the entire entity, and would be penalized. a more flexible evaluation would be possible if it were recognized that "sf3b4" and "sf3b splicing factor 3b," are acceptable alternatives to the full form. it would also allow systems to delete the organism name in, as well as the fact that it refers to the promoter region. the acceptable alternatives are always subject to the specificity constraint so that the important parts of gene/protein names are preserved. in addition to the specificity constraint described here, we applied semantic constraints to define gene/protein entities. semantic constraints are rules stating that the tagged entity must contain its true meaning in the sentence context. these constraints were geared towards multiword entities, especially ones that include numbers, letters and acronyms. for example, the name in requires rabies because rig implies that the gene mentioned in this sentence refers to the rabies immunoglobulin, and not just any immunoglobulin. in, the word receptor is necessary to differentiate igg receptor from igg, an important semantic distinction. in, the number is needed to accurately describe a specific type of tumor necrosis factor, although tumor necrosis factor alone might be adequate in a different context. 5) rabies immunoglobulin 6) igg receptor 7) tumor necrosis factor 1 genetag consists of 20k sentences that have been run through abgene and manually annotated with gene and protein names by experts in biochemistry, genetics and molecular biology. it is a heterogeneous set of sentences that contains many true positive gene names, and also many non-gene entities that are morphologically similar to gene names. there are approximately 24k instances of gene/protein names in the 20k sentences. 15k of the sentences were used in biocreative- task 1a. previous biomedical ner systems were difficult to compare because there were few large gene-tagged corpora available. although genetag was not originally intended to be widely distributed, in releasing the corpus to the larger biomedical ner community through the biocreative evaluation, we hoped to stimulate interest in this area and provide a means to evaluate multiple systems on unrestricted biomedical text. genetag annotation guidelines were designed to define true positive gene/protein names in terms of their specificity and semantics. each sentence in genetag is annotated with acceptable alternatives to the gene/protein names it contains, allowing for partial matching with semantic constraints, a more meaningful measure of the performance of an ner system than unrestricted partial matching. this paper provides some background on the corpus including 1) sentence selection, 2) definition of a gene/protein name, 3) tokenization and partial matching and 4) tagging consistency. implementation we used naïve bayesian learning to predict the likelihood that a medline document contains a gene/protein name. the classifier was trained on a set of medline documents containing known gene/protein names against the rest of medline. high-scoring documents almost always contain gene/protein names, and low-scoring documents often contain no gene/protein names. we found that we could apply our classifier to sentences as well as documents. since we required the corpus to contain both true gene/protein names and non-gene-related entities, we randomly selected 10k high- and 10k low-scoring medline sentences as the basis for our corpus. the bayesian classifier is implemented in c++. the 20k sentences were run through abgene, and the resulting gene/protein tags were manually transformed by three domain experts in genetics, biochemistry and molecular biology. annotation was done using a web interface, using check boxes under words to indicate gene/protein names for the gold standard, and manual entry of partial match alternative names and indices. a flexible evaluation program was implemented in perl.  genetag contains 20k sentences tagged with gene/protein names. an additional file of acceptable alternatives to the tagged gene/protein names is available. the gold standard combined with the acceptable alternatives allows for flexible scoring using meaningful partial matching. the flexible evaluation program checks first for the gold standard version of the name, then checks for each of the available alternatives to the gold standard version of the name. word indices ensure that specificity and semantic constraints are met for sentences that contain overlapping, repeated and/or nested names. discussion tokenization is problematic in gene/protein name ner because punctuation cannot be globally removed to make processing straightforward. gene and protein names often contain hyphens, parentheses, brackets, and other types of punctuation, thus using penn treebank style tokenization, where most punctuation is split from neighboring words, is not ideal. the penn treebank contains "subtleties" for hyphens and dashes, similarly, our original tokenization was based on rules that usually apply to gene and protein names: 1) do not split on hyphens, 2) do not split on single quotation marks, 3) do not split on commas -atpase), and 4) do not split on parentheses and brackets -lyn fusion). abgene applies these tokenization rules to terms that resemble gene names, and not to other phrases, resulting in inconsistent tokenization. some parentheses are surrounded by spaces, others are not, dependent on whether the phrase resembled a gene name or not. the corpus was later re-tokenized automatically to be closer to the penn treebank style, with the exception of hyphenations and single quotation marks. although the second tokenization is more consistent than the original tokenization, it introduces some awkward spaces into gene names, for example in, and. 8) human alpha 2-mannosidase 9) -1beta 10) d. melanogaster surf- / rpl7a exact matching to a gold standard corpus is too stringent for evaluation purposes since there is not always one and only one correct answer for each entity. however, unrestricted partial matching is suboptimal because it allows unsuitable fragments with insertions and deletions to count as true positives. canonical form matching is not possible for many organisms given current resources. in genetag we allowed partial matching, but the matches were subject to the specificity and semantic constraints described earlier for defining a gene/protein name. partial matching with semantic constraints allows for acceptable gene/protein name alternatives like p p genes, p protein, but rejects fragments like all p genes, p patients and -1beta. our flexible matching strategy entails that the tokenization of the gold standard corpus and test set be identical since it relies on the location of each word in the sentence. a positional approach is necessary because often one sentence contains many gene/protein names, including overlapping, repeated and/or nested names. to illustrate this problem, suppose an ner system identified the following gene in sentence fragment: tgf-beta it would be impossible to know which of the two instances of tgf-beta was actually extracted by the system without sentence indices. in, a similar ambiguity exists if an ner program were to extract the gene list: tnf, toll-like receptor. does tnf refer to human tumor necrosis factor or tnf receptor 1? did the system extract part of toll-like receptor or toll-like receptor 4? the exact location of gene/protein names in a sentence is essential for debugging ner programs based on contextual clues, for example, in the first instance of tgf-beta is preceded by stimulating, and the second instance has a more generic context. the requirement that the gold standard and test sets have the same tokenization is disadvantageous. however, positional information is necessary for further natural language processing, since it affects both syntactic structure and semantic interpretation. 11) inflammation has been inferred to play a major role in stimulating tgf-beta production since high concentrations of tgf-beta have been found in the lungs of patients with various diffuse inflammatory lung diseases. 12) rip3-deficient cells showed normal sensitivity to a variety of apoptotic stimuli and were indistinguishable from wild-type cells in their ability to activate nf-kappa b signaling in response to the following: human tumor necrosis factor, which selectively engages mouse tnf receptor; cross-linking of the b- or t-cell antigen receptors; peptidoglycan, which activates toll-like receptor; and lipopolysaccharide, which stimulates toll-like receptor genetag was annotated manually, thus the tags assigned were judgment calls by human annotators. annotation guidelines were established, however, many grey areas soon appeared for which no generalized rules were formulated. in particular, names containing conjunctions were difficult to tag. in, it seems excessive to require an ner program to recognize the entire fragment, however, alone is not a valid gene name. 13) src homology and 3 due to the specificity constraint, we were unable to formulate a syntactic rule for handling conjunctions systematically. for example, the underlined terms satisfying the specificity constraint in examples, and occur arbitrarily to the left or right of the word "or." accepted alternatives for were: icp. icp. promoters, mutant icp. promoters, wild-type or mutant icp. and mutant icp. accepted alternatives for were: rab1b, rab1b, - rab1b, - - and rab1b, - - - accepted alternatives for were: beta, or gamma pkc, gamma pkc, and pkc. 14) wild-type or mutant icp. promoters 15) rab1b, - - - or -11a 16) alpha, beta, or gamma pkc adjoining gene/protein names also presented a challenge during annotation, since gene/protein name boundaries are not immediately obvious in these instances. it is an intricate task to assign the exact boundaries in and, even for domain experts, and in it is unclear whether e rad and ubc are stand-alone synonyms or parts of a complex or fusion protein. similarly, it is difficult even for experts to determine how many separate entities are denoted in. often the sentence context, and sometimes the entire abstract context, is inadequate for the correct determination to be made, so other resources must be consulted. this time-consuming lookup step necessitates a trade-off between tagging consistency and annotation time. 17) stress-activated protein kinase-jun n-terminal kinase 18) tumor necrosis factor receptor-associated factor 19) e rad 20) laz3/bcl btb/poz even in unambiguous cases, tagging inconsistencies can appear due to human error. in particular, the partial matching alternatives are sensitive to inconsistencies because the names and indices were input manually into a text box on an annotation web page.  we have described the genetag corpus of tagged gene/protein names in medline text which was used in biocreative task 1a. the corpus was designed to contain both true and false positive gene/protein names in a variety of contexts. gene/protein names are defined widely, but are subject to specificity and semantic constraints. the annotation guidelines were designed with the goal of allowing flexible matching to the gold standard, while retaining the true meaning of the tagged entities. arbitrary partial matches not corresponding to a complete and meaningful entity fail to meet the annotation guidelines and are scored as false positives and/or false negatives. a more detailed definition of a gene/protein name, as well as additional annotation rules, could improve interannotator agreement and help solve some of the tagging inconsistencies. subtle tokenization issues exist in the corpus, and the requirement that the gold standard and test sets have the same tokenization is disadvantageous. however, a positional approach is necessary to disambiguate sentences which contain adjoined, repeated and/or nested gene names, and for future nlp applications. a more robust approach would use character-based rather than word-based indices to allow for a wider diversity of tokenization. authors' contributions lt outlined the annotation guidelines, annotated text, wrote the evaluation programs and drafted the manuscript. nx created the web interface for corpus annotation. lht and wm participated in designing the annotation guidelines and annotated text. wjw conceived of the project, and participated in its design and coordination. all authors read and approved the final manuscript. figures and tables
semantic constraints are rules requiring the tagged entity to contain its true meaning in the sentence context. to ensure heterogeneity of the corpus, medline sentences were first scored for term similarity to documents with known gene names, and 10k high- and 10k low-scoring sentences were chosen at random. the original 20k sentences were run through a gene/protein name tagger, and the results were modified manually to reflect a wide definition of gene/protein names subject to a specificity constraint, a rule that required the tagged entities to refer to specific entities. the annotation of such a corpus for gene/protein name ner is a difficult process due to the complexity of gene/protein names. each sentence in genetag was annotated with acceptable alternatives to the gene/protein names it contained, allowing for partial matching with semantic constraints. application of these constraints results in a more meaningful measure of the performance of an ner system than unrestricted partial matching. evaluating the performance of biomedical ner systems is impossible without a standardized test corpus.
 named entity recognition is an important first step for text mining the biomedical literature. evaluating the performance of biomedical ner systems is impossible without a standardized test corpus. the annotation of such a corpus for gene/protein name ner is a difficult process due to the complexity of gene/protein names. we describe the construction and annotation of genetag, a corpus of 20k medline® sentences for gene/protein ner. 15k genetag sentences were used for the biocreative task 1a competition.  to ensure heterogeneity of the corpus, medline sentences were first scored for term similarity to documents with known gene names, and 10k high- and 10k low-scoring sentences were chosen at random. the original 20k sentences were run through a gene/protein name tagger, and the results were modified manually to reflect a wide definition of gene/protein names subject to a specificity constraint, a rule that required the tagged entities to refer to specific entities. each sentence in genetag was annotated with acceptable alternatives to the gene/protein names it contained, allowing for partial matching with semantic constraints. semantic constraints are rules requiring the tagged entity to contain its true meaning in the sentence context. application of these constraints results in a more meaningful measure of the performance of an ner system than unrestricted partial matching.  the annotation of genetag required intricate manual judgments by annotators which hindered tagging consistency. the data were pre-segmented into words, to provide indices supporting comparison of system responses to the "gold standard". however, character-based indices would have been more robust than word-based indices. genetag train, test and round data and ancillary programs are freely available at. a newer version of genetag- will be released later this year. march 28– a critical assessment of text mining methods in molecular biology granada, spain 