BACKGROUND
named entity recognition plays an important first role in information extraction and text mining.
recently, due to the increase in biomedical research literature, biomedical ner , which aims at the automatic identification of gene and protein names, has become a research focus.
human-annotated corpora are widely used in the development of bio-ner systems.
there are several well-known corpora that have gene/protein mention annotations, such as genia  <cit> , genetag  <cit> , aimed  <cit> , pennbioie  <cit> , etc.
based on these corpora, many protein mention recognizers have been developed, some of which are able to achieve state-of-the-art performance  <cit> .
nevertheless, a well-known problem remains.
since the protein annotations are made by different groups, and lack an explicit, unanimous rule that defines what should be annotated, it is likely that the annotations in the different corpora are incompatible.
the incompatibility among the corpora brings about several significant problems.
for example, it is difficult to effectively utilize more than one corpus to develop a protein mention recognizer.
there are very few high quality protein recognizers that are developed by utilizing multiple corpora, because it is unlikely for the recognizer to directly benefit from corpus integration.
it is also difficult to compare systems that are developed using different corpora.
although there are many systems that recognize protein mentions from the pubmed text, the reported performance varies significantly, even for instance where the systems were developed with the same method.
for example, methods based on conditional random fields  report f-scores of  <dig> % on genia  <cit>  and  <dig> % on genetag  <cit> .
if each of the two recognizers were evaluated on the other corpus, the performance would be much lower  <cit> .
ultimately, the problems are largely caused by the incompatibility of different protein annotations, and cannot be resolved effectively without understanding the differences in the annotations.
in this paper, we explore the sources of incompatibilities among three well-known corpora with gene/protein annotations, genia, genetag and aimed, and seek solutions to overcome the incompatibilities.
we first observe the reduction in performance that results from using two of the three corpora together .
then, we carefully study the documentation of the three corpora, in order to determine the sources of incompatibility.
through a series of experiments, we quantify the incompatibility problems, while finding reasonable strategies to avoid the problems that are caused by the incompatibility of protein annotations.
this study aims at analyzing the sources of incompatibility, and reducing the corpus inconsistencies.
RESULTS
our research is based on the three selected corpora: genia, genetag and aimed.
here, we present the findings from studying the documentation of the corpora, and from the results of our experiments.
refer to the methods section for further details regarding preliminary experiments, significance tests, related works, and information on the corpora and on the protein mention recognizer used in this research.
characteristic of the corpora
we investigate three corpora that are often used in biomedical natural language processing .
table  <dig> summarizes the differences between the three corpora, which are determined by analyzing the corpora's published literature.
legend:
size: number of abstracts or sentences in the corpus used in this research
entity scope: types of the named entities identified in the corpus: rotein, ene, na
entity number: number of the annotated in-scope entities in the corpus
entity coverage: coverage of in-scope entity occurrences in each sentence
entity type: explicit identification of the types of the annotated in-scope entities
all three corpora contain annotations that identify entities in the text, including proteins and genes; however, only genia contains information that specifies the types of the entities.
this study benefits from the fine-grained protein annotation of genia.
we compare the other two corpora to genia.
genia aims at including an exhaustive annotation of entities of types that are relevant to the corpus, while the other two corpora have based entity annotation on a specific constraint that the tagged entity must ultimately be traceable to a specific gene/protein .
in terms of text organization, compared to genetag, genia and aimed are closer to each other.
genia and aimed are focused on the "human" domain, but genetag covers a more general domain of pubmed.
genia and aimed exhaustively collect sentences in abstracts, but genetag collects sentences that are relevant to ner, in other words, genetag contains both true and false gene/protein names in a variety of contexts.
in terms of text selection for entity annotation, genia and genetag are closer to each other, compared to aimed.
the former two corpora tend to select longer text fragments as entity references .
incompatibility of heterogeneous annotations
these differences suggest that combining the corpora into one training data will harm the performance of protein mention recognizers.
we eventually identified four main sources of incompatibilities, which thoroughly explained the performance degradation when training with the united data.
we applied corresponding strategies toward each aspect of the sources, to reduce performance degradation.
table  <dig> sums up the improvements of minimizing each of these negative effects.
the scores shown in this table are the f-scores of training with the following data: a, a+genia, g and g+genia.
as expected, the f-score performance, when training with combined data, was significantly lower than when training with the pure corpora.
based on the exact matching criterion , the performance is degraded about  <dig> % and  <dig> %, when aimed and genetag are respectively combined with the genia protein annotations as training data .
the succeeding sections detail the sources of the incompatibility, and describe each of the improvements listed in table  <dig>  ways to avoid the negative sides of the heterogeneities are also explained in detail.
incompatibility one: boundary of protein mentions
the selection of text spans, e.g.
the beginning and ending boundaries, to be annotated is identified as one of the major sources for the incompatibility that exists for protein annotation across the three corpora.
since it is difficult to determine whether the category of an entity, which occurs before or after the entity, can be considered as part of the entity name or not, the selection of boundaries is complicated  <cit> .
for example, the english word "protein" can be treated differently in the following two text expressions, "p21ras protein" and "tumor suppressor protein".
in the first case, "p21ras" is considered to be sufficient in naming the object.
the term denoting the semantic category, "protein" is redundant, and may not be annotated as a part of the entity name.
in fact, the decision to annotate "protein" in such a case often does not affect the utility of a ner system, because the system has correctly identified "p21ras" as a protein, and this information is adequate for mining the relationship between "p21ras" and another protein.
similarly, "the p21ras protein" or "the p21ras" could also be considered to be acceptable.
however, in the second case, without "protein", "tumor suppressor" is inadequate in denoting the protein.
the actual meaning is changed when omitting "protein".
the semantic category "protein", which serves as an important clue, is quite essential in distinguishing this specific entity from a more general one.
thus, the inclusion of the word "protein" in the protein annotation is dependent on annotation scheme.
genia and genetag almost always include the word "protein" in protein annotations, which indicates their schemes on words like "protein", but aimed excludes "protein" in most cases.
the difference in boundary word selection between different protein annotation schemes can be measured in two ways: by calculating the average length of protein mentions and annotation entropy of boundary words.
average length of protein mentions
the ambiguous annotation of boundary words is also a factor that affects the average length of the protein mentions in the three corpora.
the average length per protein mention is  <dig>  tokens in the aimed corpus,  <dig>  tokens in the genia corpus, and  <dig>  tokens in the genetag corpus.
many long protein mentions are introduced when we add the genia annotations into aimed, creating another possible source of performance degradation in recognizing the shorter protein mentions in the aimed corpus.
this observation suggests that the genia and genetag corpora are inclined to select more descriptive expressions for the protein annotation, in comparison with aimed.
annotation entropy of boundary words
in a given corpus, some words are annotated inside of protein mentions, while other words are not.
the annotation entropy of boundary words is calculated by formula .
where eb denotes the annotation entropy of a given word, pa denotes the percentage of the occurrences of this word that is annotated, and  denotes the percentage of the occurrences of this word that is not annotated.
for the sake of brevity, the  "word" discussed in this section describes the word that appears at the beginning or end of an annotated entity, or describes the word that abuts an annotated entity.
the value of the annotation entropy of a boundary word eb ranges from  <dig>  to  <dig> .
when the annotation entropy of a boundary word eb is  <dig>  this word is perfectly annotated , and keeps this annotation consistency throughout the entirety of the corpus.
on the contrary, when the annotation entropy of a boundary word eb is  <dig>  the word's annotation is so disorderly , that we can hardly find any rules about whether to regard this word as a part of the protein mentions or not.
in general, there are two types of ambiguous boundary words: descriptive adjectives, which usually occur before protein names as modifiers, such as "normal", "activated" or "human", and nouns, which usually occur after protein names as heads .
some boundary words appearing in each corpus are listed in table  <dig>  in order to characterize the differences among the three corpora in terms of the annotation entropy of boundary words, the words with an annotation entropy close to  <dig>  were included in table  <dig>  the genia tagger  <cit>  was used to determine the parts-of-speech of the words.
for the entire boundary words list, see "additional file 1: the boundary words of the genia, genetag and aimed corpora".
here, "noun before" indicates the noun occurring before an entity as a modifier, "noun after" indicates the noun occurring after an entity as a head.
na represents the number of the annotated occurrences, and nn represents the number of un-annotated occurrences.
from table  <dig>  the boundary annotation problem appears for various words.
the distribution of these words is diverse, particularly for adjectives.
since the number of characters tagged in the aimed corpus was kept to a minimum, only the names of protein mentions were annotated, most of which are proper nouns, and most of the adjectives were not annotated.
however, the developers of the genia corpus followed an annotation strategy for which generic terms were annotated.
in the genia annotation scheme, specifiers  do not appear in tagged entities, but qualifiers, including adjectives and noun modifiers, remain.
the adjectives before the protein mentions are annotated only if they are required for clarifying the meaning of the protein mentions .
furthermore, the genetag annotators chose some semantic constraints, which state that the tagged entity must contain its true meaning in the sentence context .
these constraints were geared towards multi-word entities, and especially for entities that include numbers, letters and acronyms.
because the conventions on boundary words vary significantly, to prevent underestimating performance of protein mention recognizers, we need an alternative matching criterion, other than the exact matching.
to provide alternative evaluation perspectives, researchers have developed a variety of evaluation criteria that loosens the matching to varying degrees.
partial matching is also considered  <cit>  , because it has previously been shown to capture the presence of an entity with disregards to its exact textual representation.
according to this partial matching criterion, the performance for the pure and combined data are shown in the second data row of table  <dig>  though by loosening the matching criterion, the performance of the training with the combined data still can not compete with the performance of the training with the pure data, under-estimation can be avoided.
incompatibility two: scope of the entities of interest
although all three corpora include gene/protein mention annotations, the target tasks are different.
the genia corpus aims at providing linguistically rich annotations on biological expressions.
the genetag annotation follows a wide definition with the constraint that a gene/protein entity annotation must refer to a specific entity.
the aimed annotation focuses on extracting interactions among individual proteins.
this difference has affected the scope of the annotated proteins: genia is concerned with all of the protein-mentioning terms, genetag is based on specific genes/proteins, while aimed focuses only on references from individual proteins.
categories of annotated entities
the extent to which the proteins annotated in the genia corpus is defined in the genia ontology  <cit> .
in addition to the protein class, other classes such as dna, rna, cell_line and cell_type are also included.
further, the protein class is categorized into seven subclasses: protein_complex, protein_domain_or_region, protein_family_or_group, protein_molecule, protein_substructure, protein_subunit and protein_etc.
thus, in genia, "protein" is defined to include these seven concepts.
for aimed, the scope of the proteins annotated is described by the following statement in the tagging guidelines: generic protein families are not tagged, only specific names that could ultimately be traced back to specific genes in the human genome are tagged  <cit> .
that is, for example, "tumor necrosis factor" would not be tagged, while "tumor necrosis factor alpha" would be tagged.
finally, based on the gene names from genbank  <cit> , the genetag annotations include domains, complexes, subunits, and promoters when the annotated entities refer to specific genes/proteins.
hence, for the scope of annotated proteins, the documentation of the three corpora explicitly states that:
the mentions of protein families are annotated in genia, but not in aimed.
individual proteins  are annotated in all of the corpora.
both genia and genetag contain protein domain, complex and subunit annotations.
compatible protein annotations
the published literature of the three corpora provides three aspects of the inclusion/exclusion of annotations of some classes of the genia protein subcategories.
for example, for aimed and genia, item  and item  relate to the protein_molecule and protein_family_or_group annotations.
for genetag and genia, item  and item  relate to the protein_molecule, protein_domain_or_region, protein_complex and protein_subunit annotations.
however, there are other protein subcategories annotated in genia; and so far, we are unable to find any clue regarding the inclusion or exclusion of these protein subcategories in the scope of the annotations in aimed or genetag.
for instance, it is unclear whether the genetag annotations are compatible with the genia protein_family_or_group annotations or not.
it is also unclear whether the genia protein_domain_or_region, protein_complex and protein_subunit annotations are compatible with the aimed annotations.
we performed a series of experiments to confirm the three aspects that we found, and to find other clues on the other protein subclasses.
for training, we used each of the genia protein subclasses, one by one, together with the aimed corpus and the genetag corpus .
each time, we regarded the annotations from a different genia protein subclass as positive examples.
the experimental results are listed in table  <dig>
as indicated on the table, it was found to be the most harmful to combine the genia protein_family_or_group annotations together with aimed, which supports item : the mentions of protein families are annotated in genia, but not in aimed.
also, item  is confirmed by the results that the genia protein_molecule annotations least negatively affect the performance in recognizing the proteins tagged in the aimed and genetag corpora.
it also indicates that the genia protein_molecule definition is the closest to the genetag and aimed protein definitions, when compared with other genia protein subcategories.
furthermore, adding the genia protein_subunit, protein_complex and protein_domain_or_region annotations also helps to recognize the aimed and genetag annotated proteins ).
because there are only a few protein_substructure and protein_etc annotations in genia , the two protein subcategories are excluded from further consideration.
in addition to the three mentioned aspects, we also found that besides the protein_molecule annotations, the protein_family_or_group annotations were the most helpful for our recognizer to find the genetag protein annotations.
we further observe that when adding only the genia protein subcategory annotations to a pure corpus, the precision on both aimed and genetag is quite high, but the recall is very low.
this observation suggests that if we also add the other helpful protein subclass annotations into the training material, we might improve the recall, while maintaining a reasonable level of precision.
the no.
<dig> and no.
<dig> data blocks in table  <dig> show the experimental results based on this hypothesis.
the experimental results show that when we collectively use the helpful genia protein subclass annotations, the recall improves significantly, while minimizing the decrease in precision.
for ease of comparison, the preliminary experimental results mentioned in the methods section are also listed in the last two data blocks of table  <dig>
here, dna represents the genia dna_domain_or_region and rna represents the genia rna_molecule.
since our goal is to find a way to prevent the system performance degradation, we set the performance of the pure aimed or genetag corpus training as the  goal.
then, the potential  reduction rate of incompatibility can be calculated by using formula :
where re denotes the corpus incompatibility reduction rate of a given experiment, fe denotes the f-score of the given experiment, fr denotes the reference f-score of the pure source corpus training , and fb denotes the baseline f-score of the training with the pure source corpus plus the genia protein annotations.
by combing the aimed corpus with the genia protein_molecule, protein_subunit and protein_complex annotations, we reduced the corpus incompatibility by  <dig> %, based on the partial matching criterion.
thus, when combining the annotations from the genia corpus with the aimed corpus, we can use the annotations of these three protein subclasses, since they were found to be compatible to some extent.
moreover, if we combine the genetag corpus with the genia protein_molecule, protein_subunit, protein_domain_or_region, protein_family_or_group and protein_complex annotations, the corpus incompatibility reduction rate  is not as notable as the rate achieved on aimed .
subsequently, other than the five protein subcategories, there might be other category annotations that are compatible with the genetag annotations.
this situation will be explored in the following paragraphs.
we found sentences that include genia protein_subunit, protein_complex and protein_domain_or_region annotations, which will not cause an incompatibility during corpus combination.
that is, in genia, these entities are regarded as proteins that are compatible with the aimed and genetag annotations, so we can introduce most of the genia annotations of these entities into aimed or genetag without causing a negative influence.
some examples are shown .
in order to show the original corpus annotation, all of the entity annotation types are presented.
ambiguity between dna and genes
the annotations in the aimed and genetag corpora also include some gene names, without differentiating them from proteins.
for the genia corpus, the protein annotation is applied only to proteins, while genes are annotated in the scope of dna annotations.
treating gene annotations in the genia corpus in the same way as for the aimed and genetag corpora would improve the consistency; nevertheless, the genia annotation does not include explicit gene annotation.
instead, genes are annotated as instances of dna_domain_or_region, which is also applied to other dna regions .
we assume that if the dna_domain_or_region annotations that are not pure genes can be filtered out from dna_domain_or_region annotations, examples in the remaining genia dna_domain_or_region annotations will positively affect the corpus combination.
consequently, if we assume that the performance of a recognizer trained with the aimed corpus is sufficiently good, it will find most of the gene mentions in the genia corpus, based on the gene definitions in aimed.
the other dna_domain_or_region annotations that are not compatible with the aimed gene definitions will then be filtered out.
the true positives, which are annotated as dna_domain_or_region in the genia corpus and also detected by the recognizer, will include dna_domain_or_region instances, which are "aimed-like" genes.
in a similar way, we can also find the "genetag-like" gene mentions.
to examine the performance of the filtering, we added all of the dna_domain_or_region annotations into the training set in one experiment, and only added the filtered "genes" into the training set of another experiment.
the results  show the disambiguation between dna and genes works, although the improvement resulting from the filtering is modest, the improvement was relatively small for genetag.
this may be due to the fact that, in addition to genes and proteins, the genetag corpus also includes rna, domains, complexes, sequences, fusion proteins, etc.
no distinction is made between these classes, so the filtering on genetag is more difficult than filtering on aimed.
the simple filtering cannot effectively filter out most of the real genes from so many other classes; however, simple filtering helps to filter out some dna_domain_or_region annotations that are not necessary for the tagger.
in fact, even some well-known machine learning algorithms did not perform well against a human-labeled model, even though human experts could not achieve a high agreement rate on protein, gene and rna labels  <cit> .
as mentioned earlier, adding only the protein_molecule, protein_subunit and protein_complex annotations from genia resulted in the best performance for the aimed corpus thus far.
then, in addition to the three annotation types, we also added the filtered dna_domain_or_region annotations to train our protein mention recognizer.
the same experiment was performed with genetag.
the experimental results are shown in the first three rows of the no.
<dig> data block in table  <dig>  in comparison the first two rows of the no.
<dig> data block with the no.
<dig> data block in table  <dig>  we can tell that the corpus incompatibility between genia and aimed is removed completely, because our current best result on aimed  is better than the performance of training with the pure aimed corpus .
in comparing the third row of the no.
<dig> data block with the no.
<dig> data block in table  <dig>  in the case of genia and genetag, the corpus incompatibility was reduced by  <dig> %.
rna annotations in genetag
in addition to the genia annotations that have already been confirmed to be compatible with the genetag annotations, there is another genia subcategory that should contain compatible annotations: namely the rna_molecule.
in addition to the six subcategory annotations mentioned above, we have also added the genia rna_molecule annotations to the genetag corpus.
the experimental results are shown in the last row of the no.
<dig> data block in table  <dig>  by comparing the f-scores shown in the last two rows of the no.
<dig> data block in table  <dig>  we can see that the reduction of the incompatibility on genetag was improved by adding the rna_molecule annotations.
the improvements thus far are shown in the third data row of table  <dig>  after we have explored as many compatible annotations between the corpora as possible, we can observe that the f-score performance gap between the pure source corpus training, and the genia-combined training is about  <dig> % in f-score on genetag alone.
for aimed, the performance of the genia-combined training competes with that of the pure source corpus training.
incompatibility three: sentence selection
although all of the possible compatible genia annotations have already been explored, there are still some sources that are responsible for the remaining incompatibilities between genia and genetag.
when some sentences with the compatible annotations are introduced, some incompatible "missing" or "extra" annotations in these sentences may also be included.
to confirm this assumption, we applied a sentence selection policy.
for a given subcategory, sentences that only contain annotations of this subcategory are selected for our corpus integration experiment.
for instance, in the case of protein_molecule, dna_domain_or_region and rna_molecule, with the exception of these three subcategories, if there are no other protein subcategory, dna subcategory, rna subcategory or peptide annotations in a sentence, then that sentence will be combined with aimed or genetag as the training data.
the results of the experiments with the selected sentences are shown in table  <dig>
the second column only shows the number of the selected genia sentences.
as evidenced in table  <dig>  adding  <dig>  selected genia sentences with the protein_molecule, protein_subunit, protein_complex, protein_domain_or_region and the filtered dna_domain_or_region annotations, with the partial matching criterion, further obtained a better performance with the corpus integration than the results obtained by training on the pure aimed corpus.
moreover, we can see that combing the selected genia sentences with genetag also helped our protein mention recognizer work better on the genetag corpus.
the corpus incompatibility reduction rate improves to  <dig> %.
we also summarize the improvement obtained by the sentence selection strategy in the last row of table  <dig>  the incompatibility between aimed and genia has then been eliminated completely, but for genetag and genia, a very small performance degradation remains .
the cause of the remaining incompatibility between genetag and genia will be explained in the next section.
since sentence selection reduces the number of potentially incompatible examples that were added, the reduced incompatibility may be explained by the down sampling involved.
to verify that the improvements are really achieved by sentence selection, we conducted experiments with randomly selected sentences.
the same number of sentences were randomly selected from genia, as introduced in the sentence selection experiments, and are repeated  <dig>  times.
we then calculated the confidence intervals of the partial matching f-scores at the 95% confidence level.
the statistical results are shown in table  <dig>  since the f-scores of the sentence selection experiments fall outside of the range of the random sentence selection f-scores, sentence selection works on both the aimed and genetag corpora.
the experiments are as described in table  <dig>  with the randomly selected genia sentences.
incompatibility four: non-overlapping data
the results thus far indicate that incompatibility still exists among the corpora; in particular, the performance of training with genetag and genia is still a little lower than the performance achieved by using only genetag.
since the sentences in the three corpora are collected by different means, it is assumed that the proteins mentioned in the three corpora are mainly heterogeneous, which results in incompatibility.
to quantify this assumption, we examined the ratio of overlapping entities among different portions of each corpus and among different corpora.
we first divided each corpus into two disjoint portions of the same size.
in order to make the comparisons, we down-sampled sentences in each portion of genia and genetag, so that the gene/protein mentions in each portion are of the same number as the ones in one aimed portion .
the numbers of overlapping entities in and across the corpora are shown in table  <dig>  table  <dig> shows that the proteins in the genetag corpus are more heterogeneous than the proteins in genia and aimed.
in comparing the number of the intra-corpus overlapping entities with that of the inter-corpora overlapping entities, it further shows that the proteins across any two different corpora are significantly more heterogeneous than the proteins in a single corpus, which supports our assumption.
we can therefore conclude that the heterogeneity of the proteins in the three corpora is another source of incompatibility.
the incompatibility, however can be considered as a potential benefit of using all three corpora, rather than as a problem.
the low overlapping ratio of the annotated entities in the three corpora implies that no single corpus can represent the annotation for the entire protein.
then, developing a ner system that can show a good performance on all three corpora would be ideal; this will remain a task for our future work.
"genia-aimed" and "genia-genetag" represent the genia data used in exp  <dig> and exp  <dig> of table  <dig>  which are compatible with aimed and genetag, respectively.
the number of gene/protein mentions in each portion is  <dig>
CONCLUSIONS
we have presented a comparative evaluation of protein annotations, by studying the genia, genetag and aimed corpora.
our preliminary experiments showed a major performance degradation in protein name identification, when using the combined corpora as the training material .
the documentation of these corpora was studied to identify the corpus heterogeneities that caused the performance degradation; in effect, a series of experiments that aimed at removing or avoiding the negative effects of the corpus differences were performed.
we also strove towards grasping a better understanding of the different protein annotations implemented in the corpora.
although all of the corpora are distributed with gene/protein annotations, the target entities are significantly different depending on the documentation of the corpora.
the emphasis on individual proteins  involved in protein-protein interactions in the aimed corpus is responsible for more than half of the disagreements between aimed and genia.
the scope of the proteins defined for the protein annotations to genetag and genia is the biggest source of the incompatibilities found between the two corpora: besides protein, the protein definition of genetag also includes gene, dna and rna, while the protein definition of genia does not.
even within a single corpus, the boundary word annotation is not guaranteed to be consistent .
finally, the low overlapping ratio of the annotated entities between each corpus is another reason for the incompatibilities.
these observations suggest the difficulty of benefiting from an increased size in training data by merging differently annotated corpora, unless the annotations of individual collections are adequately compatible.
to address this issue, we proposed ways of avoiding the heterogeneities that we have already discovered.
we showed that the proposed methods removed the incompatibilities between aimed and genia, thus showing an improved ner performance by combining the two corpora for training.
the f-score performance improves from  <dig>  to  <dig>  based on the partial matching criterion, which makes the performance competitive with the performance of the pure aimed corpus training .
for genetag and genia, we also removed a significant amount of inconsistencies and achieved a comparable ner performance when combining the two corpora for training.
the f-score performance improves from  <dig>  to  <dig> , merely  <dig> % lower than using the pure genetag corpus training.
this may be attributed in part to the fact that the genetag corpus is quite optimized for machine learning with a balanced number of positive and negative examples, while the genia corpus is not optimized.
all improvements are proved to be statistically significant.
as the increasing amount of data helps to improve ner system performance, the creation of standard annotation guidelines proves to be an important task.
the study done through this work provides insight to the existing protein annotations, which should be helpful for producing better annotation guidelines.

 named entity recognition plays an important first role in information extraction and text mining. recently, due to the increase in biomedical research literature, biomedical ner, which aims at the automatic identification of gene and protein names, has become a research focus. human-annotated corpora are widely used in the development of bio-ner systems. there are several well-known corpora that have gene/protein mention annotations, such as genia, genetag, aimed, pennbioie, etc. based on these corpora, many protein mention recognizers have been developed, some of which are able to achieve state-of-the-art performance. nevertheless, a well-known problem remains. since the protein annotations are made by different groups, and lack an explicit, unanimous rule that defines what should be annotated, it is likely that the annotations in the different corpora are incompatible. the incompatibility among the corpora brings about several significant problems. for example, it is difficult to effectively utilize more than one corpus to develop a protein mention recognizer. there are very few high quality protein recognizers that are developed by utilizing multiple corpora, because it is unlikely for the recognizer to directly benefit from corpus integration. it is also difficult to compare systems that are developed using different corpora. although there are many systems that recognize protein mentions from the pubmed text, the reported performance varies significantly, even for instance where the systems were developed with the same method. for example, methods based on conditional random fields report f-scores of % on genia and % on genetag. if each of the two recognizers were evaluated on the other corpus, the performance would be much lower. ultimately, the problems are largely caused by the incompatibility of different protein annotations, and cannot be resolved effectively without understanding the differences in the annotations. in this paper, we explore the sources of incompatibilities among three well-known corpora with gene/protein annotations, genia, genetag and aimed, and seek solutions to overcome the incompatibilities. we first observe the reduction in performance that results from using two of the three corpora together. then, we carefully study the documentation of the three corpora, in order to determine the sources of incompatibility. through a series of experiments, we quantify the incompatibility problems, while finding reasonable strategies to avoid the problems that are caused by the incompatibility of protein annotations. this study aims at analyzing the sources of incompatibility, and reducing the corpus inconsistencies.  our research is based on the three selected corpora: genia, genetag and aimed. here, we present the findings from studying the documentation of the corpora, and from the results of our experiments. refer to the methods section for further details regarding preliminary experiments, significance tests, related works, and information on the corpora and on the protein mention recognizer used in this research. characteristic of the corpora we investigate three corpora that are often used in biomedical natural language processing. table summarizes the differences between the three corpora, which are determined by analyzing the corpora's published literature. legend: size: number of abstracts or sentences in the corpus used in this research entity scope: types of the named entities identified in the corpus: rotein, ene, na entity number: number of the annotated in-scope entities in the corpus entity coverage: coverage of in-scope entity occurrences in each sentence entity type: explicit identification of the types of the annotated in-scope entities all three corpora contain annotations that identify entities in the text, including proteins and genes; however, only genia contains information that specifies the types of the entities. this study benefits from the fine-grained protein annotation of genia. we compare the other two corpora to genia. genia aims at including an exhaustive annotation of entities of types that are relevant to the corpus, while the other two corpora have based entity annotation on a specific constraint that the tagged entity must ultimately be traceable to a specific gene/protein. in terms of text organization, compared to genetag, genia and aimed are closer to each other. genia and aimed are focused on the "human" domain, but genetag covers a more general domain of pubmed. genia and aimed exhaustively collect sentences in abstracts, but genetag collects sentences that are relevant to ner, in other words, genetag contains both true and false gene/protein names in a variety of contexts. in terms of text selection for entity annotation, genia and genetag are closer to each other, compared to aimed. the former two corpora tend to select longer text fragments as entity references. incompatibility of heterogeneous annotations these differences suggest that combining the corpora into one training data will harm the performance of protein mention recognizers. we eventually identified four main sources of incompatibilities, which thoroughly explained the performance degradation when training with the united data. we applied corresponding strategies toward each aspect of the sources, to reduce performance degradation. table sums up the improvements of minimizing each of these negative effects. the scores shown in this table are the f-scores of training with the following data: a, a+genia, g and g+genia. as expected, the f-score performance, when training with combined data, was significantly lower than when training with the pure corpora. based on the exact matching criterion, the performance is degraded about % and %, when aimed and genetag are respectively combined with the genia protein annotations as training data. the succeeding sections detail the sources of the incompatibility, and describe each of the improvements listed in table ways to avoid the negative sides of the heterogeneities are also explained in detail. incompatibility one: boundary of protein mentions the selection of text spans, e.g. the beginning and ending boundaries, to be annotated is identified as one of the major sources for the incompatibility that exists for protein annotation across the three corpora. since it is difficult to determine whether the category of an entity, which occurs before or after the entity, can be considered as part of the entity name or not, the selection of boundaries is complicated. for example, the english word "protein" can be treated differently in the following two text expressions, "p21ras protein" and "tumor suppressor protein". in the first case, "p21ras" is considered to be sufficient in naming the object. the term denoting the semantic category, "protein" is redundant, and may not be annotated as a part of the entity name. in fact, the decision to annotate "protein" in such a case often does not affect the utility of a ner system, because the system has correctly identified "p21ras" as a protein, and this information is adequate for mining the relationship between "p21ras" and another protein. similarly, "the p21ras protein" or "the p21ras" could also be considered to be acceptable. however, in the second case, without "protein", "tumor suppressor" is inadequate in denoting the protein. the actual meaning is changed when omitting "protein". the semantic category "protein", which serves as an important clue, is quite essential in distinguishing this specific entity from a more general one. thus, the inclusion of the word "protein" in the protein annotation is dependent on annotation scheme. genia and genetag almost always include the word "protein" in protein annotations, which indicates their schemes on words like "protein", but aimed excludes "protein" in most cases. the difference in boundary word selection between different protein annotation schemes can be measured in two ways: by calculating the average length of protein mentions and annotation entropy of boundary words. average length of protein mentions the ambiguous annotation of boundary words is also a factor that affects the average length of the protein mentions in the three corpora. the average length per protein mention is tokens in the aimed corpus, tokens in the genia corpus, and tokens in the genetag corpus. many long protein mentions are introduced when we add the genia annotations into aimed, creating another possible source of performance degradation in recognizing the shorter protein mentions in the aimed corpus. this observation suggests that the genia and genetag corpora are inclined to select more descriptive expressions for the protein annotation, in comparison with aimed. annotation entropy of boundary words in a given corpus, some words are annotated inside of protein mentions, while other words are not. the annotation entropy of boundary words is calculated by formula. where eb denotes the annotation entropy of a given word, pa denotes the percentage of the occurrences of this word that is annotated, and denotes the percentage of the occurrences of this word that is not annotated. for the sake of brevity, the "word" discussed in this section describes the word that appears at the beginning or end of an annotated entity, or describes the word that abuts an annotated entity. the value of the annotation entropy of a boundary word eb ranges from to. when the annotation entropy of a boundary word eb is this word is perfectly annotated, and keeps this annotation consistency throughout the entirety of the corpus. on the contrary, when the annotation entropy of a boundary word eb is the word's annotation is so disorderly, that we can hardly find any rules about whether to regard this word as a part of the protein mentions or not. in general, there are two types of ambiguous boundary words: descriptive adjectives, which usually occur before protein names as modifiers, such as "normal", "activated" or "human", and nouns, which usually occur after protein names as heads. some boundary words appearing in each corpus are listed in table in order to characterize the differences among the three corpora in terms of the annotation entropy of boundary words, the words with an annotation entropy close to were included in table the genia tagger was used to determine the parts-of-speech of the words. for the entire boundary words list, see "additional file 1: the boundary words of the genia, genetag and aimed corpora". here, "noun before" indicates the noun occurring before an entity as a modifier, "noun after" indicates the noun occurring after an entity as a head. na represents the number of the annotated occurrences, and nn represents the number of un-annotated occurrences. from table the boundary annotation problem appears for various words. the distribution of these words is diverse, particularly for adjectives. since the number of characters tagged in the aimed corpus was kept to a minimum, only the names of protein mentions were annotated, most of which are proper nouns, and most of the adjectives were not annotated. however, the developers of the genia corpus followed an annotation strategy for which generic terms were annotated. in the genia annotation scheme, specifiers do not appear in tagged entities, but qualifiers, including adjectives and noun modifiers, remain. the adjectives before the protein mentions are annotated only if they are required for clarifying the meaning of the protein mentions. furthermore, the genetag annotators chose some semantic constraints, which state that the tagged entity must contain its true meaning in the sentence context. these constraints were geared towards multi-word entities, and especially for entities that include numbers, letters and acronyms. because the conventions on boundary words vary significantly, to prevent underestimating performance of protein mention recognizers, we need an alternative matching criterion, other than the exact matching. to provide alternative evaluation perspectives, researchers have developed a variety of evaluation criteria that loosens the matching to varying degrees. partial matching is also considered, because it has previously been shown to capture the presence of an entity with disregards to its exact textual representation. according to this partial matching criterion, the performance for the pure and combined data are shown in the second data row of table though by loosening the matching criterion, the performance of the training with the combined data still can not compete with the performance of the training with the pure data, under-estimation can be avoided. incompatibility two: scope of the entities of interest although all three corpora include gene/protein mention annotations, the target tasks are different. the genia corpus aims at providing linguistically rich annotations on biological expressions. the genetag annotation follows a wide definition with the constraint that a gene/protein entity annotation must refer to a specific entity. the aimed annotation focuses on extracting interactions among individual proteins. this difference has affected the scope of the annotated proteins: genia is concerned with all of the protein-mentioning terms, genetag is based on specific genes/proteins, while aimed focuses only on references from individual proteins. categories of annotated entities the extent to which the proteins annotated in the genia corpus is defined in the genia ontology. in addition to the protein class, other classes such as dna, rna, cell_line and cell_type are also included. further, the protein class is categorized into seven subclasses: protein_complex, protein_domain_or_region, protein_family_or_group, protein_molecule, protein_substructure, protein_subunit and protein_etc. thus, in genia, "protein" is defined to include these seven concepts. for aimed, the scope of the proteins annotated is described by the following statement in the tagging guidelines: generic protein families are not tagged, only specific names that could ultimately be traced back to specific genes in the human genome are tagged. that is, for example, "tumor necrosis factor" would not be tagged, while "tumor necrosis factor alpha" would be tagged. finally, based on the gene names from genbank, the genetag annotations include domains, complexes, subunits, and promoters when the annotated entities refer to specific genes/proteins. hence, for the scope of annotated proteins, the documentation of the three corpora explicitly states that: the mentions of protein families are annotated in genia, but not in aimed. individual proteins are annotated in all of the corpora. both genia and genetag contain protein domain, complex and subunit annotations. compatible protein annotations the published literature of the three corpora provides three aspects of the inclusion/exclusion of annotations of some classes of the genia protein subcategories. for example, for aimed and genia, item and item relate to the protein_molecule and protein_family_or_group annotations. for genetag and genia, item and item relate to the protein_molecule, protein_domain_or_region, protein_complex and protein_subunit annotations. however, there are other protein subcategories annotated in genia; and so far, we are unable to find any clue regarding the inclusion or exclusion of these protein subcategories in the scope of the annotations in aimed or genetag. for instance, it is unclear whether the genetag annotations are compatible with the genia protein_family_or_group annotations or not. it is also unclear whether the genia protein_domain_or_region, protein_complex and protein_subunit annotations are compatible with the aimed annotations. we performed a series of experiments to confirm the three aspects that we found, and to find other clues on the other protein subclasses. for training, we used each of the genia protein subclasses, one by one, together with the aimed corpus and the genetag corpus. each time, we regarded the annotations from a different genia protein subclass as positive examples. the experimental results are listed in table as indicated on the table, it was found to be the most harmful to combine the genia protein_family_or_group annotations together with aimed, which supports item: the mentions of protein families are annotated in genia, but not in aimed. also, item is confirmed by the results that the genia protein_molecule annotations least negatively affect the performance in recognizing the proteins tagged in the aimed and genetag corpora. it also indicates that the genia protein_molecule definition is the closest to the genetag and aimed protein definitions, when compared with other genia protein subcategories. furthermore, adding the genia protein_subunit, protein_complex and protein_domain_or_region annotations also helps to recognize the aimed and genetag annotated proteins ). because there are only a few protein_substructure and protein_etc annotations in genia, the two protein subcategories are excluded from further consideration. in addition to the three mentioned aspects, we also found that besides the protein_molecule annotations, the protein_family_or_group annotations were the most helpful for our recognizer to find the genetag protein annotations. we further observe that when adding only the genia protein subcategory annotations to a pure corpus, the precision on both aimed and genetag is quite high, but the recall is very low. this observation suggests that if we also add the other helpful protein subclass annotations into the training material, we might improve the recall, while maintaining a reasonable level of precision. the no. and no. data blocks in table show the experimental results based on this hypothesis. the experimental results show that when we collectively use the helpful genia protein subclass annotations, the recall improves significantly, while minimizing the decrease in precision. for ease of comparison, the preliminary experimental results mentioned in the methods section are also listed in the last two data blocks of table here, dna represents the genia dna_domain_or_region and rna represents the genia rna_molecule. since our goal is to find a way to prevent the system performance degradation, we set the performance of the pure aimed or genetag corpus training as the goal. then, the potential reduction rate of incompatibility can be calculated by using formula: where re denotes the corpus incompatibility reduction rate of a given experiment, fe denotes the f-score of the given experiment, fr denotes the reference f-score of the pure source corpus training, and fb denotes the baseline f-score of the training with the pure source corpus plus the genia protein annotations. by combing the aimed corpus with the genia protein_molecule, protein_subunit and protein_complex annotations, we reduced the corpus incompatibility by %, based on the partial matching criterion. thus, when combining the annotations from the genia corpus with the aimed corpus, we can use the annotations of these three protein subclasses, since they were found to be compatible to some extent. moreover, if we combine the genetag corpus with the genia protein_molecule, protein_subunit, protein_domain_or_region, protein_family_or_group and protein_complex annotations, the corpus incompatibility reduction rate is not as notable as the rate achieved on aimed. subsequently, other than the five protein subcategories, there might be other category annotations that are compatible with the genetag annotations. this situation will be explored in the following paragraphs. we found sentences that include genia protein_subunit, protein_complex and protein_domain_or_region annotations, which will not cause an incompatibility during corpus combination. that is, in genia, these entities are regarded as proteins that are compatible with the aimed and genetag annotations, so we can introduce most of the genia annotations of these entities into aimed or genetag without causing a negative influence. some examples are shown. in order to show the original corpus annotation, all of the entity annotation types are presented. ambiguity between dna and genes the annotations in the aimed and genetag corpora also include some gene names, without differentiating them from proteins. for the genia corpus, the protein annotation is applied only to proteins, while genes are annotated in the scope of dna annotations. treating gene annotations in the genia corpus in the same way as for the aimed and genetag corpora would improve the consistency; nevertheless, the genia annotation does not include explicit gene annotation. instead, genes are annotated as instances of dna_domain_or_region, which is also applied to other dna regions. we assume that if the dna_domain_or_region annotations that are not pure genes can be filtered out from dna_domain_or_region annotations, examples in the remaining genia dna_domain_or_region annotations will positively affect the corpus combination. consequently, if we assume that the performance of a recognizer trained with the aimed corpus is sufficiently good, it will find most of the gene mentions in the genia corpus, based on the gene definitions in aimed. the other dna_domain_or_region annotations that are not compatible with the aimed gene definitions will then be filtered out. the true positives, which are annotated as dna_domain_or_region in the genia corpus and also detected by the recognizer, will include dna_domain_or_region instances, which are "aimed-like" genes. in a similar way, we can also find the "genetag-like" gene mentions. to examine the performance of the filtering, we added all of the dna_domain_or_region annotations into the training set in one experiment, and only added the filtered "genes" into the training set of another experiment. the results show the disambiguation between dna and genes works, although the improvement resulting from the filtering is modest, the improvement was relatively small for genetag. this may be due to the fact that, in addition to genes and proteins, the genetag corpus also includes rna, domains, complexes, sequences, fusion proteins, etc. no distinction is made between these classes, so the filtering on genetag is more difficult than filtering on aimed. the simple filtering cannot effectively filter out most of the real genes from so many other classes; however, simple filtering helps to filter out some dna_domain_or_region annotations that are not necessary for the tagger. in fact, even some well-known machine learning algorithms did not perform well against a human-labeled model, even though human experts could not achieve a high agreement rate on protein, gene and rna labels. as mentioned earlier, adding only the protein_molecule, protein_subunit and protein_complex annotations from genia resulted in the best performance for the aimed corpus thus far. then, in addition to the three annotation types, we also added the filtered dna_domain_or_region annotations to train our protein mention recognizer. the same experiment was performed with genetag. the experimental results are shown in the first three rows of the no. data block in table in comparison the first two rows of the no. data block with the no. data block in table we can tell that the corpus incompatibility between genia and aimed is removed completely, because our current best result on aimed is better than the performance of training with the pure aimed corpus. in comparing the third row of the no. data block with the no. data block in table in the case of genia and genetag, the corpus incompatibility was reduced by %. rna annotations in genetag in addition to the genia annotations that have already been confirmed to be compatible with the genetag annotations, there is another genia subcategory that should contain compatible annotations: namely the rna_molecule. in addition to the six subcategory annotations mentioned above, we have also added the genia rna_molecule annotations to the genetag corpus. the experimental results are shown in the last row of the no. data block in table by comparing the f-scores shown in the last two rows of the no. data block in table we can see that the reduction of the incompatibility on genetag was improved by adding the rna_molecule annotations. the improvements thus far are shown in the third data row of table after we have explored as many compatible annotations between the corpora as possible, we can observe that the f-score performance gap between the pure source corpus training, and the genia-combined training is about % in f-score on genetag alone. for aimed, the performance of the genia-combined training competes with that of the pure source corpus training. incompatibility three: sentence selection although all of the possible compatible genia annotations have already been explored, there are still some sources that are responsible for the remaining incompatibilities between genia and genetag. when some sentences with the compatible annotations are introduced, some incompatible "missing" or "extra" annotations in these sentences may also be included. to confirm this assumption, we applied a sentence selection policy. for a given subcategory, sentences that only contain annotations of this subcategory are selected for our corpus integration experiment. for instance, in the case of protein_molecule, dna_domain_or_region and rna_molecule, with the exception of these three subcategories, if there are no other protein subcategory, dna subcategory, rna subcategory or peptide annotations in a sentence, then that sentence will be combined with aimed or genetag as the training data. the results of the experiments with the selected sentences are shown in table the second column only shows the number of the selected genia sentences. as evidenced in table adding selected genia sentences with the protein_molecule, protein_subunit, protein_complex, protein_domain_or_region and the filtered dna_domain_or_region annotations, with the partial matching criterion, further obtained a better performance with the corpus integration than the results obtained by training on the pure aimed corpus. moreover, we can see that combing the selected genia sentences with genetag also helped our protein mention recognizer work better on the genetag corpus. the corpus incompatibility reduction rate improves to %. we also summarize the improvement obtained by the sentence selection strategy in the last row of table the incompatibility between aimed and genia has then been eliminated completely, but for genetag and genia, a very small performance degradation remains. the cause of the remaining incompatibility between genetag and genia will be explained in the next section. since sentence selection reduces the number of potentially incompatible examples that were added, the reduced incompatibility may be explained by the down sampling involved. to verify that the improvements are really achieved by sentence selection, we conducted experiments with randomly selected sentences. the same number of sentences were randomly selected from genia, as introduced in the sentence selection experiments, and are repeated times. we then calculated the confidence intervals of the partial matching f-scores at the 95% confidence level. the statistical results are shown in table since the f-scores of the sentence selection experiments fall outside of the range of the random sentence selection f-scores, sentence selection works on both the aimed and genetag corpora. the experiments are as described in table with the randomly selected genia sentences. incompatibility four: non-overlapping data the results thus far indicate that incompatibility still exists among the corpora; in particular, the performance of training with genetag and genia is still a little lower than the performance achieved by using only genetag. since the sentences in the three corpora are collected by different means, it is assumed that the proteins mentioned in the three corpora are mainly heterogeneous, which results in incompatibility. to quantify this assumption, we examined the ratio of overlapping entities among different portions of each corpus and among different corpora. we first divided each corpus into two disjoint portions of the same size. in order to make the comparisons, we down-sampled sentences in each portion of genia and genetag, so that the gene/protein mentions in each portion are of the same number as the ones in one aimed portion. the numbers of overlapping entities in and across the corpora are shown in table table shows that the proteins in the genetag corpus are more heterogeneous than the proteins in genia and aimed. in comparing the number of the intra-corpus overlapping entities with that of the inter-corpora overlapping entities, it further shows that the proteins across any two different corpora are significantly more heterogeneous than the proteins in a single corpus, which supports our assumption. we can therefore conclude that the heterogeneity of the proteins in the three corpora is another source of incompatibility. the incompatibility, however can be considered as a potential benefit of using all three corpora, rather than as a problem. the low overlapping ratio of the annotated entities in the three corpora implies that no single corpus can represent the annotation for the entire protein. then, developing a ner system that can show a good performance on all three corpora would be ideal; this will remain a task for our future work. "genia-aimed" and "genia-genetag" represent the genia data used in exp and exp of table which are compatible with aimed and genetag, respectively. the number of gene/protein mentions in each portion is  we have presented a comparative evaluation of protein annotations, by studying the genia, genetag and aimed corpora. our preliminary experiments showed a major performance degradation in protein name identification, when using the combined corpora as the training material. the documentation of these corpora was studied to identify the corpus heterogeneities that caused the performance degradation; in effect, a series of experiments that aimed at removing or avoiding the negative effects of the corpus differences were performed. we also strove towards grasping a better understanding of the different protein annotations implemented in the corpora. although all of the corpora are distributed with gene/protein annotations, the target entities are significantly different depending on the documentation of the corpora. the emphasis on individual proteins involved in protein-protein interactions in the aimed corpus is responsible for more than half of the disagreements between aimed and genia. the scope of the proteins defined for the protein annotations to genetag and genia is the biggest source of the incompatibilities found between the two corpora: besides protein, the protein definition of genetag also includes gene, dna and rna, while the protein definition of genia does not. even within a single corpus, the boundary word annotation is not guaranteed to be consistent. finally, the low overlapping ratio of the annotated entities between each corpus is another reason for the incompatibilities. these observations suggest the difficulty of benefiting from an increased size in training data by merging differently annotated corpora, unless the annotations of individual collections are adequately compatible. to address this issue, we proposed ways of avoiding the heterogeneities that we have already discovered. we showed that the proposed methods removed the incompatibilities between aimed and genia, thus showing an improved ner performance by combining the two corpora for training. the f-score performance improves from to based on the partial matching criterion, which makes the performance competitive with the performance of the pure aimed corpus training. for genetag and genia, we also removed a significant amount of inconsistencies and achieved a comparable ner performance when combining the two corpora for training. the f-score performance improves from to, merely % lower than using the pure genetag corpus training. this may be attributed in part to the fact that the genetag corpus is quite optimized for machine learning with a balanced number of positive and negative examples, while the genia corpus is not optimized. all improvements are proved to be statistically significant. as the increasing amount of data helps to improve ner system performance, the creation of standard annotation guidelines proves to be an important task. the study done through this work provides insight to the existing protein annotations, which should be helpful for producing better annotation guidelines. 
however, in the biomedical community, there is yet no general consensus regarding named entity annotation; thus, the resources are largely incompatible, and it is difficult to compare the performance of systems developed on resources that were divergently annotated. thus, it becomes a task of interest to integrate the heterogeneous annotations in these resources. this degradation may be caused by the newly added heterogeneous annotations, and cannot be fixed without an understanding of the heterogeneities that exist among the corpora. we find that the f-score performance declines tremendously when training with integrated data, instead of training with pure data; in some cases, the performance drops nearly 12%. we explore the potential sources of incompatibility among gene and protein annotations that were made for three common corpora: genia, genetag and aimed. our analyses indicate that incompatibilities in the gene/protein annotations exist mainly in the following four areas: the boundary annotation conventions, the scope of the entities of interest, the distribution of annotated entities, and the ratio of overlap between annotated entities. to show the inconsistency in the corpora annotations, we first tackle the incompatibility problem caused by corpus integration, and we quantitatively measure the effect of this incompatibility on protein mention recognition.
 the number of corpora, collections of structured texts, has been increasing, as a result of the growing interest in the application of natural language processing methods to biological texts. many named entity recognition systems have been developed based on these corpora. however, in the biomedical community, there is yet no general consensus regarding named entity annotation; thus, the resources are largely incompatible, and it is difficult to compare the performance of systems developed on resources that were divergently annotated. on the other hand, from a practical application perspective, it is desirable to utilize as many existing annotated resources as possible, because annotation is costly. thus, it becomes a task of interest to integrate the heterogeneous annotations in these resources.  we explore the potential sources of incompatibility among gene and protein annotations that were made for three common corpora: genia, genetag and aimed. to show the inconsistency in the corpora annotations, we first tackle the incompatibility problem caused by corpus integration, and we quantitatively measure the effect of this incompatibility on protein mention recognition. we find that the f-score performance declines tremendously when training with integrated data, instead of training with pure data; in some cases, the performance drops nearly 12%. this degradation may be caused by the newly added heterogeneous annotations, and cannot be fixed without an understanding of the heterogeneities that exist among the corpora. motivated by the result of this preliminary experiment, we further qualitatively analyze a number of possible sources for these differences, and investigate the factors that would explain the inconsistencies, by performing a series of well-designed experiments. our analyses indicate that incompatibilities in the gene/protein annotations exist mainly in the following four areas: the boundary annotation conventions, the scope of the entities of interest, the distribution of annotated entities, and the ratio of overlap between annotated entities. we further suggest that almost all of the incompatibilities can be prevented by properly considering the four aspects aforementioned.  our analysis covers the key similarities and dissimilarities that exist among the diverse gene/protein corpora. this paper serves to improve our understanding of the differences in the three studied corpora, which can then lead to a better understanding of the performance of protein recognizers that are based on the corpora. 