BACKGROUND
the exponential growth of the amount of scientific literature in the fields of biomedicine and genomics has made it increasingly difficult for scientists to keep up with the state of the art.
the txm project  <cit> , a three-year project which aims to produce software tools to aid curation of biomedical papers, targets this problem and exploits natural language processing  technology in an attempt to automatically extract enriched protein-protein interactions  and tissue expressions  from biomedical text.
a critical task in txm is term identification , the task of grounding mentions of biomedical named entities to identifiers in referent databases.
ti can be seen as an intermediate task that builds on the previous component in an information extraction  pipeline, i.e., named entity recognition , and provides crucial information as input to the more complex module of relation extraction .
the structure of the ie pipeline resembles a typical curation process by human biologists.
for example, when curating protein-protein interactions , a curator would first mark up the protein mentions in text, and then identify the mentions by finding their unique identifiers from standard protein databases such as refseq  <cit> , and finally curate pairs of ids as ppis.
ti is a matching and disambiguation process  <cit> , and a primary source of ambiguity lies in the model organisms of the terms.
in curation tasks, one often needs to deal with collections of articles that involve entities of a large variety of species.
for example, our collection of articles from pubmed and pubmed central involve over  <dig> model organisms.
also, it is often the case that more than one species appear in the same document, especially when the document is a full-length article.
in our dataset, 74% of the articles concern more than one organism.
in many standard databases, such as refseq and swissprot, homolog proteins in different species, which often contain nearly identical synonym lists, are assigned distinct identifiers.
this makes biomedical terms even more polysemous and hence species disambiguation becomes crucial to ti.
for example, querying refseq with the protein mention plk <dig> resulted in  <dig> hits.
by adding a species to the query, e.g.
mouse, one can significantly reduce the number of results to two.
the most relevant work to ours are the gene normalisation  tasks  <cit>  in the biocreative i & ii workshops  <cit> .
the data provided in the gn tasks, however, were species-specific, which means that the lexicons and datasets were concerned with single model organisms and thus species disambiguation was not required.
a few participating systems, however, integrated a filter to rule out entities with erroneous species  <cit> , which were reported to be helpful.
another difference between our task and the biocreative gn ones is that we carry out ti on entity level while gn on document level.
it is worth mentioning that the protein-protein interaction task  in biocreative ii has taken into account species ambiguity.
the ips task resembles the work-flow of manual curation of ppis in articles involving multiple species, and to accomplish the task, one would require a full pipeline of ie systems, including named entity recognition, term identification and relation extraction.
the best result for ips <cit>  was fairly low at  <dig> % f <dig>  which reflects the difficulty of the task.
some participants of ips have reported  that resolving species ambiguity was one of the biggest challenges.
our analysis of the ips training data revealed that the interacting proteins in this corpus belong to over  <dig> species, and only  <dig> % of them are human.
as noted in previous work  <cit> , determining the correct species for the protein mentions is a very important step towards ti.
however, as far as we know, there has been little work in species disambiguation and in to what extent resolving species ambiguity at an entity level can help ti.
RESULTS
species disambiguation
the species tagger was developed on the iti txm corpora  <cit> , which were produced as part of the txm project  <cit> .
we created two corpora in slightly different domains, eppi and te.
the eppi corpus consists of  <dig> full-text papers selected from pubmed and pubmed central and domain experts annotated all documents for both protein entities and ppis, as well as extra  information associated with the ppis and normalisations of the proteins to publicly available ontologies.
the te corpus consists of  <dig> full-text papers, in which entities such as proteins, tissues, genes and mrnacdnas were identified, and a new tissue expression relation was marked up.
we used these corpora to develop a species tagging system.
the biomedical entities in the data were manually assigned with standard database identifiers, where genes were assigned with entrezgene ids, and proteins and mrnacdnas with refseq ids.
hence it was straightforward to obtain their species ids through the mappings provided by entrezgene and refseq.
in more detail, proteins, protein complexes, genes and mrnacdnas in both eppi and te datasets were assigned with ncbi taxonomy ids   <cit> , to denote their species.
the eppi and te datasets have different distributions of species.
for example, the entities in the eppi training data belong to  <dig> species with human being the most frequent at  <dig> %, and those in the te training set are across  <dig> species and mouse was the most frequent at  <dig> %.
to calculate the inter-annotator-agreement, about 40% of the documents were doubly annotated by different annotators.
the averaged f <dig> scores of species annotation on the doubly annotated eppi and te datasets are  <dig> % and  <dig> %, respectively, indicating that human annotators have high agreement when assigning species to biomedical entities.
to assess how much species ambiguity accounts for the overall ambiguity in biomedical entities, we estimated the averaged ambiguity rates for the protein entities in the txm datasets, without and with the species information.
suppose there are n unique protein mentions in a dataset.
first, we look up the refseq database by exact match with every unique protein mention mi, where i ∈ { <dig> .n - 1}, and for each mi we retrieve two lists of identifiers: li and l′i, where li consists of all identifiers and l′i only contains the identifiers whose model organism matches the manually tagged species of the protein mention.
the ambiguity rates without and with species are computed by ∑i=0n−1|li|n and ∑i=0n−1|l′i|n, respectively.
table  <dig> shows the ambiguity rates on the eppi and te datasets.
using the iti txm corpora, we first devised a number of rule-based species disambiguation systems.
it is intuitive that a species word that occurs near an entity  is a strong indicator of its species.
to assess this intuition, we developed a set of rules using heuristics and the species words detected by a species word tagger .
• prewd: if the word preceding an entity is a species word, assign the species indicated by that word to the entity.
• prewd sent: if a species word occurs to the left of an entity and in the same sentence, assign the species indicated by that word to the entity.
• prefix: if an entity has a species-indicating prefix, e.g., msos- <dig>  then tag the species to that entity.
• spread: spread the species of an entity e to all entities in the same document that have the same surface form with e. this rule must be used in conjunction with the other rules.
• majority vote: count the species words in a document and assign as a weight to each species the proportion of all species words in the document that refer to the species.
for example, if there are n species words in a document and nhuman are associated with human, the human species weight is calculated as nhumann.
tag all entities in the document the species with the highest weight, defaulting to human in the case of a tie.
this rule was used by default in the rule-based ti system, described later in this paper.
it is slightly counter-intuitive that using a rule such as 'prewd' did not achieve perfect precision.
closer inspection revealed that most of the false positives were due to a few problematic guidelines in the annotation process.
for example,
• "the amounts of human and mouse cd200r ...", where 'cd200r' was tagged as mouse  by the system but the gold-standard answer was human .
this was due to the fact that the annotation tool was not able to assign multiple correct species to a single entity.
• "... wheat eifiso4g ...", where 'eifiso4g' was tagged as wheat  but the annotator thought it was triticum .
in this case, taxid  <dig> is a species under genus  <dig>  and arguably is also a correct answer.
other similar cases include xenopus vs. xenopus tropicalis, and rattus vs. rattus norvegicus, etc.
this is the main cause for the false positives as our system always predicts species instead of genus or taxids of any other ranks, which the annotators occasionally employed.
furthermore, we split the eppi and te datasets into training and development test  sets and developed a machine-learning  based species tagger.
using the training splits, we trained a maximum entropy classifier  <cit>  on a number of features such as contextual words and nearly species words, which will be detailed later.
the results of the ml species tagger are shown in table  <dig>  we measure the performance in accuracy instead of f <dig> because the ml based tagger assigns a species tag to every entity occurrence, and therefore precision is equal to recall.
we tested four models on the devtest portions of the eppi and te corpora:
• bl: a baseline system, which tags the devtest instances using the most frequent species occurring in the corresponding training dataset.
for example, human is the most frequent species in the eppi training data, and therefore all entities in the eppi devtest dataset were tagged with human.
• eppi model: obtained by training the maxent classifier on the eppi training data.
• te model: obtained by training the maxent classifier on the te training data.
• combined model: obtained by training the maxent classifier on a joint dataset consisting of both the eppi and te training corpora.
finally, we devised a hybrid species tagging system.
as we have shown, the rules 'prewd' and 'prefix' achieved very good precision but low recall, which suggests that when these rules were applicable, it is highly likely that they would get the correct species.
based on this observation, we combined the ml approach and the rule-based approach in such a way that the rules 'prewd' and 'prefix' were applied on top of ml and to override predictions made by ml.
the hybrid systems were tested on the same datasets and the results are shown in the right  <dig> columns in table  <dig>  we performed significance tests on the results in table  <dig>  first, a friedman test was used to determine whether the  <dig> sets of results were significantly different, and then pairwise wilcoxon signed rank tests were employed to tell whether any system performed significantly better than others.
on both datasets, the  <dig> machine-learning models significantly outperformed the baseline .
on eppi devtest dataset, the eppi models  and the combined models outperformed the te models , while on te dataset, the te models and the combined models outperformed the eppi models .
applying the post filtering rules did not significantly improve the ml models, although it appears that adding the rules consistently increased the accuracy.
term identification with species disambiguation
experiments on the iti txm corpora
to identify whether species disambiguation can improve performance of ti, we ran the ti system on the eppi and te datasets in the iti txm corpora.
we tested the ti systems with or without help from a number of species tagging systems, including:
• baseline: run ti without species tags.
note that the ti system already integrated a basic species tagging system that uses the majority vote rule.
thus this is a fairly high 'baseline'.
• gold species: run ti with manually tagged species.
this is the upper-bound performance.
• rule: run ti with species predicted by the rule-based species tagger using rules "prewd" and "prefix".
• ml: run ti with the species that occurs most frequently in the training datasets .
• ml: run ti with species predicted by the ml tagger trained on the eppi training dataset.
• ml+rule: run ti with species predicted by the hybrid system using both ml and the rules.
• ml: run ti with species predicted by the ml tagger trained on the te training dataset.
• ml+rule: run ti with species predicted by the hybrid system using both ml and the rules.
• ml: run ti with species predicted by the ml tagger trained on both eppi and te training data.
• ml+rule: run ti with species predicted by the hybrid system using both ml and the rules.
we score the systems using top n precision, where n ∈ { <dig>   <dig>   <dig>   <dig>  20}.
the argument for this evaluation scheme is that if a ti system is not good enough in predicting a single identifier correctly, a 'bag' of ids with the correct answer included would also be helpful.
the 'avg.
rank' field denotes the averaged position where the correct answer lies in, and the lower the value is, the better the ti system performs.
for example, a ti system with an 'avg.
rank' of  <dig> would be ideal, as it would always return the correct id at the top of the list.
note that in the te data, not only protein entities, but also genes, mrnacdna, and gomops were tagged, where a gomop denotes an entity being either a gene, or an mrnacdna, or a protein.
as shown in tables  <dig> and  <dig>  on both datasets, using the gold standard species much improved accuracy of ti .
also, automatically predicted species tags were proven to be helpful.
on the eppi data, the ml+rule outperformed other systems.
note that the species distribution in the devtest dataset is strongly biased to human, which explains why the ml system performed nearly as well.
however, defaulting to human was not guaranteed to succeed because one would not be able to know the prior species in a collection of unseen documents.
indeed, on the te data, the system ml, which uses mouse as default, yielded poor results.
experiments on biocreative data
to assess the portability of the species tagging approaches, an "artificial" dataset was created by joining the species-specific datasets from biocreative  <dig> &  <dig> gn tasks to form a corpus consisting of four species.
in detail, four datasets were taken, three from biocreative  <dig> task 1b  and one from biocreative  <dig> task gn .
assuming genes in each dataset are species-specific, we can train/test ml models for species disambiguation and apply them to help ti.
this task is more difficult than the original biocreative gn tasks due to the additional ambiguity caused by multiple model organisms.
note that the above assumption is not strictly true because each dataset may contain genes of other species, and it would be hard to assess how true it is as abstracts in the biocreative gn datasets are not normalised to an entity level.
we first carried out experiments on species disambiguation.
in addition to the txm +rule model) and the majority vote taggers, we trained the species tagger on a dataset comprising of the devtest sets from the biocreative i & ii gn tasks.
in more detail, we first pre-processed the dataset and marked up gene entities with an ner system  <cit> , which was trained on biocreative ii gm training and test datasets.
the entities were tagged with the species as indicated by the source dataset where they were drawn from, which were used as the 'gold' species.
using the same algorithm and feature set as described previously, a bc model was trained.
as shown in table  <dig>  except on human, the txm model yielded very disappointing results, whereas the bc model did well overall.
this was because the txm model was trained on a dataset where fly and yeast entities occur rarely with only 2% and 5% of the training instances belonging to these species, respectively, which again revealed the influence of the bias introduced in the training material to the ml models.
using the species disambiguation models, we carried out ti experiments, using the same procedure as we did on the txm data.
the results were obtained using the official biocreative gn scorers and are presented in table  <dig>  performance of ti assisted by all three species taggers were much behind that of ti using the gold-standard species, which shows species-tagging can potentially enhance ti performance and there is much room for improving the species disambiguation systems.
on the other hand, it was disappointing that the 'majority vote' system, which did not use any external species tagger, achieved the best results, while ti with the 'bc model' tagger yielded slightly worse results and the txm model performed poorly.
one possible reason that the 'majority vote' tagger yielded reasonably good result on the biocreative dataset, but unsatisfactory result on the txm datasets was due to the difference in document length in the two corpora: the biocreative corpus is comprised of abstracts and the txm corpora consist of only full-length articles.
in abstracts, authors are inclined to only talk about the main biomedical entities described in the paper, whereas in full articles, they tend to describe a larger number of entities, possibly in multiple species, for the purposes of describing related work or comparison.
recall that the 'majority vote' rule outputs the species indicated by the majority of the species words, which would obviously perform better on abstracts, where more likely only one species is described, than on full-length articles.
table  <dig> shows the number of species per document in the txm data, where most documents  involve more than one species, in which cases the 'majority vote' would not be able to take obvious advantage.
CONCLUSIONS
we have presented a range of solutions to the task of species disambiguation and evaluated their performance on the iti txm corpus, and on a joint dataset from biocreative i & ii gn tasks.
we showed that rule-based species tagging systems that exploit heuristics, such as previous species words or species prefix, can achieve very high precision but low recall.
ml species taggers, on the other hand, can achieve good overall performance, under the condition that the species distributions in training and test datasets are not too distant.
our best performing species tagger is a hybrid system that first uses ml to predict species and then applies certain rules to correct errors.
we also performed ti experiments with help from the species tags assigned by human annotators, or predicted by the automatic species taggers.
on all datasets, the gold-standard species tags much improved ti performance:  <dig> % on the eppi devtest set,  <dig> % on the te devtest set, and  <dig> % on the biocreative gn test datasets, which clearly shows that species information is indeed very important for ti.
on the eppi and te datasets, the species predicted by the best-performing hybrid system improved ti by  <dig> % and  <dig> %, respectively.
on the combined dataset from biocreative gn tasks, however, it did not work as well as expected.

 the exponential growth of the amount of scientific literature in the fields of biomedicine and genomics has made it increasingly difficult for scientists to keep up with the state of the art. the txm project, a three-year project which aims to produce software tools to aid curation of biomedical papers, targets this problem and exploits natural language processing technology in an attempt to automatically extract enriched protein-protein interactions and tissue expressions from biomedical text. a critical task in txm is term identification, the task of grounding mentions of biomedical named entities to identifiers in referent databases. ti can be seen as an intermediate task that builds on the previous component in an information extraction pipeline, i.e., named entity recognition, and provides crucial information as input to the more complex module of relation extraction. the structure of the ie pipeline resembles a typical curation process by human biologists. for example, when curating protein-protein interactions, a curator would first mark up the protein mentions in text, and then identify the mentions by finding their unique identifiers from standard protein databases such as refseq, and finally curate pairs of ids as ppis. ti is a matching and disambiguation process, and a primary source of ambiguity lies in the model organisms of the terms. in curation tasks, one often needs to deal with collections of articles that involve entities of a large variety of species. for example, our collection of articles from pubmed and pubmed central involve over model organisms. also, it is often the case that more than one species appear in the same document, especially when the document is a full-length article. in our dataset, 74% of the articles concern more than one organism. in many standard databases, such as refseq and swissprot, homolog proteins in different species, which often contain nearly identical synonym lists, are assigned distinct identifiers. this makes biomedical terms even more polysemous and hence species disambiguation becomes crucial to ti. for example, querying refseq with the protein mention plk resulted in hits. by adding a species to the query, e.g. mouse, one can significantly reduce the number of results to two. the most relevant work to ours are the gene normalisation tasks in the biocreative i & ii workshops. the data provided in the gn tasks, however, were species-specific, which means that the lexicons and datasets were concerned with single model organisms and thus species disambiguation was not required. a few participating systems, however, integrated a filter to rule out entities with erroneous species, which were reported to be helpful. another difference between our task and the biocreative gn ones is that we carry out ti on entity level while gn on document level. it is worth mentioning that the protein-protein interaction task in biocreative ii has taken into account species ambiguity. the ips task resembles the work-flow of manual curation of ppis in articles involving multiple species, and to accomplish the task, one would require a full pipeline of ie systems, including named entity recognition, term identification and relation extraction. the best result for ips was fairly low at % f which reflects the difficulty of the task. some participants of ips have reported that resolving species ambiguity was one of the biggest challenges. our analysis of the ips training data revealed that the interacting proteins in this corpus belong to over species, and only % of them are human. as noted in previous work, determining the correct species for the protein mentions is a very important step towards ti. however, as far as we know, there has been little work in species disambiguation and in to what extent resolving species ambiguity at an entity level can help ti.  species disambiguation the species tagger was developed on the iti txm corpora, which were produced as part of the txm project. we created two corpora in slightly different domains, eppi and te. the eppi corpus consists of full-text papers selected from pubmed and pubmed central and domain experts annotated all documents for both protein entities and ppis, as well as extra information associated with the ppis and normalisations of the proteins to publicly available ontologies. the te corpus consists of full-text papers, in which entities such as proteins, tissues, genes and mrnacdnas were identified, and a new tissue expression relation was marked up. we used these corpora to develop a species tagging system. the biomedical entities in the data were manually assigned with standard database identifiers, where genes were assigned with entrezgene ids, and proteins and mrnacdnas with refseq ids. hence it was straightforward to obtain their species ids through the mappings provided by entrezgene and refseq. in more detail, proteins, protein complexes, genes and mrnacdnas in both eppi and te datasets were assigned with ncbi taxonomy ids, to denote their species. the eppi and te datasets have different distributions of species. for example, the entities in the eppi training data belong to species with human being the most frequent at %, and those in the te training set are across species and mouse was the most frequent at %. to calculate the inter-annotator-agreement, about 40% of the documents were doubly annotated by different annotators. the averaged f scores of species annotation on the doubly annotated eppi and te datasets are % and %, respectively, indicating that human annotators have high agreement when assigning species to biomedical entities. to assess how much species ambiguity accounts for the overall ambiguity in biomedical entities, we estimated the averaged ambiguity rates for the protein entities in the txm datasets, without and with the species information. suppose there are n unique protein mentions in a dataset. first, we look up the refseq database by exact match with every unique protein mention mi, where i ∈ {.n - 1}, and for each mi we retrieve two lists of identifiers: li and l′i, where li consists of all identifiers and l′i only contains the identifiers whose model organism matches the manually tagged species of the protein mention. the ambiguity rates without and with species are computed by ∑i=0n−1|li|n and ∑i=0n−1|l′i|n, respectively. table shows the ambiguity rates on the eppi and te datasets. using the iti txm corpora, we first devised a number of rule-based species disambiguation systems. it is intuitive that a species word that occurs near an entity is a strong indicator of its species. to assess this intuition, we developed a set of rules using heuristics and the species words detected by a species word tagger. • prewd: if the word preceding an entity is a species word, assign the species indicated by that word to the entity. • prewd sent: if a species word occurs to the left of an entity and in the same sentence, assign the species indicated by that word to the entity. • prefix: if an entity has a species-indicating prefix, e.g., msos- then tag the species to that entity. • spread: spread the species of an entity e to all entities in the same document that have the same surface form with e. this rule must be used in conjunction with the other rules. • majority vote: count the species words in a document and assign as a weight to each species the proportion of all species words in the document that refer to the species. for example, if there are n species words in a document and nhuman are associated with human, the human species weight is calculated as nhumann. tag all entities in the document the species with the highest weight, defaulting to human in the case of a tie. this rule was used by default in the rule-based ti system, described later in this paper. it is slightly counter-intuitive that using a rule such as'prewd' did not achieve perfect precision. closer inspection revealed that most of the false positives were due to a few problematic guidelines in the annotation process. for example, • "the amounts of human and mouse cd200r...", where'cd200r' was tagged as mouse by the system but the gold-standard answer was human. this was due to the fact that the annotation tool was not able to assign multiple correct species to a single entity. • "... wheat eifiso4g...", where'eifiso4g' was tagged as wheat but the annotator thought it was triticum. in this case, taxid is a species under genus and arguably is also a correct answer. other similar cases include xenopus vs. xenopus tropicalis, and rattus vs. rattus norvegicus, etc. this is the main cause for the false positives as our system always predicts species instead of genus or taxids of any other ranks, which the annotators occasionally employed. furthermore, we split the eppi and te datasets into training and development test sets and developed a machine-learning based species tagger. using the training splits, we trained a maximum entropy classifier on a number of features such as contextual words and nearly species words, which will be detailed later. the results of the ml species tagger are shown in table we measure the performance in accuracy instead of f because the ml based tagger assigns a species tag to every entity occurrence, and therefore precision is equal to recall. we tested four models on the devtest portions of the eppi and te corpora: • bl: a baseline system, which tags the devtest instances using the most frequent species occurring in the corresponding training dataset. for example, human is the most frequent species in the eppi training data, and therefore all entities in the eppi devtest dataset were tagged with human. • eppi model: obtained by training the maxent classifier on the eppi training data. • te model: obtained by training the maxent classifier on the te training data. • combined model: obtained by training the maxent classifier on a joint dataset consisting of both the eppi and te training corpora. finally, we devised a hybrid species tagging system. as we have shown, the rules'prewd' and'prefix' achieved very good precision but low recall, which suggests that when these rules were applicable, it is highly likely that they would get the correct species. based on this observation, we combined the ml approach and the rule-based approach in such a way that the rules'prewd' and'prefix' were applied on top of ml and to override predictions made by ml. the hybrid systems were tested on the same datasets and the results are shown in the right columns in table we performed significance tests on the results in table first, a friedman test was used to determine whether the sets of results were significantly different, and then pairwise wilcoxon signed rank tests were employed to tell whether any system performed significantly better than others. on both datasets, the machine-learning models significantly outperformed the baseline. on eppi devtest dataset, the eppi models and the combined models outperformed the te models, while on te dataset, the te models and the combined models outperformed the eppi models. applying the post filtering rules did not significantly improve the ml models, although it appears that adding the rules consistently increased the accuracy. term identification with species disambiguation experiments on the iti txm corpora to identify whether species disambiguation can improve performance of ti, we ran the ti system on the eppi and te datasets in the iti txm corpora. we tested the ti systems with or without help from a number of species tagging systems, including: • baseline: run ti without species tags. note that the ti system already integrated a basic species tagging system that uses the majority vote rule. thus this is a fairly high'baseline'. • gold species: run ti with manually tagged species. this is the upper-bound performance. • rule: run ti with species predicted by the rule-based species tagger using rules "prewd" and "prefix". • ml: run ti with the species that occurs most frequently in the training datasets. • ml: run ti with species predicted by the ml tagger trained on the eppi training dataset. • ml+rule: run ti with species predicted by the hybrid system using both ml and the rules. • ml: run ti with species predicted by the ml tagger trained on the te training dataset. • ml+rule: run ti with species predicted by the hybrid system using both ml and the rules. • ml: run ti with species predicted by the ml tagger trained on both eppi and te training data. • ml+rule: run ti with species predicted by the hybrid system using both ml and the rules. we score the systems using top n precision, where n ∈ { 20}. the argument for this evaluation scheme is that if a ti system is not good enough in predicting a single identifier correctly, a'bag' of ids with the correct answer included would also be helpful. the'avg. rank' field denotes the averaged position where the correct answer lies in, and the lower the value is, the better the ti system performs. for example, a ti system with an'avg. rank' of would be ideal, as it would always return the correct id at the top of the list. note that in the te data, not only protein entities, but also genes, mrnacdna, and gomops were tagged, where a gomop denotes an entity being either a gene, or an mrnacdna, or a protein. as shown in tables and on both datasets, using the gold standard species much improved accuracy of ti. also, automatically predicted species tags were proven to be helpful. on the eppi data, the ml+rule outperformed other systems. note that the species distribution in the devtest dataset is strongly biased to human, which explains why the ml system performed nearly as well. however, defaulting to human was not guaranteed to succeed because one would not be able to know the prior species in a collection of unseen documents. indeed, on the te data, the system ml, which uses mouse as default, yielded poor results. experiments on biocreative data to assess the portability of the species tagging approaches, an "artificial" dataset was created by joining the species-specific datasets from biocreative & gn tasks to form a corpus consisting of four species. in detail, four datasets were taken, three from biocreative task 1b and one from biocreative task gn. assuming genes in each dataset are species-specific, we can train/test ml models for species disambiguation and apply them to help ti. this task is more difficult than the original biocreative gn tasks due to the additional ambiguity caused by multiple model organisms. note that the above assumption is not strictly true because each dataset may contain genes of other species, and it would be hard to assess how true it is as abstracts in the biocreative gn datasets are not normalised to an entity level. we first carried out experiments on species disambiguation. in addition to the txm +rule model) and the majority vote taggers, we trained the species tagger on a dataset comprising of the devtest sets from the biocreative i & ii gn tasks. in more detail, we first pre-processed the dataset and marked up gene entities with an ner system, which was trained on biocreative ii gm training and test datasets. the entities were tagged with the species as indicated by the source dataset where they were drawn from, which were used as the'gold' species. using the same algorithm and feature set as described previously, a bc model was trained. as shown in table except on human, the txm model yielded very disappointing results, whereas the bc model did well overall. this was because the txm model was trained on a dataset where fly and yeast entities occur rarely with only 2% and 5% of the training instances belonging to these species, respectively, which again revealed the influence of the bias introduced in the training material to the ml models. using the species disambiguation models, we carried out ti experiments, using the same procedure as we did on the txm data. the results were obtained using the official biocreative gn scorers and are presented in table performance of ti assisted by all three species taggers were much behind that of ti using the gold-standard species, which shows species-tagging can potentially enhance ti performance and there is much room for improving the species disambiguation systems. on the other hand, it was disappointing that the'majority vote' system, which did not use any external species tagger, achieved the best results, while ti with the'bc model' tagger yielded slightly worse results and the txm model performed poorly. one possible reason that the'majority vote' tagger yielded reasonably good result on the biocreative dataset, but unsatisfactory result on the txm datasets was due to the difference in document length in the two corpora: the biocreative corpus is comprised of abstracts and the txm corpora consist of only full-length articles. in abstracts, authors are inclined to only talk about the main biomedical entities described in the paper, whereas in full articles, they tend to describe a larger number of entities, possibly in multiple species, for the purposes of describing related work or comparison. recall that the'majority vote' rule outputs the species indicated by the majority of the species words, which would obviously perform better on abstracts, where more likely only one species is described, than on full-length articles. table shows the number of species per document in the txm data, where most documents involve more than one species, in which cases the'majority vote' would not be able to take obvious advantage.  we have presented a range of solutions to the task of species disambiguation and evaluated their performance on the iti txm corpus, and on a joint dataset from biocreative i & ii gn tasks. we showed that rule-based species tagging systems that exploit heuristics, such as previous species words or species prefix, can achieve very high precision but low recall. ml species taggers, on the other hand, can achieve good overall performance, under the condition that the species distributions in training and test datasets are not too distant. our best performing species tagger is a hybrid system that first uses ml to predict species and then applies certain rules to correct errors. we also performed ti experiments with help from the species tags assigned by human annotators, or predicted by the automatic species taggers. on all datasets, the gold-standard species tags much improved ti performance: % on the eppi devtest set, % on the te devtest set, and % on the biocreative gn test datasets, which clearly shows that species information is indeed very important for ti. on the eppi and te datasets, the species predicted by the best-performing hybrid system improved ti by % and %, respectively. on the combined dataset from biocreative gn tasks, however, it did not work as well as expected. 
however, full-length articles often describe entities across a number of species, in which case resolving the ambiguity of model organisms in entities is critical to achieving accurate term identification. this paper shows that, in the context of identifying terms involving multiple model organisms, integration of an accurate species disambiguation system can significantly improve the performance of term identification systems. by utilising the species information predicted by the hybrid tagger, our rule-based term identification system was improved significantly by up to %. we developed and compared a number of rule-based and machine-learning based approaches to resolving species ambiguity in mentions of biomedical named entities, and demonstrated that a hybrid method achieved the best overall accuracy at %, as tested on the gold-standard iti-txm corpora. term identification is the task of grounding ambiguous mentions of biomedical named entities in text to unique database identifiers. previous work on term identification has focused on studying species-specific documents.
 term identification is the task of grounding ambiguous mentions of biomedical named entities in text to unique database identifiers. previous work on term identification has focused on studying species-specific documents. however, full-length articles often describe entities across a number of species, in which case resolving the ambiguity of model organisms in entities is critical to achieving accurate term identification.  we developed and compared a number of rule-based and machine-learning based approaches to resolving species ambiguity in mentions of biomedical named entities, and demonstrated that a hybrid method achieved the best overall accuracy at %, as tested on the gold-standard iti-txm corpora. by utilising the species information predicted by the hybrid tagger, our rule-based term identification system was improved significantly by up to %.  this paper shows that, in the context of identifying terms involving multiple model organisms, integration of an accurate species disambiguation system can significantly improve the performance of term identification systems. natural language processing in biomedicine acl workshop columbus, oh, usa 