BACKGROUND
word sense disambiguation  deals with relating the occurrence of a word in a text to a specific meaning, which is distinguishable from other meanings that can potentially be related to that same word  <cit> .
wsd is essentially a classification problem: given an input text and a set of sense tags for the ambiguous words in the text, assign the correct senses to these words.
sense assignment often involves two assumptions: a. within a discourse, e.g.
a document, a word is only used in one sense  <cit>  and b. words have a tendency to exhibit only one sense in a given collocation – neighbouring words  <cit> .
during the last years, word sense disambiguation has become a hot topic in the biomedical domain.
the challenge here is the rapid growth of the biomedical literature in terms of new words and their senses, with the situation getting worse with the use of abbreviations and synonyms.
this illustrates the exact need in the case of the biomedical domain; the development of statistical approaches that utilize "established knowledge"  and require no or only some parsing of the text in order to perform the correct annotation.
two main decision points for wsd in the biomedical domain are the granularity to which wsd should be performed and the selection of an appropriate corpus for training and evaluation.
concerning granularity, some tasks are easier than others .
concerning the biomedical corpora, those are few, mainly due to the time-consuming and labor-intensive process of manual or semi-automatic annotation.
examples of such datasets are the nlm wsd test collection  <cit> , medstract for acronyms  <cit>  and the biocreative set for mouse, fruitfly, and yeast  <cit> .
however, depending on the task, researchers need to collect their own gold standard datasets.
algorithms for word sense disambiguation
as shown in table  <dig>  wsd algorithms can be distinguished as supervised, unsupervised, or using established knowledge  <cit> .
in the biomedical domain researchers have focused on supervised methods  <cit>  and using established knowledge  <cit>  to perform gene name normalization and resolve abbreviations.
according to the recent biocreative challenge, the former problem can be solved with up to 81% success rate  <cit>  for human genes, which are challenging with  <dig>  synonyms per name .
resolving ambiguous abbreviations achieves higher success rates of close to 100%, as the task is less complex when long forms of the abbreviated terms are in the document  <cit> .
the above approaches use cosine similarity  <cit> , svm  <cit> , bayes, decision trees, induced rules  <cit> , and background knowledge sources such as the unified medical language system   <cit> , medical subject headings   <cit> , and the gene ontology   <cit> .
two approaches use metadata, such as authors  <cit>  and journal descriptor indexing  <cit> .
most of the unsupervised approaches so far were evaluated outside the biomedical domain  <cit> , with the exception of  <cit> , who used relations between terms given by the umls for unsupervised wsd of medical documents and achieved 74% precision and 49% recall.
another approach that uses the umls as background knowledge for wsd is that of  <cit> , who compared the results from a naive bayes classifier and other algorithms  to conclude that different senses in the umls could contribute to inaccuracies in the gold standard used for training, leading to varied performance of the wsd techniques.
another approach by  <cit>  is based on a graph model representing words and relationships  between them and uses wordnet  <cit>  for assigning labels.
interestingly, most of the above approaches consider the background knowledge sources as terminologies, without taking into account the taxonomic structure or the terms' semantic similarity  <cit> .
here, we fill this gap by systematically comparing three approaches using ontologies with inference and semantic similarity and the use of metadata to solve the problem of wsd for ontological terms.
the goal is to establish how the use of ontologies and metadata can improve results.
in previous work  <cit> , we proposed and evaluated two approaches to the wsd problem, namely term co-occurrences in pubmed abstracts and document clustering.
we proposed a methodology for finding whether an article in an automatically annotated database is likely to be true or false with respect to the biological meaning and constructed a co-occurrence graph of go terms based on gene ontology annotations   <cit> .
in the present article we extend this approach  in the following ways: first, we additionally disambiguate mesh terms and use a larger training corpus to get the co-occurrence scores, since there exist ~ <dig>   <dig>   <dig> documents to which experts have assigned mesh terms.
second, we make use of the hierarchy structure in both go and mesh , whereas before we used term co-occurrence without any inference.
we therefore investigate how two different hierarchies influence the performance of disambiguation.
third, we combine our graph-based decision function with a support vector machine, arranged in a co-training scheme, to learn and improve models without any labelled data.
finally, we test the disambiguation performance in new larger benchmark datasets of varying curation quality.
the 'term cooc' approach is similar to dorow's approach  <cit> , with the difference that we construct the co-occurrence graph based on goa and mesh, which are manually annotated datasets.
therefore, our graphs contain only relations  between terms  that are semantically meaningful in the context of an article.
dorow's graph contains all the nouns that co-occur with one another, but in the case of the biological context, we are interested only in a local subgraph of dorow's graph .
another difference is that we use established knowledge in go and mesh to draw the nodes and in the different configurations of our method we use a support vector machine and/or incorporate the term relationships in go and mesh.
we introduce two more methods for disambiguation, differing from the co-occurrences approach in terms of automation and background knowledge required.
the 'closest sense' approach computes similarities between the senses of the ambiguous term, the senses of its neighbours  and the type of relations that could occur between them.
'closest sense'  uses the umls semantic network as background knowledge, like  <cit> , who rely on the context of the ambiguous term in order to compute a score for each sense candidate.
this score consists of the number of terms in the document which are related, in the umls, with the different senses of the ambiguous term.
in comparison to the cs method, this approach is different in two main points:  it does not take advantage of the hierarchies of concepts and relations in the umls and  it ignores terms which co-occur with the ambiguous term in the same context but do not have a direct link with it in the umls.
the 'metadata' method uses maximum entropy for modelling the behaviour of occurrence of contextual terms and phrases in text together with a potentially ambiguous term.
the features selected are n-tuples of word stems and metadata such as the journal and document title.
the method requires a set of labelled documents for each term to be disambiguated.
we evaluate and compare the three strategies for wsd, starting from the unsupervised/automated to the least automated one.
the comparison includes each method's requirements and limitations in terms of training data and automation, the behaviour of the methods during the use of different taxonomies  as well as comparison against a classical stem co-occurrence approach.
we additionally make the benchmark datasets created for the purpose of disambiguation available, since the collection process is time-consuming and labour-intensive.
these include  <dig> manually curated documents of high/medium curation quality for  <dig> selected go and mesh terms.
methods
terminology and classification of approaches
the types of relations between terms in the gene ontology , the medical subject headings  and the unified medical language system  semantic network make them completely different knowledge sources  <cit> .
go has a simple structure in the form of a directed acyclic graph and go terms are interconnected via is_a and part_of relations.
the semantics of relations used in mesh make it a terminology rather than an ontology.
terms in mesh are related through a narrower than b relations, giving users who are interested in bs the option to look at as.
the umls is considered to be a terminology integration system comprising over  <dig> biomedical vocabularies and relations like subclassof or subpropertyof between terms.
therefore, it is located in the space between a structured terminology and an ontology.
the most popular semantic web formalisms for representing taxonomies, ontologies and terminologies in general are the resource description framework  and the web ontology language , with owl being more suitable for ontologies and rdf sufficient for terminologies.
the simple knowledge organization systems  is an area of work developing specifications and standards to support the use of knowledge organisation systems  such as thesauri, classification schemes, subject heading systems and taxonomies within the framework of the semantic web.
skos provides a standard way to represent knowledge organisation systems using rdf.
lately, there have also been provided owl translations of go and mesh by the responsible consortia.
we designed, implemented, and evaluated three wsd methods that we refer to as: closest sense , term cooc , and metadata .
these differ as follows:
background knowledge
closest sense  uses the umls semantic network; it represents an abstract as a list of umls terms occurring in the abstract.
term cooc  uses co-occurrences of terms in go and mesh, built from a curated dataset; it represents a document abstract as a list of go and mesh terms occurring in the abstract.
the metadata method  uses metadata about the journal and title; it represents a document abstract as n-tuples of word stems and metadata.
classification
cs uses shortest semantic distance of co-occurrences to sense.
tc uses svms and co-occurrences from a training dataset for finding boundary between senses.
md uses the maximum entropy to model the behavior of the co-occurrence of contextual words and metadata with the ambiguous term.
closest sense method
this wsd approach was initially used to address the ambiguity problems in the meatannot system  <cit> .
the main idea of the approach is the following: given a set of different senses of the ambiguous term, the co-occurring terms in the same text and the hierarchy where they belong , decide which sense is true based on the  distance to the senses of the co-occuring terms.
to clarify this, we can have the following sentence as an example: 'i also tracked lipid profiles, hba1c, blood pressure, body mass index, hostility and nicotine use'.
the term 'blood pressure' can have three senses, namely 'organism function', 'diagnostic procedure' and 'laboratory or test result'.
the senses of the co-occurring terms are 'laboratory procedure' , 'gene or genome' , 'diagnostic procedure' , 'mental process'  and 'organic chemical' .
the sense of 'diagnostic procedure' for blood pressure is in average closer to the senses of the co-occurring terms than the other candidate senses.
for the example in figure  <dig>  the cs method determines the meaning of 'thrush' by examining what appears in the same sentence and/or paragraph  and then computing a similarity based on semantic distances to 'songbird' and 'oral candidiasis' in the umls hierarchy; the highest similarity determines the result.
intuitively, with semantic distances, two senses are close if there exists a possibility to use them in a concise annotation graph.
algorithm
the 'closest sense' algorithm takes as input:  the ambiguous term τ,  the vector vτ of different senses of τ,  the vector vcτ of senses found in the context , and  the umls semantic network.
first, the disambiguator builds a vector vcτ of senses describing the context of the ambiguous term τ.
this vector includes the senses of terms that are neighbours of τ. then, it computes the similarity between each sense in vectors vτ and vcτ.
the resulting similarity is the average of similarities between senses in the two vectors.
finally, the sense in vτ that has the highest average similarity to vcτ is proposed as the best for τ.
semantic distances
the distance metrics used to find the correct sense are the subsumption distance and the subtype-aware signature distance.
the subsumption distance is the length of the shortest path between two senses in the hierarchy of senses, where the length of an individual subsumption link gets exponentially smaller with the depth of the senses it links in the hierarchy.
the subtype-aware signature distance is the length of the shortest path between two concepts/terms through the graph formed by the property types with their range links and domain links.
with this new semantic distance we merge signature and hierarchies graphs.
the main idea is to find a path between two concepts/terms by using the ontology structure  and the signature of relations .
the subtype-aware signature consists of relations in the hierarchy  additional to the common signature .
it is aware of the properties of a term , the position of the term in the hierarchy  and the hierarchy of the properties .
term cooc method
the term cooc method relies on selecting the term that most frequently co-occurs with the ambiguous term in the training corpus.
it selects the highest co-occurring term with the ambiguous term for defining the given sense as true or false.
in order to formalize the notion of term co-occurrences , we consider pairs of go/mesh terms that appear in the same abstract and we represent all such pairs of terms in a manually annotated goa or mesh co-occurrence graph .
each node in the co-occurrence graph represents a go or mesh manual annotation.
an edge between nodes α and β represents a real number, the log-odds score, representing the frequency log – odds of the terms' α and β co-occurrence over all articles, weighted by their total number of occurrences.
for the example in figure  <dig>  the tc method determines the meaning of 'thrush' by examining what appears in the same abstract  and then considering all known co-occurrences between ontology terms in a training corpus; the value of the highest co-occurrence determines the result, e.g.
'swallows' would have relatively high co-occurrence with 'thrush' songbird.
algorithm
first, we use a simple threshold considering how close to an ambiguous term the highest co-occurring term  is; if below a user-defined threshold θ, the ambiguous term is negative, else it is positive with respect to the term.
second, we use support vector machines  trained on all tokens of a text  <cit> .
the method first runs a binary svm against a set of articles ordered by maximum co-occurrence with the ambiguous term.
the highest and lowest 10% of articles in the set are labelled as positive and negative; then the svm is trained on lower 10%  and upper 10% .
after the initial convergence is achieved, the error  will be low, likely near  <dig>  the algorithm next improves this result by iteratively re-classifying the remaining articles that have less extreme co-occurrences with the ambiguous term, one-by-one, followed by re-training the svm on the newly relabelled data set.
this continues until no more articles are left.
the steps are:
<dig>  set s = order articles based on their highest co-occurring go/mesh annotation  with the ambiguous term.
<dig>  t = lowest and highest 10% of s; label t as negative and positive; train svm with t; remove t from s.
<dig>  for s ∈ s: move s to t; classify s; re-train svm with t.
training with co-occurrence graphs
these graphs are used in the tc method for training.
for wsd of go terms, the co-occurrence graph derived from the gene ontology annotations   <cit> , and for the mesh terms from the medical subject headings   <cit> .
goa represents articles manually annotated with go terms and consists of ~ <dig>  articles.
there exist ~ <dig> , <dig> documents to which experts have assigned mesh terms.
we found co-occurring terms in the goa and mesh annotated articles and built a co-occurrence graph representing how frequently pairs of go or mesh terms co-occur.
nodes represent annotations and edges represent the frequency of co-occurrence of two annotations in the same article, normalized based on each goa/mesh annotation's individual occurrence frequency in the specific corpus.
training with inferred co-occurrences
we extend the co-occurrences in a hierarchical fashion to ensure that given a goa-derived co-occurrence between a pair of terms, goacooc, the ancestors of α and β in the ontology are updated with the co-occurrence such that only the maximum co-occurrence is kept.
this is important given the few annotations in goa and the is_a relationships between go terms, since ancestors inherit the co-occurrences of their children.
with the inferred co-occurrences, given an ambiguous term α, the co-occurrence of α with a term β will not be lower than α's co-occurrence with any of β's descendants.
metadata method
as an alternative method for wsd we use a maximum entropy approach as described in  <cit> .
maximum entropy models have been successfully used in tasks like part of speech tagging, sentence detection, prepositional phrase attachment, and named entity recognition.
for the example in figure  <dig>  the md method determines the meaning of 'thrush' by using metadata for the document and then deciding based on what was previously learned about this metadata from training examples.
the metadata used are n-tuples of word stems from different scopes, namely the paper title, the sentence including the ambiguous term or the whole abstract, the journal title as well as the publication period, since some topics can be popular in different decades.
the occurrence of contextual words and phrases in a text together with a potentially ambiguous term can be seen as a random process.
maximum entropy modelling aims at modelling the behavior of this random process.
provided a large amount of training examples, the algorithm automatically extracts a set of relationships inherent in the examples, and then combines these rules into a model of the data that is both accurate and compact.
algorithm
the training and test data in our case are sentences containing the potentially ambiguous term flagged with the sense.
each training example, one sentence each, is represented as a set of features.
an implementation of the porter stemmer is used  <cit>  and as features we select n-tuples of word stems and meta information of the document, such as the journal and title words and the publication period .
the implementation  <cit>  takes a series of events to train a model.
each event is a configuration of binary relations associated with a label.
the resulting model is applied to an unknown configuration of binary relations.
the result is the predicted probability for the previously trained outcomes.
mesh terms already assigned to the articles are excluded, for the performance evaluation to be independent of them.
given the abstract of a scientific article and the ambiguous term, the steps followed are:
<dig>  extract binary features
<dig>  get scalar product of feature vector and model
<dig>  the result is the probabilities for predefined outcomes
<dig>  if above a threshold  <dig> , the term is true, else false.
as an illustrating example of the features extracted, articles mentioning 'signal transduction', 'kinase', 'embryo', 'neuron' or 'stage' are more likely to refer to 'multicellular organismal development' than to another sense, such as development of an algorithm or a disease in an organism.
articles mentioning 'anxiety', 'behaviour', 'memory', 'social' and 'fear' are more likely to refer to 'psychological inhibition', instead of 'enzyme inhibition'.
experimental setup
classification task and limitations
the disambiguation performed here is mainly a classification task; it represents the prediction whether an annotation is positive or negative with respect to the go/mesh sense.
we do not assign one of the numerous different senses to a term, but instead a positive or negative label to it, when it corresponds to the go/mesh sense or not, respectively.
we do not handle acronym ambiguity separately.
however, in cases where an acronym belongs to an ontology term label , this is disambiguated in the same way as all ontology term labels.
as mentioned in the introduction, some disambiguation tasks are easier than others; 'bank' the building and the 'bank' gene will appear in completely different context, whereas the 'bank' gene, protein or mrna are even likely to appear in the same article abstract, making the disambiguation task often difficult even for a domain expert.
'transport by air' or 'patient transport' will be easier to distinguish from the go sense of transport, but 'transport of virus cultures' will appear in a closer molecular biology context.
distinguishing between 'transport', 'rna transport', 'trna transport' or 'ion transport' can become less difficult by using the hierarchical information in the ontology .
some terms are also easier to disambiguate in the same task, depending on the number of their different senses  and the distance between them, the way they appear in text  and the number of tokens they consist of .
examples of the senses  per ambiguous term included in the benchmark dataset collected.
the ambiguous terms examined are the go terms 'development' , 'spindle' , 'nucleus'  and 'transport'  and the mesh terms 'thrush' , 'lead' and 'inhibition' .
most of the different senses of the terms examined  belonged as well to the biomedical domain, making the disambiguation task more difficult .
the limited number of terms examined is due to the labor-intensive process of manual collection of proper benchmark datasets.
as mentioned in the introduction, there exist few annotated biomedical corpora for evaluation and depending on the task, researchers need to collect their own gold standard datasets.
we collected datasets for a list of ambiguous terms based on the amount of true/false data available and the frequency of occurrence in pubmed .
we aimed at keeping the ratio of true/false abstracts close to  <dig>  giving a 50% chance to each appearance of the term to be true or false with respect to the go/mesh sense .
we first examined the umls wsd collection  <cit>  for ambiguous go/mesh terms and data availability and later a list of common false positive terms based on manual curations in gopubmed .
from the umls wsd collection we selected terms that were go/mesh terms and the senses provided were distant to each other, i.e.
in the case of 'lead', the two senses with short semantic distance  were considered as one, as they both are about the compound.
a semantically more distant sense is that of the verb to lead/result in.
regarding the false/positive ratio limitation/criterion, for some terms this was not satisfied, not allowing the inclusion into the evaluation dataset.
for example, for 'transport' the umls wsd collection contained  <dig> abstracts classified as sense <dig>  and only  <dig> as other .
we therefore needed to manually collect false examples containing other senses for a balanced corpus.
another question was whether the definition of the negative datasets would influence the results.
to test this, we defined a more general negative dataset by completely randomly choosing articles.
defining a random set as a negative set is common practice, e.g.
in predicting protein-protein interactions.
obviously, the random negative dataset is very different from the positive dataset, since most likely it does not contain any of the negative senses at all, but is just the bias for the "average" paper.
results showed a decrease of ~7% in the performance of the methods.
this argument can be turned around.
while our initial negative dataset was carefully and manually chosen, it could be further improved by letting its composition of other senses reflect the distribution of use of these senses in pubmed as a whole.
however, achieving this ideal would require annotating all articles with the term in the whole of pubmed with the senses.
given that pubmed has for example more than  <dig> million articles on development, this cannot be easily accomplished.
however, the composition of negative senses in our negative dataset aims to reflect the composition of negative senses in pubmed as well as possible through the query and annotation strategy, that was pursued.
since we needed to include every possible sense of the ambiguous terms, the queries formed were such that could collect representative abstracts for each sense, a process that was manual and time-consuming.
the collection of the positive examples was easier, since there was one sense  and also more frequent in pubmed, therefore the term itself or one of its synonyms were enough to be put in the query to pubmed.
the collection of negative examples was as expected harder, since they were not frequent in pubmed and we needed to include enough examples of every possible sense.
most of the queries used for this included the ambiguous term or synonyms of it and keywords that were often in the context, based on personal experience from previous curation of automatic annotations in gopubmed.
for example, for 'development', the queries used were 'development and staff', 'development and algorithm', 'development and software', 'development and treatment', 'development and method', etc.
for 'thrush', since we could only locate one negative sense, we used queries such as 'thrush nightingale', 'thrush and songbird', 'mountain thrush', etc.
the other aspect is the question of size composition of positive and negative.
we chose roughly 50% positive and 50% negative.
this basically means that the a priori likelihood is 50% for the corect sense.
if, instead, we aim to identify each sense correctly, the following problem arises: assume there are ten senses, i.e.
<dig> positive and  <dig> negative.
then the a priori probability for the classification would be 10% and then a simple strategy would be to always vote negative.
overall, the approach pursued  plus equally weighted positive and negative datasets is a suitable approach for evaluation.
datasets
we collected three different benchmark datasets  to evaluate the performance of the three methods.
they differ in quality and quantity, depending on their collection process .
the common reference dataset between the three methods is the manually annotated by a domain expert one:
the above datasets contain manually collected pubmed articles by one expert , manually curated articles by a group of non-experts  and semi-automatically collected articles .
see datasets section for details.
high quality, low quantity corpus
this corpus consists of ~ <dig> true and  <dig> false example documents  per ambiguous term.
for the ambiguous go terms examined and the mesh term thrush we collected both true and false examples manually.
true examples are abstracts that discuss, for instance, 'development', in the sense specified by go.
false examples also contained the ambiguous term, but in other senses, closer or not .
for the ambiguous mesh terms 'lead' and 'inhibition ', the test set originated from the umls wsd corpus  <cit> .
these two were the only terms depicting mesh terms.
all other terms in the umls wsd  were only found in go or mesh as substrings and would thus not be contained in either co-occurrence graph as single nodes.
medium quality, medium quantity corpus
this corpus consists of documents for which the annotation has been manually confirmed by a group of expert and non-expert curators.
we asked colleagues to confirm or reject the automatic annotations  provided by gopubmed for a collection of article abstracts.
this collection has been mainly automatically created, as described next .
for each of the automatic annotations, the curators could select among three options: a. true and important for the context of the publication, b. of minor importance/relevance and c. false annotation.
the curation tool is available via gopubmed  <cit> .
low quality, high quantity corpus
this corpus was created mainly automatically.
we implemented similarity-based clustering of abstracts with literal occurrence of the ambiguous terms.
each abstract was matched to its nearest abstract, conceptualized as a directed edge from the former to the latter.
then every connected component was considered as a cluster.
from an initial manual evaluation of the clustering results, clusters of size >  <dig> were consistent enough, meaning that articles in such clusters were referring to one sense of the ambiguous term in 72–95% of the cases.
each cluster's abstracts were input into a system developed in-house  that generated a list of terms describing each cluster based on term frequency inverse document frequency .
the top  <dig> terms of the list were later evaluated by an expert which labelled the clustered articles as true or false for the respective go/mesh term.
the above facilitated and accelerated the dataset collection process without any significant loss in data quality .
experiment
for evaluation and comparison purpose, each method's performance was tested  on the high quality/low quantity dataset .
we also applied classical stem co-occurrence analysis as a baseline on the same dataset; this consisted of basic maximum entropy modelling on stems without any use of metadata or hierarchical information .
we additionally tested each method's performance separately with different test datasets.
for the 'term cooc' method , the performance of co-occurrences of go/mesh terms and inferred co-occurrences of go/mesh terms  was tested in the three benchmark datasets described earlier, in order to evaluate the method in larger  datasets, since it has been shown that sample size, sense distribution and degree of difficulty impact on the classification task  <cit> .
input to this method were the automatic annotations per article provided by gopubmed  and the respective co-occurrence graph.
as a side experiment, we tested the tc method for the disambiguation of mesh terms without including the mesh hand annotations in the automatic annotations provided by gopubmed, to estimate how the quality of the input influences the quality of the results.
for the 'closest sense' method , input was the umls semantic network and the article abstracts.
this method was additionally tested on the wsd test collection  <cit> .
for the 'metadata' method , we used the three different datasets for training and testing.
the high quality dataset was used in an initial experiment  as training and testing dataset, in a 5-fold cross validation.
then the medium quality and low quality datasets were separately used as training sets, with testing of the method on the high quality dataset .
cs <dig> column contains the results  for the closest sense  approach with the use of the classic distance .
cs <dig> column contains the results for the cs approach with the use of the optimized signature together with the subsumption distance.
tc <dig> and tc <dig> contain the results of the term cooc  approach, when the co-occurrences or the inferred co-occurrences are used, respectively.
tc <dig> contains the results for the tc approach with co-occurrences and support vector machines, and tc <dig> when inferred co-occurrences and svms are used.
bme column contains the results for the baseline method , trained and tested on the high quality corpus in a 5-fold cross validation.
md <dig> is for the metadata approach, trained and tested on the high quality corpus in a 5-fold cross validation.
md <dig> is trained on the medium quality/quantity corpus and tested on the high quality one.
md <dig> was trained on the low quality/high quantity corpus and tested on the high quality corpus.
some terms  are easier to disambiguate than others .
overall, all methods perform well between 73–96% f-measure ).
RESULTS
the performance of the three disambiguation approaches  and the baseline  was tested on a common high quality/low quantity dataset.
the overall results of this comparison are shown in table  <dig> .
all methods perform well between 73–96% average f-measure.
in particular, the metadata  approach is the best one: when trained on high quality data , it achieves 96% f-measure.
when the metadata are not used  the accuracy falls to 90%.
the term cooc  method follows with 81% and the closest sense  approach with 77% .
all methods present low f-measure for 'development' and 'lead' .
the best results  are obtained for go terms 'transport', 'nucleus' and 'spindle' .
as far as the closest sense approach is concerned, there is a clear improvement in the results  with the use of the optimized signature together with the subsumption distance.
for the term cooc approach, when the inferred co-occurrences are taken into account  in the case of the go terms the results remain the same, whereas in the case of the mesh terms the results are worse, mostly in terms of recall .
for go terms, the results are best when inferred co-occurrences are combined with svms , whereas for mesh terms, the best f-measure  is achieved when co-occurrences with svms are used, without the inferred co-occurrences .
this difference can be explained by the different structure of the two hierarchies.
go can be described as "tall and thin" , whereas mesh is "short and fat" .
additionally, the relations between terms in mesh are not exact is_a relations, but rather is_related_to.
therefore, propagating the term co-occurrences in mesh does not improve the results, since it does not necessarily mean that annotating with term meshx also means all of x's ancestors.
on the contrary, in go this is more likely to hold.
the metadata method gives – as expected – the best results.
when the method is trained and tested on the same high quality test , it results in an average f-measure of 96%.
when trained on the medium quality  and low quality  corpora and tested against the high quality corpus, the f-measure decreases into 81% and 70%, respectively, which are nonetheless high, compared to the quality of the training sets.
the high performance of the metadata approach is mainly due to the use of metadata as the title of the abstract and the journal.
for example, for the terms 'inhibition' and 'spindle' it achieves 100% f-measure and for 'nucleus' 99%.
the true sense of inhibition for mesh is psychological inhibition, which is easier to disambiguate, since it will mostly appear in psychology/psychiatry journals.
the same applies for 'spindle', which will mostly occur in cell biology and cell division/cycle journals.
we additionally tested each method's performance separately with different test datasets .
the 'closest sense' method was also tested on the nlm umls wsd collection  <cit>  to compare four versions of semantic distance computation in order to disambiguate term mapping to the umls semantic network.
the experiment showed that the use of the ontology definition can improve significantly the precision.
over the  <dig> ambiguous terms examined, the overall average precision was 83%.
for the 'term cooc' method, the performance of the different variants  was tested in the three benchmark datasets described earlier , in order to evaluate the method in larger  datasets.
testing the method from the highest towards the lowest quality , the f-measure decreases only by 3–10%, indicating a consistent behavior of the method.
as a side experiment, we tested the 'term cooc' method for the disambiguation of mesh terms without including the mesh hand annotations in the automatic annotations provided by gopubmed, to estimate how the quality of the input influences the quality of the results.
as expected, the results decreased dramatically , indicating that the mesh hand annotations provided per article are important for the disambiguation .
discussion
overall, the md method gave the highest f-measures among all methods.
the results became worse for the medium and high quantity datasets, since these were of lower quality in terms of correctness.
the md approach's consistency with giving the highest results is due to integrating metadata, such as journal and title, which are representative of the true meaning of an ambiguous term.
the md approach needs plenty of labelled data for training.
when comparing the results of the tc and cs methods to the baseline method  that performs only maximum entropy modelling of stems , bme still gives better results, but this is due to the available training data of high quality.
the disadvantage of md and bme compared to tc and cs is the need of high quality training data.
the md approach is less scalable in terms of storage demands as the number of articles increases, while the cs and tc approaches have constant storage demands .
in the tc method the svms increase the results up to 98%.
the tc method requires an ontology and co-occurrence graphs.
the origin of this graph should be a manually curated data source, in our case goa and mesh.
the quality of the graph will heavily depend on its origin and quality of the data.
the inferred co-occurrences improve the results for go, while for mesh they get worse.
this is due to the different structures of the two semantic hierarchies; the ancestors of an applicable go term are more likely to also be applicable to the same article, because of go's structure that is "tall and thin".
but mesh's structure is "short and fat" and is not always a thesaurus; not all of a node's ancestors are also applicable.
moreover, in the tc method the inferred co-occurrences only improve the result if combined with the svm.
this is because the inferred co-occurrences make the extreme co-occurrences with the ambiguous term, which the svm uses for training, more representative of an ambiguous term's true meaning.
figure  <dig> shows that the most extreme co-occurrences with the ambiguous term are most likely to be classified correctly, since the inferred co-occurrences make more precise the highest and lowest co-occurrences with an ambiguous term.
the middle co-occurrences are not necessarily made more precise with inferred co-occurrences.
that is why inferred co-occurrences help with the  svm training; while later on for middle co-occurrences the errors accumulate.
the cs approach needs only a semantic hierarchy in the form of an ontology, and in this sense is the most automated of the three methods.
moreover, cs gives good results, where the only problematic term is 'lead'.
however, cs is sensitive to the design of the ontology or subdomain of umls used, which reflects the view of the designers.
as shown by the accuracy of  <cit> , umls may not be the best choice to be used as background knowledge as the different parts of the hierarchy are modelled differently , resulting in different granularity.
different groups of people design ontologies differently; the various subdomains of an ontology will reflect the designers' views respecting depth, number of nodes, and structure.
therefore, the subdomains of the ontology influence the performance of the cs method, and the design rationale of the ontology may be ultimately responsible for performance differences on various terms.
for example, 'nucleus' is a subtree root in both go and snomed ; in go there are  <dig> descendants of nucleus, while in snomed  <dig>
CONCLUSIONS
based on the results, metadata and training data of high quality seem to be the key point for the increase of the accuracy.
when such training data are not available – as happens in most of the cases – co-occurrence of ontology/taxonomy/thesaurus terms can provide the way to the right decision.
moreover, the hierarchy of the terms and the subdomain, when consistently modelled, can depict the correct sense of an ambiguous term.
the md method produced the best results by including metadata in the wsd decision, but it requires high quality training data.
the most interesting thing about the tc and cs methods is that they are semi-automated, given a co-occurrence graph or ontology; then the training does not require manual intervention.
tc requires well modelled ontologies such as go, and deteriorates as the structure becomes less rigorous as in mesh.
cs requires large and consistently modelled ontologies, which are two opposing requirements.
thus, for tc and cs the structure of the ontology and subdomain affect the distance metric used and wsd quality.
future work will include identifying ambiguous terms for a certain corpus automatically.
for this purpose, we will employ wordnet, clustering, part of speech and noun phrase statistics, and expert input.
for tc and cs, we assumed that the other terms in the context are correct and independent of one another; in fact, they could also be ambiguous and therefore false.
for cs we will optimize the distance computation and propose other distances, taking into account existing annotation bases and ontology structure.
so far, the disambiguation performed was between the true sense in the hierarchy and all other senses that were considered as false.
a possible extension of the methods would be to correctly identify if a sense occurs that is not included in the thesaurus/ontology and possibly add it.
the closest sense method can potentially do this by setting a threshold.
from all distances below this certain threshold, one should be clearly shortest.
if not, then this indicates a new sense.
the term cooc and metadata approaches could be adjusted to identifying new senses by training each method on each sense and setting a certain threshold.
if the sense found is not above the threshold, then this can be a new sense.
abbreviations
wsd: word sense disambiguation; go: gene ontology; goa: gene ontology annotations; mesh: medical subject headings; umls: unified medical language system; snomed: systematized nomenclature of medicine; svm: support vector machine; tfidf: term frequency inverse document frequency; cs: closest sense method; tc: term cooc method; md: metadata method; bme: basic maximum entropy .
authors' contributions
da and ba developed the term cooc method with the assistance of jh.
ad implemented the metadata method and kk and fg the closest sense method.
tw implemented the terminology extraction method used in data collection.
hd assisted in data collection.
da performed the manual collection and evaluation of the benchmark datasets and drafted the manuscript.
ms supervised and coordinated the project.
all authors have read and accepted the final manuscript.
supplementary material
additional file 1
benchmark datasets used in the experiments.
the three corpora  are given in the form of pubmed identifiers  for true/false cases for the  <dig> ambiguous terms examined .
click here for file
additional file 2
detailed results for all methods.
detailed results for all methods given in terms of precision, recall, specificity and f-measure.
additional training and testing of the metadata method in all three corpora, closest sense results with different threshold, term cooc results with mesh text-mined annotations only, closest sense results on terms from the wsd test collection  <cit> .
click here for file
acknowledgements
the authors acknowledge the eu sealife project and the curators: a. tuukkanen, m. reimann, a. elefsinioti, a. marsico, l. royer, m. altrichter, c. winter, r. winnenburg, a. simeone, e. fritzilas.

 word sense disambiguation deals with relating the occurrence of a word in a text to a specific meaning, which is distinguishable from other meanings that can potentially be related to that same word. wsd is essentially a classification problem: given an input text and a set of sense tags for the ambiguous words in the text, assign the correct senses to these words. sense assignment often involves two assumptions: a. within a discourse, e.g. a document, a word is only used in one sense and b. words have a tendency to exhibit only one sense in a given collocation – neighbouring words. during the last years, word sense disambiguation has become a hot topic in the biomedical domain. the challenge here is the rapid growth of the biomedical literature in terms of new words and their senses, with the situation getting worse with the use of abbreviations and synonyms. this illustrates the exact need in the case of the biomedical domain; the development of statistical approaches that utilize "established knowledge" and require no or only some parsing of the text in order to perform the correct annotation. two main decision points for wsd in the biomedical domain are the granularity to which wsd should be performed and the selection of an appropriate corpus for training and evaluation. concerning granularity, some tasks are easier than others. concerning the biomedical corpora, those are few, mainly due to the time-consuming and labor-intensive process of manual or semi-automatic annotation. examples of such datasets are the nlm wsd test collection, medstract for acronyms and the biocreative set for mouse, fruitfly, and yeast. however, depending on the task, researchers need to collect their own gold standard datasets. algorithms for word sense disambiguation as shown in table wsd algorithms can be distinguished as supervised, unsupervised, or using established knowledge. in the biomedical domain researchers have focused on supervised methods and using established knowledge to perform gene name normalization and resolve abbreviations. according to the recent biocreative challenge, the former problem can be solved with up to 81% success rate for human genes, which are challenging with synonyms per name. resolving ambiguous abbreviations achieves higher success rates of close to 100%, as the task is less complex when long forms of the abbreviated terms are in the document. the above approaches use cosine similarity, svm, bayes, decision trees, induced rules, and background knowledge sources such as the unified medical language system, medical subject headings, and the gene ontology. two approaches use metadata, such as authors and journal descriptor indexing. most of the unsupervised approaches so far were evaluated outside the biomedical domain, with the exception of, who used relations between terms given by the umls for unsupervised wsd of medical documents and achieved 74% precision and 49% recall. another approach that uses the umls as background knowledge for wsd is that of, who compared the results from a naive bayes classifier and other algorithms to conclude that different senses in the umls could contribute to inaccuracies in the gold standard used for training, leading to varied performance of the wsd techniques. another approach by is based on a graph model representing words and relationships between them and uses wordnet for assigning labels. interestingly, most of the above approaches consider the background knowledge sources as terminologies, without taking into account the taxonomic structure or the terms' semantic similarity. here, we fill this gap by systematically comparing three approaches using ontologies with inference and semantic similarity and the use of metadata to solve the problem of wsd for ontological terms. the goal is to establish how the use of ontologies and metadata can improve results. in previous work, we proposed and evaluated two approaches to the wsd problem, namely term co-occurrences in pubmed abstracts and document clustering. we proposed a methodology for finding whether an article in an automatically annotated database is likely to be true or false with respect to the biological meaning and constructed a co-occurrence graph of go terms based on gene ontology annotations. in the present article we extend this approach in the following ways: first, we additionally disambiguate mesh terms and use a larger training corpus to get the co-occurrence scores, since there exist ~ documents to which experts have assigned mesh terms. second, we make use of the hierarchy structure in both go and mesh, whereas before we used term co-occurrence without any inference. we therefore investigate how two different hierarchies influence the performance of disambiguation. third, we combine our graph-based decision function with a support vector machine, arranged in a co-training scheme, to learn and improve models without any labelled data. finally, we test the disambiguation performance in new larger benchmark datasets of varying curation quality. the'term cooc' approach is similar to dorow's approach, with the difference that we construct the co-occurrence graph based on goa and mesh, which are manually annotated datasets. therefore, our graphs contain only relations between terms that are semantically meaningful in the context of an article. dorow's graph contains all the nouns that co-occur with one another, but in the case of the biological context, we are interested only in a local subgraph of dorow's graph. another difference is that we use established knowledge in go and mesh to draw the nodes and in the different configurations of our method we use a support vector machine and/or incorporate the term relationships in go and mesh. we introduce two more methods for disambiguation, differing from the co-occurrences approach in terms of automation and background knowledge required. the'closest sense' approach computes similarities between the senses of the ambiguous term, the senses of its neighbours and the type of relations that could occur between them.'closest sense' uses the umls semantic network as background knowledge, like, who rely on the context of the ambiguous term in order to compute a score for each sense candidate. this score consists of the number of terms in the document which are related, in the umls, with the different senses of the ambiguous term. in comparison to the cs method, this approach is different in two main points: it does not take advantage of the hierarchies of concepts and relations in the umls and it ignores terms which co-occur with the ambiguous term in the same context but do not have a direct link with it in the umls. the'metadata' method uses maximum entropy for modelling the behaviour of occurrence of contextual terms and phrases in text together with a potentially ambiguous term. the features selected are n-tuples of word stems and metadata such as the journal and document title. the method requires a set of labelled documents for each term to be disambiguated. we evaluate and compare the three strategies for wsd, starting from the unsupervised/automated to the least automated one. the comparison includes each method's requirements and limitations in terms of training data and automation, the behaviour of the methods during the use of different taxonomies as well as comparison against a classical stem co-occurrence approach. we additionally make the benchmark datasets created for the purpose of disambiguation available, since the collection process is time-consuming and labour-intensive. these include manually curated documents of high/medium curation quality for selected go and mesh terms. methods terminology and classification of approaches the types of relations between terms in the gene ontology, the medical subject headings and the unified medical language system semantic network make them completely different knowledge sources. go has a simple structure in the form of a directed acyclic graph and go terms are interconnected via is_a and part_of relations. the semantics of relations used in mesh make it a terminology rather than an ontology. terms in mesh are related through a narrower than b relations, giving users who are interested in bs the option to look at as. the umls is considered to be a terminology integration system comprising over biomedical vocabularies and relations like subclassof or subpropertyof between terms. therefore, it is located in the space between a structured terminology and an ontology. the most popular semantic web formalisms for representing taxonomies, ontologies and terminologies in general are the resource description framework and the web ontology language, with owl being more suitable for ontologies and rdf sufficient for terminologies. the simple knowledge organization systems is an area of work developing specifications and standards to support the use of knowledge organisation systems such as thesauri, classification schemes, subject heading systems and taxonomies within the framework of the semantic web. skos provides a standard way to represent knowledge organisation systems using rdf. lately, there have also been provided owl translations of go and mesh by the responsible consortia. we designed, implemented, and evaluated three wsd methods that we refer to as: closest sense, term cooc, and metadata. these differ as follows: background knowledge closest sense uses the umls semantic network; it represents an abstract as a list of umls terms occurring in the abstract. term cooc uses co-occurrences of terms in go and mesh, built from a curated dataset; it represents a document abstract as a list of go and mesh terms occurring in the abstract. the metadata method uses metadata about the journal and title; it represents a document abstract as n-tuples of word stems and metadata. classification cs uses shortest semantic distance of co-occurrences to sense. tc uses svms and co-occurrences from a training dataset for finding boundary between senses. md uses the maximum entropy to model the behavior of the co-occurrence of contextual words and metadata with the ambiguous term. closest sense method this wsd approach was initially used to address the ambiguity problems in the meatannot system. the main idea of the approach is the following: given a set of different senses of the ambiguous term, the co-occurring terms in the same text and the hierarchy where they belong, decide which sense is true based on the distance to the senses of the co-occuring terms. to clarify this, we can have the following sentence as an example:'i also tracked lipid profiles, hba1c, blood pressure, body mass index, hostility and nicotine use'. the term'blood pressure' can have three senses, namely'organism function','diagnostic procedure' and'laboratory or test result'. the senses of the co-occurring terms are'laboratory procedure','gene or genome','diagnostic procedure','mental process' and'organic chemical'. the sense of'diagnostic procedure' for blood pressure is in average closer to the senses of the co-occurring terms than the other candidate senses. for the example in figure the cs method determines the meaning of'thrush' by examining what appears in the same sentence and/or paragraph and then computing a similarity based on semantic distances to'songbird' and'oral candidiasis' in the umls hierarchy; the highest similarity determines the result. intuitively, with semantic distances, two senses are close if there exists a possibility to use them in a concise annotation graph. algorithm the'closest sense' algorithm takes as input: the ambiguous term τ, the vector vτ of different senses of τ, the vector vcτ of senses found in the context, and the umls semantic network. first, the disambiguator builds a vector vcτ of senses describing the context of the ambiguous term τ. this vector includes the senses of terms that are neighbours of τ. then, it computes the similarity between each sense in vectors vτ and vcτ. the resulting similarity is the average of similarities between senses in the two vectors. finally, the sense in vτ that has the highest average similarity to vcτ is proposed as the best for τ. semantic distances the distance metrics used to find the correct sense are the subsumption distance and the subtype-aware signature distance. the subsumption distance is the length of the shortest path between two senses in the hierarchy of senses, where the length of an individual subsumption link gets exponentially smaller with the depth of the senses it links in the hierarchy. the subtype-aware signature distance is the length of the shortest path between two concepts/terms through the graph formed by the property types with their range links and domain links. with this new semantic distance we merge signature and hierarchies graphs. the main idea is to find a path between two concepts/terms by using the ontology structure and the signature of relations. the subtype-aware signature consists of relations in the hierarchy additional to the common signature. it is aware of the properties of a term, the position of the term in the hierarchy and the hierarchy of the properties. term cooc method the term cooc method relies on selecting the term that most frequently co-occurs with the ambiguous term in the training corpus. it selects the highest co-occurring term with the ambiguous term for defining the given sense as true or false. in order to formalize the notion of term co-occurrences, we consider pairs of go/mesh terms that appear in the same abstract and we represent all such pairs of terms in a manually annotated goa or mesh co-occurrence graph. each node in the co-occurrence graph represents a go or mesh manual annotation. an edge between nodes α and β represents a real number, the log-odds score, representing the frequency log – odds of the terms' α and β co-occurrence over all articles, weighted by their total number of occurrences. for the example in figure the tc method determines the meaning of'thrush' by examining what appears in the same abstract and then considering all known co-occurrences between ontology terms in a training corpus; the value of the highest co-occurrence determines the result, e.g.'swallows' would have relatively high co-occurrence with'thrush' songbird. algorithm first, we use a simple threshold considering how close to an ambiguous term the highest co-occurring term is; if below a user-defined threshold θ, the ambiguous term is negative, else it is positive with respect to the term. second, we use support vector machines trained on all tokens of a text. the method first runs a binary svm against a set of articles ordered by maximum co-occurrence with the ambiguous term. the highest and lowest 10% of articles in the set are labelled as positive and negative; then the svm is trained on lower 10% and upper 10%. after the initial convergence is achieved, the error will be low, likely near the algorithm next improves this result by iteratively re-classifying the remaining articles that have less extreme co-occurrences with the ambiguous term, one-by-one, followed by re-training the svm on the newly relabelled data set. this continues until no more articles are left. the steps are: set s = order articles based on their highest co-occurring go/mesh annotation with the ambiguous term. t = lowest and highest 10% of s; label t as negative and positive; train svm with t; remove t from s. for s ∈ s: move s to t; classify s; re-train svm with t. training with co-occurrence graphs these graphs are used in the tc method for training. for wsd of go terms, the co-occurrence graph derived from the gene ontology annotations, and for the mesh terms from the medical subject headings. goa represents articles manually annotated with go terms and consists of ~ articles. there exist ~, documents to which experts have assigned mesh terms. we found co-occurring terms in the goa and mesh annotated articles and built a co-occurrence graph representing how frequently pairs of go or mesh terms co-occur. nodes represent annotations and edges represent the frequency of co-occurrence of two annotations in the same article, normalized based on each goa/mesh annotation's individual occurrence frequency in the specific corpus. training with inferred co-occurrences we extend the co-occurrences in a hierarchical fashion to ensure that given a goa-derived co-occurrence between a pair of terms, goacooc, the ancestors of α and β in the ontology are updated with the co-occurrence such that only the maximum co-occurrence is kept. this is important given the few annotations in goa and the is_a relationships between go terms, since ancestors inherit the co-occurrences of their children. with the inferred co-occurrences, given an ambiguous term α, the co-occurrence of α with a term β will not be lower than α's co-occurrence with any of β's descendants. metadata method as an alternative method for wsd we use a maximum entropy approach as described in. maximum entropy models have been successfully used in tasks like part of speech tagging, sentence detection, prepositional phrase attachment, and named entity recognition. for the example in figure the md method determines the meaning of'thrush' by using metadata for the document and then deciding based on what was previously learned about this metadata from training examples. the metadata used are n-tuples of word stems from different scopes, namely the paper title, the sentence including the ambiguous term or the whole abstract, the journal title as well as the publication period, since some topics can be popular in different decades. the occurrence of contextual words and phrases in a text together with a potentially ambiguous term can be seen as a random process. maximum entropy modelling aims at modelling the behavior of this random process. provided a large amount of training examples, the algorithm automatically extracts a set of relationships inherent in the examples, and then combines these rules into a model of the data that is both accurate and compact. algorithm the training and test data in our case are sentences containing the potentially ambiguous term flagged with the sense. each training example, one sentence each, is represented as a set of features. an implementation of the porter stemmer is used and as features we select n-tuples of word stems and meta information of the document, such as the journal and title words and the publication period. the implementation takes a series of events to train a model. each event is a configuration of binary relations associated with a label. the resulting model is applied to an unknown configuration of binary relations. the result is the predicted probability for the previously trained outcomes. mesh terms already assigned to the articles are excluded, for the performance evaluation to be independent of them. given the abstract of a scientific article and the ambiguous term, the steps followed are: extract binary features get scalar product of feature vector and model the result is the probabilities for predefined outcomes if above a threshold, the term is true, else false. as an illustrating example of the features extracted, articles mentioning'signal transduction','kinase','embryo','neuron' or'stage' are more likely to refer to'multicellular organismal development' than to another sense, such as development of an algorithm or a disease in an organism. articles mentioning'anxiety','behaviour','memory','social' and'fear' are more likely to refer to'psychological inhibition', instead of'enzyme inhibition'. experimental setup classification task and limitations the disambiguation performed here is mainly a classification task; it represents the prediction whether an annotation is positive or negative with respect to the go/mesh sense. we do not assign one of the numerous different senses to a term, but instead a positive or negative label to it, when it corresponds to the go/mesh sense or not, respectively. we do not handle acronym ambiguity separately. however, in cases where an acronym belongs to an ontology term label, this is disambiguated in the same way as all ontology term labels. as mentioned in the introduction, some disambiguation tasks are easier than others;'bank' the building and the'bank' gene will appear in completely different context, whereas the'bank' gene, protein or mrna are even likely to appear in the same article abstract, making the disambiguation task often difficult even for a domain expert.'transport by air' or'patient transport' will be easier to distinguish from the go sense of transport, but'transport of virus cultures' will appear in a closer molecular biology context. distinguishing between'transport','rna transport','trna transport' or'ion transport' can become less difficult by using the hierarchical information in the ontology. some terms are also easier to disambiguate in the same task, depending on the number of their different senses and the distance between them, the way they appear in text and the number of tokens they consist of. examples of the senses per ambiguous term included in the benchmark dataset collected. the ambiguous terms examined are the go terms'development','spindle','nucleus' and'transport' and the mesh terms'thrush','lead' and'inhibition'. most of the different senses of the terms examined belonged as well to the biomedical domain, making the disambiguation task more difficult. the limited number of terms examined is due to the labor-intensive process of manual collection of proper benchmark datasets. as mentioned in the introduction, there exist few annotated biomedical corpora for evaluation and depending on the task, researchers need to collect their own gold standard datasets. we collected datasets for a list of ambiguous terms based on the amount of true/false data available and the frequency of occurrence in pubmed. we aimed at keeping the ratio of true/false abstracts close to giving a 50% chance to each appearance of the term to be true or false with respect to the go/mesh sense. we first examined the umls wsd collection for ambiguous go/mesh terms and data availability and later a list of common false positive terms based on manual curations in gopubmed. from the umls wsd collection we selected terms that were go/mesh terms and the senses provided were distant to each other, i.e. in the case of'lead', the two senses with short semantic distance were considered as one, as they both are about the compound. a semantically more distant sense is that of the verb to lead/result in. regarding the false/positive ratio limitation/criterion, for some terms this was not satisfied, not allowing the inclusion into the evaluation dataset. for example, for'transport' the umls wsd collection contained abstracts classified as sense and only as other. we therefore needed to manually collect false examples containing other senses for a balanced corpus. another question was whether the definition of the negative datasets would influence the results. to test this, we defined a more general negative dataset by completely randomly choosing articles. defining a random set as a negative set is common practice, e.g. in predicting protein-protein interactions. obviously, the random negative dataset is very different from the positive dataset, since most likely it does not contain any of the negative senses at all, but is just the bias for the "average" paper. results showed a decrease of ~7% in the performance of the methods. this argument can be turned around. while our initial negative dataset was carefully and manually chosen, it could be further improved by letting its composition of other senses reflect the distribution of use of these senses in pubmed as a whole. however, achieving this ideal would require annotating all articles with the term in the whole of pubmed with the senses. given that pubmed has for example more than million articles on development, this cannot be easily accomplished. however, the composition of negative senses in our negative dataset aims to reflect the composition of negative senses in pubmed as well as possible through the query and annotation strategy, that was pursued. since we needed to include every possible sense of the ambiguous terms, the queries formed were such that could collect representative abstracts for each sense, a process that was manual and time-consuming. the collection of the positive examples was easier, since there was one sense and also more frequent in pubmed, therefore the term itself or one of its synonyms were enough to be put in the query to pubmed. the collection of negative examples was as expected harder, since they were not frequent in pubmed and we needed to include enough examples of every possible sense. most of the queries used for this included the ambiguous term or synonyms of it and keywords that were often in the context, based on personal experience from previous curation of automatic annotations in gopubmed. for example, for'development', the queries used were'development and staff','development and algorithm','development and software','development and treatment','development and method', etc. for'thrush', since we could only locate one negative sense, we used queries such as'thrush nightingale','thrush and songbird','mountain thrush', etc. the other aspect is the question of size composition of positive and negative. we chose roughly 50% positive and 50% negative. this basically means that the a priori likelihood is 50% for the corect sense. if, instead, we aim to identify each sense correctly, the following problem arises: assume there are ten senses, i.e. positive and negative. then the a priori probability for the classification would be 10% and then a simple strategy would be to always vote negative. overall, the approach pursued plus equally weighted positive and negative datasets is a suitable approach for evaluation. datasets we collected three different benchmark datasets to evaluate the performance of the three methods. they differ in quality and quantity, depending on their collection process. the common reference dataset between the three methods is the manually annotated by a domain expert one: the above datasets contain manually collected pubmed articles by one expert, manually curated articles by a group of non-experts and semi-automatically collected articles. see datasets section for details. high quality, low quantity corpus this corpus consists of ~ true and false example documents per ambiguous term. for the ambiguous go terms examined and the mesh term thrush we collected both true and false examples manually. true examples are abstracts that discuss, for instance,'development', in the sense specified by go. false examples also contained the ambiguous term, but in other senses, closer or not. for the ambiguous mesh terms'lead' and'inhibition', the test set originated from the umls wsd corpus. these two were the only terms depicting mesh terms. all other terms in the umls wsd were only found in go or mesh as substrings and would thus not be contained in either co-occurrence graph as single nodes. medium quality, medium quantity corpus this corpus consists of documents for which the annotation has been manually confirmed by a group of expert and non-expert curators. we asked colleagues to confirm or reject the automatic annotations provided by gopubmed for a collection of article abstracts. this collection has been mainly automatically created, as described next. for each of the automatic annotations, the curators could select among three options: a. true and important for the context of the publication, b. of minor importance/relevance and c. false annotation. the curation tool is available via gopubmed. low quality, high quantity corpus this corpus was created mainly automatically. we implemented similarity-based clustering of abstracts with literal occurrence of the ambiguous terms. each abstract was matched to its nearest abstract, conceptualized as a directed edge from the former to the latter. then every connected component was considered as a cluster. from an initial manual evaluation of the clustering results, clusters of size > were consistent enough, meaning that articles in such clusters were referring to one sense of the ambiguous term in 72–95% of the cases. each cluster's abstracts were input into a system developed in-house that generated a list of terms describing each cluster based on term frequency inverse document frequency. the top terms of the list were later evaluated by an expert which labelled the clustered articles as true or false for the respective go/mesh term. the above facilitated and accelerated the dataset collection process without any significant loss in data quality. experiment for evaluation and comparison purpose, each method's performance was tested on the high quality/low quantity dataset. we also applied classical stem co-occurrence analysis as a baseline on the same dataset; this consisted of basic maximum entropy modelling on stems without any use of metadata or hierarchical information. we additionally tested each method's performance separately with different test datasets. for the'term cooc' method, the performance of co-occurrences of go/mesh terms and inferred co-occurrences of go/mesh terms was tested in the three benchmark datasets described earlier, in order to evaluate the method in larger datasets, since it has been shown that sample size, sense distribution and degree of difficulty impact on the classification task. input to this method were the automatic annotations per article provided by gopubmed and the respective co-occurrence graph. as a side experiment, we tested the tc method for the disambiguation of mesh terms without including the mesh hand annotations in the automatic annotations provided by gopubmed, to estimate how the quality of the input influences the quality of the results. for the'closest sense' method, input was the umls semantic network and the article abstracts. this method was additionally tested on the wsd test collection. for the'metadata' method, we used the three different datasets for training and testing. the high quality dataset was used in an initial experiment as training and testing dataset, in a 5-fold cross validation. then the medium quality and low quality datasets were separately used as training sets, with testing of the method on the high quality dataset. cs column contains the results for the closest sense approach with the use of the classic distance. cs column contains the results for the cs approach with the use of the optimized signature together with the subsumption distance. tc and tc contain the results of the term cooc approach, when the co-occurrences or the inferred co-occurrences are used, respectively. tc contains the results for the tc approach with co-occurrences and support vector machines, and tc when inferred co-occurrences and svms are used. bme column contains the results for the baseline method, trained and tested on the high quality corpus in a 5-fold cross validation. md is for the metadata approach, trained and tested on the high quality corpus in a 5-fold cross validation. md is trained on the medium quality/quantity corpus and tested on the high quality one. md was trained on the low quality/high quantity corpus and tested on the high quality corpus. some terms are easier to disambiguate than others. overall, all methods perform well between 73–96% f-measure ).  the performance of the three disambiguation approaches and the baseline was tested on a common high quality/low quantity dataset. the overall results of this comparison are shown in table. all methods perform well between 73–96% average f-measure. in particular, the metadata approach is the best one: when trained on high quality data, it achieves 96% f-measure. when the metadata are not used the accuracy falls to 90%. the term cooc method follows with 81% and the closest sense approach with 77%. all methods present low f-measure for'development' and'lead'. the best results are obtained for go terms'transport','nucleus' and'spindle'. as far as the closest sense approach is concerned, there is a clear improvement in the results with the use of the optimized signature together with the subsumption distance. for the term cooc approach, when the inferred co-occurrences are taken into account in the case of the go terms the results remain the same, whereas in the case of the mesh terms the results are worse, mostly in terms of recall. for go terms, the results are best when inferred co-occurrences are combined with svms, whereas for mesh terms, the best f-measure is achieved when co-occurrences with svms are used, without the inferred co-occurrences. this difference can be explained by the different structure of the two hierarchies. go can be described as "tall and thin", whereas mesh is "short and fat". additionally, the relations between terms in mesh are not exact is_a relations, but rather is_related_to. therefore, propagating the term co-occurrences in mesh does not improve the results, since it does not necessarily mean that annotating with term meshx also means all of x's ancestors. on the contrary, in go this is more likely to hold. the metadata method gives – as expected – the best results. when the method is trained and tested on the same high quality test, it results in an average f-measure of 96%. when trained on the medium quality and low quality corpora and tested against the high quality corpus, the f-measure decreases into 81% and 70%, respectively, which are nonetheless high, compared to the quality of the training sets. the high performance of the metadata approach is mainly due to the use of metadata as the title of the abstract and the journal. for example, for the terms'inhibition' and'spindle' it achieves 100% f-measure and for'nucleus' 99%. the true sense of inhibition for mesh is psychological inhibition, which is easier to disambiguate, since it will mostly appear in psychology/psychiatry journals. the same applies for'spindle', which will mostly occur in cell biology and cell division/cycle journals. we additionally tested each method's performance separately with different test datasets. the'closest sense' method was also tested on the nlm umls wsd collection to compare four versions of semantic distance computation in order to disambiguate term mapping to the umls semantic network. the experiment showed that the use of the ontology definition can improve significantly the precision. over the ambiguous terms examined, the overall average precision was 83%. for the'term cooc' method, the performance of the different variants was tested in the three benchmark datasets described earlier, in order to evaluate the method in larger datasets. testing the method from the highest towards the lowest quality, the f-measure decreases only by 3–10%, indicating a consistent behavior of the method. as a side experiment, we tested the'term cooc' method for the disambiguation of mesh terms without including the mesh hand annotations in the automatic annotations provided by gopubmed, to estimate how the quality of the input influences the quality of the results. as expected, the results decreased dramatically, indicating that the mesh hand annotations provided per article are important for the disambiguation. discussion overall, the md method gave the highest f-measures among all methods. the results became worse for the medium and high quantity datasets, since these were of lower quality in terms of correctness. the md approach's consistency with giving the highest results is due to integrating metadata, such as journal and title, which are representative of the true meaning of an ambiguous term. the md approach needs plenty of labelled data for training. when comparing the results of the tc and cs methods to the baseline method that performs only maximum entropy modelling of stems, bme still gives better results, but this is due to the available training data of high quality. the disadvantage of md and bme compared to tc and cs is the need of high quality training data. the md approach is less scalable in terms of storage demands as the number of articles increases, while the cs and tc approaches have constant storage demands. in the tc method the svms increase the results up to 98%. the tc method requires an ontology and co-occurrence graphs. the origin of this graph should be a manually curated data source, in our case goa and mesh. the quality of the graph will heavily depend on its origin and quality of the data. the inferred co-occurrences improve the results for go, while for mesh they get worse. this is due to the different structures of the two semantic hierarchies; the ancestors of an applicable go term are more likely to also be applicable to the same article, because of go's structure that is "tall and thin". but mesh's structure is "short and fat" and is not always a thesaurus; not all of a node's ancestors are also applicable. moreover, in the tc method the inferred co-occurrences only improve the result if combined with the svm. this is because the inferred co-occurrences make the extreme co-occurrences with the ambiguous term, which the svm uses for training, more representative of an ambiguous term's true meaning. figure shows that the most extreme co-occurrences with the ambiguous term are most likely to be classified correctly, since the inferred co-occurrences make more precise the highest and lowest co-occurrences with an ambiguous term. the middle co-occurrences are not necessarily made more precise with inferred co-occurrences. that is why inferred co-occurrences help with the svm training; while later on for middle co-occurrences the errors accumulate. the cs approach needs only a semantic hierarchy in the form of an ontology, and in this sense is the most automated of the three methods. moreover, cs gives good results, where the only problematic term is'lead'. however, cs is sensitive to the design of the ontology or subdomain of umls used, which reflects the view of the designers. as shown by the accuracy of, umls may not be the best choice to be used as background knowledge as the different parts of the hierarchy are modelled differently, resulting in different granularity. different groups of people design ontologies differently; the various subdomains of an ontology will reflect the designers' views respecting depth, number of nodes, and structure. therefore, the subdomains of the ontology influence the performance of the cs method, and the design rationale of the ontology may be ultimately responsible for performance differences on various terms. for example,'nucleus' is a subtree root in both go and snomed; in go there are descendants of nucleus, while in snomed  based on the results, metadata and training data of high quality seem to be the key point for the increase of the accuracy. when such training data are not available – as happens in most of the cases – co-occurrence of ontology/taxonomy/thesaurus terms can provide the way to the right decision. moreover, the hierarchy of the terms and the subdomain, when consistently modelled, can depict the correct sense of an ambiguous term. the md method produced the best results by including metadata in the wsd decision, but it requires high quality training data. the most interesting thing about the tc and cs methods is that they are semi-automated, given a co-occurrence graph or ontology; then the training does not require manual intervention. tc requires well modelled ontologies such as go, and deteriorates as the structure becomes less rigorous as in mesh. cs requires large and consistently modelled ontologies, which are two opposing requirements. thus, for tc and cs the structure of the ontology and subdomain affect the distance metric used and wsd quality. future work will include identifying ambiguous terms for a certain corpus automatically. for this purpose, we will employ wordnet, clustering, part of speech and noun phrase statistics, and expert input. for tc and cs, we assumed that the other terms in the context are correct and independent of one another; in fact, they could also be ambiguous and therefore false. for cs we will optimize the distance computation and propose other distances, taking into account existing annotation bases and ontology structure. so far, the disambiguation performed was between the true sense in the hierarchy and all other senses that were considered as false. a possible extension of the methods would be to correctly identify if a sense occurs that is not included in the thesaurus/ontology and possibly add it. the closest sense method can potentially do this by setting a threshold. from all distances below this certain threshold, one should be clearly shortest. if not, then this indicates a new sense. the term cooc and metadata approaches could be adjusted to identifying new senses by training each method on each sense and setting a certain threshold. if the sense found is not above the threshold, then this can be a new sense. abbreviations wsd: word sense disambiguation; go: gene ontology; goa: gene ontology annotations; mesh: medical subject headings; umls: unified medical language system; snomed: systematized nomenclature of medicine; svm: support vector machine; tfidf: term frequency inverse document frequency; cs: closest sense method; tc: term cooc method; md: metadata method; bme: basic maximum entropy. authors' contributions da and ba developed the term cooc method with the assistance of jh. ad implemented the metadata method and kk and fg the closest sense method. tw implemented the terminology extraction method used in data collection. hd assisted in data collection. da performed the manual collection and evaluation of the benchmark datasets and drafted the manuscript. ms supervised and coordinated the project. all authors have read and accepted the final manuscript. supplementary material additional file 1 benchmark datasets used in the experiments. the three corpora are given in the form of pubmed identifiers for true/false cases for the ambiguous terms examined. click here for file additional file 2 detailed results for all methods. detailed results for all methods given in terms of precision, recall, specificity and f-measure. additional training and testing of the metadata method in all three corpora, closest sense results with different threshold, term cooc results with mesh text-mined annotations only, closest sense results on terms from the wsd test collection. click here for file acknowledgements the authors acknowledge the eu sealife project and the curators: a. tuukkanen, m. reimann, a. elefsinioti, a. marsico, l. royer, m. altrichter, c. winter, r. winnenburg, a. simeone, e. fritzilas.
it computes the shortest path of co-occurring terms in the document to one of these senses. to evaluate these approaches we defined a manually curated training corpus of documents for seven ambiguous terms from the gene ontology and mesh. the'term cooc' approach performs better on gene ontology than on mesh as mesh is not a strict is-a/part-of, but rather a loose is-related-to hierarchy. ontology term labels can be ambiguous and have multiple senses. metadata is valuable for disambiguation, but requires high quality training data. however, most treat ontologies as simple terminologies, without making use of the ontology structure or the semantic similarity between terms. overall, the results show that well structured ontologies can play a very important role to improve disambiguation. the'closest sense' method assumes that the ontology defines multiple senses of the term. the'closest sense' approach achieves on average 80% success rate. the'term cooc' method defines a log-odds ratio for co-occurring terms including co-occurrences inferred from the ontology structure. it does not require any ontology, but requires training data, which the other methods do not. its performance deteriorates as quality of the training data decreases. term cooc achieves greater 90% success given a consistently modelled ontology. the three benchmark datasets created for the purpose of disambiguation are available in additional file classical approaches to word sense disambiguation use co-occurring words or terms. here, we systematically compare three approaches to word sense disambiguation, which use ontologies and metadata, respectively.
 ontology term labels can be ambiguous and have multiple senses. while this is no problem for human annotators, it is a challenge to automated methods, which identify ontology terms in text. classical approaches to word sense disambiguation use co-occurring words or terms. however, most treat ontologies as simple terminologies, without making use of the ontology structure or the semantic similarity between terms. another useful source of information for disambiguation are metadata. here, we systematically compare three approaches to word sense disambiguation, which use ontologies and metadata, respectively.  the'closest sense' method assumes that the ontology defines multiple senses of the term. it computes the shortest path of co-occurring terms in the document to one of these senses. the'term cooc' method defines a log-odds ratio for co-occurring terms including co-occurrences inferred from the ontology structure. the'metadata' approach trains a classifier on metadata. it does not require any ontology, but requires training data, which the other methods do not. to evaluate these approaches we defined a manually curated training corpus of documents for seven ambiguous terms from the gene ontology and mesh. all approaches over all conditions achieve 80% success rate on average. the'metadata' approach performed best with 96%, when trained on high-quality data. its performance deteriorates as quality of the training data decreases. the'term cooc' approach performs better on gene ontology than on mesh as mesh is not a strict is-a/part-of, but rather a loose is-related-to hierarchy. the'closest sense' approach achieves on average 80% success rate.  metadata is valuable for disambiguation, but requires high quality training data. closest sense requires no training, but a large, consistently modelled ontology, which are two opposing conditions. term cooc achieves greater 90% success given a consistently modelled ontology. overall, the results show that well structured ontologies can play a very important role to improve disambiguation. availability the three benchmark datasets created for the purpose of disambiguation are available in additional file 