BACKGROUND
large-scale information extraction  from scientific literature is increasingly used to support advanced knowledge management and discovery systems  <cit> .
the utility of such systems depends on the quality of the extracted information.
manually annotated gold-standard corpora are critical for evaluating the accuracy and usefulness of information extraction systems  <cit> .
in the biomedical domain, various corpora annotated for semantic phenomena have been constructed in recent years; annotations range from named entities  <cit> , to semantic relations, such as protein-protein interactions  <cit> , protein/gene/rna relationships  <cit> , disease-treatment relations  <cit> , clinical relations  <cit> , biological events  <cit> , and gene regulation events  <cit> .
more recently, the notion of "silver standard" has also been introduced  <cit> , referring to harmonization of automated system annotations, as a proxy to labor-intensive gold standard annotation.
the gold standard corpora have often focused on text drawn from a narrow subdomain, adopting a particular semantic representation, addressing a small set of semantic types and aiming to provide training and evaluation support for specialized ie systems.
these corpora differ with respect to their level of granularity and whether there is an ontological basis to the entity and relationship types used.
for example, one of the most popular corpora in recent years has been the genia event corpus  <cit> , drawn from the scientific literature on transcription factors in human blood cells.
it is based on the notion of biological events, uses a few dozen gene ontology   <cit>  event types and has been the basis for recent biological event extraction systems as well as two bionlp shared task competitions  <cit> .
the generally narrow focus of such corpora and their specific representation formalisms render them largely unsuitable for evaluating ie systems using different formalisms or resources.
we have been developing a semantic interpreter, semrep  <cit> , which extracts content from biomedical text in the form of semantic predications.
a semantic predication is a logical subject-predicate-logical object triple whose elements are drawn from the umls knowledge sources  <cit> ; the subject and object pair corresponds to umls metathesaurus concepts and the predicate to a relation type in an extended version of umls semantic network.
while the umls semantic network has not been designed as an ontology in a strict sense, the extended version that semrep uses  <cit>  serves as an ontological resource: it defines a domain model consisting of concept types , relation types  and the relationships that can hold between concept types .
each semantic predication extracted by semrep is an instantiation of an ontological predication.
we refer to this extended version of the umls semantic network as the semrep ontology henceforth.
semrep extracts a range of predicates relating to clinical medicine , substance interactions , genetic etiology of disease , and pharmacogenomics .
for example, the program identifies the semantic predications in  from the input text in .
arguments of the semantic predications  have the form conceptidentifier: conceptname .
textual mentions corresponding to the arguments are in bold, and those corresponding to the predicates are underlined in .
mri revealed a lacunar infarction in the left internal capsule.
c0024485: magnetic resonance imaging -diagnoses-c0333559: infarction, lacunar
c0152341: internal capsule -location_of-c0333559: infarction, lacunar
semrep processing is supported by an underspecified syntactic analysis based on the umls specialist lexicon  <cit>  and the medpost part-of-speech tagger  <cit> .
metamap  <cit>  is used to map simple noun phrases to umls metathesaurus concepts.
entrez gene  <cit>  serves as a supplementary source to the umls metathesaurus with respect to gene/protein terms, which are identified using abgene  <cit> , in addition to metamap.
indicator rules map syntactic phenomena, such as verbs, nominalizations, prepositions, and modifier-head structure in the simple noun phrase, to ontological predicates from the semrep ontology.
semrep currently uses the 2006aa release of the umls knowledge sources, due to the prevalence of ambiguity in later releases.
the lack of a suitable, manually annotated gold standard corpus has so far precluded a formal evaluation of semrep ; system improvements and modifications have been informally evaluated through error analysis.
a formal evaluation requires conceptual annotation with respect to the umls; that is, text fragments need to be mapped to concepts and relations in the umls, which provides a formal representation of domain knowledge.
considering that the umls metathesaurus, the basis for conceptual information, consists of  <dig> source vocabularies and more than  <dig>  million concepts , it is clear that such conceptual annotation is an extremely challenging task.
large-scale conceptual annotation is not generally attempted in the biomedical domain.
in fact, apart from the recent craft corpus  <cit>  and the clef corpus  <cit> , we are not aware of any such annotation work.
the ontology-based semantic annotation of the craft corpus  <cit>  concentrates on biomolecular entities and processes, including gene/gene products, chemicals, sequence types, molecular functions, and cellular components.
ninety-seven full-text articles were annotated with concepts from eight terminologies, six of them from the obo library  <cit> .
they report that relationship annotation between concepts is ongoing work.
intended for clinical ie research, the clef corpus  <cit>  annotates information about clinical entities and the relations between them, along with temporal information.
it is limited to clinical notes and reports of deceased patients who had a diagnosis of neoplasms.
clef builds on a relevant subset of umls concepts and relations as domain knowledge.
some semantic types are conflated and relations  renamed in accordance with the goals of the project.
for instance, clef semantic type condition includes symptoms, complications, functions, diagnosis, and problems, conflating several umls semantic types.
in contrast to these manual conceptual annotation efforts, jimeno et al.
<cit>  semi-automatically created a small corpus annotated with disease concepts from umls metathesaurus.
their semi-automatic methodology involves domain expert assessment of disease concepts identified by metamap, a dictionary lookup method, and a statistical method.
in this article, we present an annotation study in which we annotated  <dig> sentences from medline abstracts with  <dig> semantic predications.
our annotation follows the conceptual annotation paradigm and adopts the entire umls as the domain model.
furthermore, we do not limit ourselves to text from a specific subdomain or adopt specific terminologies, in contrast to more focused efforts such as clef and craft.
while our methodology bears some similarities to the clef methodology in terms of the domain model and the guidelines, we use fine-grained umls semantic types, rather than coarse semantic groups, allowing for more flexibility.
with respect to our ongoing research, the resulting gold standard reference is mainly intended to  facilitate comparison between semrep releases and  guide further system development by allowing annotators to add comments and notes in a standardized manner so that problem areas can be identified.
several limitations of the gold standard reference may include its relatively small size, its sentence-bound annotation, and its binary, umls semantic network semantic relation formalism, arguably not as rich a semantic representation as predicate-argument structures with semantic roles .
while small corpus size may present challenges for learning purposes, it serves well for our primary goal of evaluation.
on the other hand, the semantic predication representation  has been shown to be simple, intuitively accessible, and tractable for large-scale knowledge discovery applications  <cit> .
furthermore, this representation lends itself readily to the semantic web and linked data movements, which aim to encode knowledge in subject-predicate-object triples for large-scale automatic reasoning  <cit> .
considering the challenges posed by this task, we believe our corpus presents a good first step in a larger-scale conceptual/relational annotation.
it can serve as a gold standard reference for evaluating umls-based relation extraction systems and the lessons learned can provide guidance for future efforts in this area.
methods
we conducted the annotation study in three phases: a) practice annotation phase, b) main annotation phase and c) adjudication phase.
before explaining these phases in more detail, we briefly discuss two fundamental aspects of our study: annotators and interannotator agreement.
annotators
three annotators, all authors of this paper, were involved in the annotation process.
the annotators have diverse backgrounds; annotator a is a linguist, b a computer scientist and c a physician/biomedical informatics researcher.
all three annotators have natural language processing experience and are experts in the semrep methodology as well as the umls knowledge sources.
using domain experts as annotators is often considered a good strategy to ensure validity and reliability of annotation.
however, the tendency of domain experts to rely on inference due to their background knowledge has also been noted  <cit> .
when annotation is concerned with a relatively narrow biomedical subdomain, it is generally feasible to recruit domain experts as annotators.
however, several aspects of our annotation study make it difficult to find such annotators.
first, we do not focus on a narrow biomedical subdomain.
the consequence is that we need to either recruit experts who are knowledgeable in almost all aspects of biomedicine, or to find tens of annotators who can annotate different sentences on topics of their expertise.
neither option seemed feasible within the scope of our annotation.
furthermore, the two non-physician annotators  work with biomedical text on a full-time basis, and they expressed comfort with annotation after the practice phase.
secondly, in our annotation study, umls expertise is perhaps as crucial as domain expertise, and all three annotators are intimately familiar with the umls knowledge sources.
recruiting domain experts who are also familiar with the umls would clearly be even more challenging.
we believe that our small team of annotators with their expertise in umls and our multi-phase annotation methodology allows us to strike a balance between annotation reliability and validity.
interannotator agreement
the common approach to calculating interannotator agreement on classification tasks is to use kappa  statistic  <cit> , defined as κ =  - pr)/), where pr is the relative observed agreement between annotators, and pr is the chance agreement.
calculating κ for semantic predication annotations is non-trivial, since this type of linguistic annotation is not a simple classification task.
semantic predication annotation task can be decomposed into several subtasks for which agreement can be measured independently:  finding relation indicators,  finding textual mentions of arguments,  mapping the relation indicators to ontological predicates, and  mapping the textual mentions of arguments to concepts.
with the exception of subtask , the space of possible annotations is either not clearly defined or very large, making the calculation of pr and therefore κ, challenging.
for example, consider subtask .
one way to calculate κ for this subtask is to impose constraints on what can be annotated as an indicator.
for example, only verbs may be considered as indicators and the number of verbs in a sentence can be used to calculate chance agreement pr.
we did not impose such constraints since we aimed for breadth of coverage in our annotation study.
this leads to an explosion of possible indicator annotations and to the case in which pr essentially approaches zero.
in such cases, it has been shown that κ approximates f-measure among pairs of annotators  <cit> .
based on these observations and in line with annotation studies that share similarities with ours  <cit> , we adopted f-measure for interannotator agreement.
between two sets of annotations, we calculated it as the f-score of one set of annotations, when the second is taken as the gold standard.
practice phase
for the practice annotation phase, we randomly selected  <dig> sentences  from  <dig> medline abstracts published in the last  <dig> years.
these sentences were extracted from the database of semantic medline  <cit> , a web application to manage the results of pubmed searches, and were manually checked by the first author to ensure correct sentence segmentation.
all three annotators participated in the practice annotation phase, each annotating the same set of  <dig> sentences.
given the annotators' familiarity with semrep, minimal guidance for annotation was provided at this phase and the annotators were asked to annotate semantic predications expressed in the sentences, the textual mentions of their arguments and the predicate and the indicator type , preposition , nominalization , participle , etc.).
to lighten the burden of finding an appropriate umls metathesaurus concept corresponding to a textual mention, umls metathesaurus concepts were extracted from these sentences using metamap  <cit>  and were provided to the annotators .
the guidance at this phase consisted of the following items:
<dig>  a list of core ontological predicates relevant to semrep and their definitions from the umls.
for ontological predicates that are not part of the official umls semantic network but are in the semrep ontology, we used our own definitions.
we provide these definitions in the appendix.
<dig>  an inventory of predicate types , also provided in the appendix.
<dig>  a sample sentence annotation, provided in the appendix and illustrated in figure  <dig>
<dig>  basic instructions consisted of the following:
a) annotation should be restricted to semantic predications involving the core ontological predicates that are provided.
other predicates, even though they may be legitimate, should be ignored.
on the other hand, the annotators are not restricted to follow the ontological predications that exist in the semrep ontology.
b) the umls concepts extracted by metamap are not necessarily the best possible mappings.
in addition, metamap may be unable to find a mapping.
when in doubt, the annotator should try to find a concept that better matches the text  using the umls terminology services  or entrez gene  <cit> , keeping in mind that semrep currently uses the 2006aa version of the umls knowledge sources.
c) the annotation should be text-bound.
that is to say, domain knowledge and inference should play a minimal role in annotation and the annotator should be concerned with what is explicitly stated in the text.
to ensure this, the annotator should explicitly indicate the textual mentions that provide the basis for the annotation , as well as the indicator type for the annotation.
providing only minimal guidance, we aimed to identify the major challenges in annotating predications and find ways to deal with them in the main annotation phase.
this first phase was collaborative: the annotators were free to discuss sentences, concerns and difficulties and develop solutions.
we chose not to use a particular annotation tool, since none of the existing tools fully met our needs, especially with respect to access to terminological resources.
we decided not to develop a study-specific, in-house annotation tool due to time constraints.
instead, the annotators were instructed to simply type semantic predications in a text document, along with the textual mentions that trigger the predication components.
then, based on the results of the first phase, annotators were provided scripts that recognized formatting, spelling errors, and inconsistencies in annotation.
these scripts were used by annotators in subsequent phases and helped them resolve such errors.
the practice annotation phase was completed in three weeks.
after this phase concluded, the first author analyzed each annotation set to identify annotation patterns.
the analysis results were then discussed among the annotators and served to refine the guidelines for the second phase.
we present these refinements in the results section.
we also computed a baseline interannotator agreement between pairs of annotators.
to assess agreement with respect to textual mentions vs. conceptual information, we calculated interannotator agreement using two criteria:
a. strict equivalence criterion, where for two semantic predications to be considered equal, their subject-predicate-object triples must match exactly.
b. relaxed equivalence criterion, where the exact match of the textual mentions of the arguments and of the predicate that establishes their relationship are considered sufficient for equality.
in other words, conceptual match is not required between predication elements.
main annotation phase
in the main annotation phase, two annotators  annotated a set of  <dig> randomly selected sentences drawn from  <dig> medline abstracts from the past  <dig> years .
similar to the practice phase, sentences were drawn from the semantic medline database, were checked for integrity, and umls metathesaurus concepts extracted by metamap were provided for reference .
the annotators worked independently in this phase, which concluded in eight weeks.
based on our observations from the practice phase, we extended the equivalence criteria underlying interannotator agreement in two ways:
a. predication equivalence : a pair of distinct semantic predications may be considered equivalent under specific conditions when one inverts the arguments of the other and the predicates correspond to certain types.
for instance, a predication x-location_of-y may be considered equivalent to y-part_of-x predicates when arguments  correspond to biomolecular entities.
b. gene/gene product correspondence : a pair of concepts may be considered equivalent when one corresponds to a gene term and the other corresponds to its gene product.
for instance, the concept "c0287531: dusp <dig> protein, human"  is considered equivalent to "c1333257: dusp <dig> gene" .
we also assessed the effect on annotation of domain knowledge provided to the annotators.
for this purpose, we distinguished between two types of domain knowledge, conceptual and relational, and measured interannotator agreement on a subset of predications based on the following availability criteria, illustrated in  below:
a. availability of conceptual knowledge : a predication fulfills this criterion if both the subject and object arguments were extracted by metamap and, thus, were provided to the annotators.
b. availability of relational knowledge : a predication fulfills this criterion if it is sanctioned by the semrep ontology.
in other words, it corresponds to an existing ontological predication.
c. availability of conceptual and relational knowledge : a predication fulfills this criterion if it satisfies both  and , the previous two criteria.
for illustration, consider the sentence fragment in .
relevant concepts identified by metamap are given in , and an annotated predication in .
... udp-glc is required in the synthesis of proteoglycans.
...
c0041986: uridine diphosphate
c0008556: chromatography, gas-liquid
c0033692: proteoglycan
c0041988: uridine diphosphate glucose -produces-c0033692: proteoglycan
while the predication in  is correct, it does not fulfill criterion  above because metamap fails to identify the subject "c0041988: uridine diphosphate glucose ," an argument absent in .
it also fails to fulfill criterion  because the corresponding ontological predication "biologically active substance-produces-amino acid, peptide, or protein" is not licensed by the current semrep ontology.
since neither criterion  nor  are met, it follows that the predication does not fulfil criterion , either.
as a result, the predication in  is not included when interannotator agreement calculation is restricted by any of the available domain knowledge criteria above.
adjudication
finally, the third annotator  examined the annotations provided by each annotator and adjudicated them to create the current gold standard.
during this phase, annotator c was free to discuss the annotations with the actual annotator to understand his/her reasoning.
this phase concluded in four weeks.
RESULTS
in this section, we provide detailed information regarding different phases of the annotation, including the number of semantic predications annotated, their distribution by ontological predicate types, and indicator types.
we report strict and relaxed interannotator agreement using various equivalence criteria and domain knowledge perspectives.
for the main annotation phase, we measure interannotator agreement at the ontological predicate and ontological predication level, highlighting some of the annotation difficulties with respect to biomedical subdomains.
examining annotation differences in the practice phase, we refined our guidelines for the main annotation phase and extended interannotator agreement measures, which we also discuss in this section.
practice phase
in this first phase,  <dig> sentences were annotated, with an average of  <dig>  semantic predications per sentence.
we show the number of predications annotated by each annotator at this phase in table  <dig>  and the most frequently annotated ontological predicates with their annotation frequency in table  <dig>  indicator types that signal predications are provided in table  <dig>  which shows that verbal predicates are considered at similar rates among annotators, while there are larger gaps in consideration of other types.
interannotator agreement was fair to moderate in this phase, as shown in table  <dig>  the highest agreement occurred between annotators a and c , consistent with the largely similar patterns they exhibited in the types of ontological predicates they annotated, as shown in table  <dig>  unsurprisingly, the interannotator agreement increases, albeit at different rates, when equivalence of textual mentions is also considered as the basis for agreement  as well as the conceptual equivalence .
refining the guidelines
the results of the practice phase helped us identify several aspects of annotation that are challenging.
these were discussed among the annotators and several additions and clarifications were made in the annotation guidelines.
these included the following:
selecting appropriate umls metathesaurus concepts
as stated earlier, we provided the annotators umls metathesaurus concepts identified by metamap for reference.
after the first phase, our analysis revealed that annotation behavior was diverse with regard to these metamap-provided concepts.
one annotator almost exclusively relied on them, while the other two more extensively utilized the umls to find more adequate concepts.
in addition, one of the annotators considered metathesaurus concepts from newer umls releases.
in the light of discussions over these differences, annotators were instructed to use the following decision criteria in selecting an appropriate concept:
<dig>  if the concept identified by metamap adequately describes the textual mention, use it.
<dig>  a concept that is clearly more general than the one stated in the text  relation holds between them) cannot be used as a replacement.
<dig>  if metamap does not associate any concept with the textual mention or if the associated concept seems inadequate, try to find a umls 2006aa concept that is appropriate.
<dig>  when there is no corresponding umls 2006aa concept, a concept from a newer umls release may be used, provided that it does not violate the decision criterion  above.
finding entrez gene terms
semrep uses entrez gene  <cit>  as a supplementary vocabulary to umls metathesaurus when it identifies gene/protein terms.
the mapping to entrez gene terms is achieved via pattern matching in semrep.
when available, these symbols were also provided to annotators in the practice phase.
similar to their reactions to metamap-supplied umls concepts, annotators also relied on entrez gene terms to varying degrees.
one annotator tried to find entrez gene terms for all gene/protein mentions, whether or not a corresponding umls concept was found, and disambiguate model organisms using the context surrounding the textual mention, while another completely ignored entrez gene terms.
we identified two issues regarding these terms:  when is it required to annotate a textual mention with an entrez gene term?
is disambiguation with respect to model organisms necessary?
we resolved these issues by concluding that the umls is the primary knowledge source for semrep, and a gene/protein mention needs to be annotated with an entrez gene term only if no corresponding umls concept is found.
further, we decided that entrez gene terms should be limited to those in the homo sapiens taxon, since semrep currently only considers this taxon, and context beyond individual sentences may need to be taken into account for determining the model organism.
semantic network ontological predicates
the practice phase revealed some confusion with respect to the difference between an ontological predicate and its textual mention.
for instance, for the fragment "... an association of 5-httlpr with intensity dependence of auditory-evoked potentials...", two annotators annotated the semantic predication "c0170657: serotonin transporter -associated_with-c0015215:auditory evoked potentials ."
the ontological predicate associated_with is used in semrep in a restricted sense, referring to only gene-disease association.
although the semantic type of the object in the predication is not a disorder, the annotators used this ontological predicate because they were influenced by the choice of textual mention "association."
the difference was made explicit in the guidelines.
we further clarified the definitions of several ontological predicates, taking into account how they are conceptualized in the semrep ontology.
for instance, we noted that comparative predicates  are limited to substance and therapeutic procedure semantic types for the time being, while process_of is limited to disorder subjects.
we also distinguished interacts_with/affects and inhibits/disrupts predicate pairs more explicitly: interacts_with and inhibits relations hold between substances, while affects and disrupts take processes as objects.
hypernymic relations
there was disagreement over what constitutes a hypernymic  relation.
problematic annotations included "c0050940: lansoprazole -isa-c0599473: enantiomer " for the noun phrase "lansoprazole enantiomers" and "c1443775: epidermal growth factor receptor inhibitor -isa-c0450442:agent " for the fragment "two oral egfr inhibitors ... are small-molecule agents ...." we concluded our discussion on this topic by distinguishing between taxonomic relations that pertain to structural aspects  and other kinds of taxonomic relations, including those pertaining to functional aspects .
it was clarified in the guidelines that structural taxonomic relations are not hypernymic.
extending interannotator agreement calculation
as briefly mentioned earlier, based on analysis of the agreement results in the practice phase, we extended equivalence criteria for interannotator agreement calculation.
we describe these extensions in more detail in this section.
equivalence of semantic predications
we found that two distinct semantic predications derived from the same textual mentions may be too close in meaning to select one over the other.
for instance, we identified cases where one annotator used a predication with the predicate location_of, while the other preferred one with the predicate part_of and inverse arguments.
for the textual fragment "... alleles in two cell lines", one annotator annotated "c0002085: alleles -part_of-c0007600: cell line ", while the other annotated "c0007600: cell line -location_of-c0002085: alleles ."
such similarity in meaning is partly rooted in the umls semantic network.
for instance, the ontological predication "virus-location_of-biologically active substance" is a valid ontological predication, as is one with inversion of these arguments and part_of as predicate .
in addition, some indicator expressions can be ambiguous with respect to their meaning; for instance, the preposition "in" may map to either predicate in this example.
examining instances of the location_of/part_of variation in the practice phase, we concluded that at the biomolecular level, the difference in meaning is blurred to the extent that two semantic predications may be considered equivalent for interannotator agreement calculation.
however, we also noted that there are clear exceptions.
for example, consider the sentence fragment "... truncated bid , which translocates to mitochondria ...".
while the predication "c0026237: mitochondria -location_of-c1144558: tbid protein " seems acceptable for this fragment, its counterpart with the predicate part_of -part_of-c0026237: mitochondria ") does not.
to handle such exceptions, we manually judged the cases in which there was agreement due to this equivalence criterion for correctness as a post-processing step to interannotator agreement calculation and corrected the agreement score accordingly.
a situation similar to location_of/part_of equivalence concerns produces/part_of predicates and the verbal indicator "express" and its derivations.
a relation indicating gene expression events does not currently exist in the semrep ontology, leading one annotator to use part_of consistently for such events, and the other to use produces.
in the fragment "the expressions of c-myc, ki- <dig>  mmp- <dig> ... in cancer tissues", one annotator chose "c1334508: mki <dig> gene -part_of-c0040300: body tissue " and the other "c0040300: body tissue -produces-c1334508: mki <dig> gene ."
we treat these cases as equivalent as well.
interannotator agreement using these two equivalence criteria is denoted as pe below.
correspondence of gene and gene products
in the molecular biology literature, gene names are often used to denote gene products , leading to term ambiguity.
for instance, in the fragment "tnfr1/fas engagement results in the cleavage of cytosolic bid to ...", without knowing the context, it is difficult to determine whether tnfr <dig>  fas or bid refer to genes or gene products.
this ambiguity extends to the umls metathesaurus as well.
for instance, bid maps to "c1332410: bid gene " and "c0531588: bid protein ."
on the other hand, tnfr <dig> is mapped to "c0255808: tumor necrosis factor receptor 1a " as well as "c1363984: tnfrsf1a gene , while the equivalence in the case of tnfr <dig> is ignored.
although the two concepts  corresponding to the textual mention  are considered related terms in umls, it is difficult to establish their correspondence from their concept names alone.
correspondence of this type is denoted as gp below.
limiting agreement calculation by available domain knowledge
to assess the difficulty of annotation due to the open-ended, exploratory nature of our annotation, we also limited interannotator agreement calculation based on the availability of conceptual knowledge to the annotator  as well as that of relational knowledge  and both , as mentioned earlier.
our intuition in limiting calculation to a subset of annotated predications was that there would be more substantial agreement when the annotator chooses concepts and ontological predications from a predefined set.
main annotation phase
two annotators  were involved in the main annotation phase.
the average number of semantic predication annotations per sentence was slightly lower than that in the practice phase ; one annotator  was relatively consistent between the practice and main annotation phases, while the other  annotated significantly fewer in the main phase, as shown in table  <dig>
the interannotator agreement results for the main annotation phase are presented in table  <dig>  with the improved guidelines provided to the annotators, the basic strict agreement between the annotators increased from  <dig>  to  <dig> , while the basic relaxed agreement increase was higher: from  <dig>  to  <dig> .
additionally, we computed interannotator agreement using the extended equivalence criteria .
adding the predication equivalence criterion  increased the agreement by approximately 3%, while gene/gene product correspondence criterion  provided a small increase of  <dig> % overall.
limiting the comparison to conceptual knowledge provided , the agreement reaches substantial levels with an increase of approximately 12%.
relational domain knowledge alone  had less effect on agreement .
with both conceptual and relational knowledge provided to annotators , the agreement rises more than 15% in comparison to the basic case.
we only show relaxed agreement in the basic case, where the increase from the strict counterpart is about  <dig> %.
we note that this trend was observable in the cases of predication filtering, as well .
we consider strict agreement with base + pe + gp equivalence as the main agreement criterion in the rest of the paper .
pe: predication equivalence, gp: gene/gene product correspondence, ck: conceptual knowledge, rk: relational knowledge, crk: conceptual and relational knowledge
when we consider the indicator types that signal predications, modifier-head constructions  appear more frequently as indicators than in the practice phase, as shown in table  <dig>  in addition, due to a clarification regarding how the indicator type for hypernymic relations should be annotated, we observed an increase in the frequency of the indicator type spec, which essentially indicates that the hypernymic relation between the concepts should be licensed by the umls metathesaurus hierarchy in addition to being indicated by a textual clue.
the ontological predicate distribution in the main annotation phase is presented in table  <dig>  comparing the distribution to that in the practice phase , one noticeable trend is a relative increase in the number of process_of, part_of, treats and affects predicates and a decrease in inhibits and interacts_with predicates, possibly due to clarifications over definitions of the predicates after the practice phase.
affects and inhibits predicates are not shown in table  <dig> and table  <dig>  respectively, since they occur less frequently in the respective phases.
however, we note that the increase in the number of affects predicates from the practice phase to the main annotation phase is  <dig> %, whereas the decrease in that of inhibits is 8%.
it is also worth mentioning that the high frequency of inhibits in the practice phase seems artificial, since most of those annotations were the result of a single instance that involved complex coordination.
during this phase, we also computed interannotator agreement  at the ontological predicate level to assess whether some types are more difficult to annotate than others.
among the most frequent predicates, there is less agreement on predicates relating biomolecular entities or processes .
on the other hand, predicates concerning disorders, anatomical parts and population groups yield overall higher agreement.
further examining ontological predicates with highest and lowest interannotator agreement , these findings are confirmed.
among ontological predicates annotated more than  <dig> times, the highest disagreement rates were associated with those involving biomolecular entities and processes .
on the other hand, the highest agreement rate was for process_of and prevents, both of which are disease-related ontological predicates.
similar topical trends were observable at the ontological predication level, as well .
among those ontological predications annotated more than  <dig> times between annotators, the highest disagreement rates were found in those concerning the biomolecular entities and processes, while disagreement was lowest in predications involving disorders and population groups.
these results clearly establish bio-molecular relations and disorder/population relations as two extremes in the spectrum in terms of ease of annotation.
aapp: amino acid, peptide, or protein; celf: cell function; cell: cell; dsyn: disease or syndrome; gngm: gene or genome; humn: human; inpo: injury or poisoning; mamm: mammal; neop: neoplastic process; rcpt: receptor; sosy: sign or symptom
annotators were allowed to provide comments for their annotations, and we examined the comments corresponding to the predications on which there was disagreement.
the most frequent comments in disagreement cases are given below.
the number of disagreement instances and the corresponding interannotator agreement scores are given in parentheses.
the interannotator agreement score for a specific type of comment was calculated by considering all the predications marked with that comment, regardless of whether there was disagreement or not.
a. unsatisfactory umls concept extracted by metamap: the umls concept identified by metamap is replaced with a more appropriate concept from the umls 2006aa release.
b. no indicator rule: there is no indicator rule mapping from the textual mention to the ontological predicate in semrep.
c. relational domain knowledge unavailable: the annotated semantic predication does not correspond to an ontological predication in the semrep ontology.
d. newer umls release concept: a non-2006aa umls concept is preferred.
e. complete metamap miss: a 2006aa umls concept that metamap fails to identify is used.
adjudication phase
in the adjudication phase, annotator c  examined the annotations provided by a and b, resolving differences and determining the gold standard.
as well as selecting one annotation from one set over another from the other set, the adjudicator could also override annotations from both sets  or keep annotations from both sets .
the final semantic predication count is  <dig>  an average of  <dig>  predication per sentence.
the maximum number of predications per sentence is  <dig>  the frequency distributions by ontological predicate types and indicator types after reconciliation are presented in table  <dig> and table  <dig>  respectively.
while interannotator agreement at this stage is not meaningful since the annotation sets were available to the adjudicator, we measured it to assess whether the adjudicator clearly prefers one set of annotation to the other.
the adjudicator agreed with annotator b at a much higher rate .
this seems likely due to the fact that the adjudicator took b annotations as the basis and only added an a annotation when b annotation was deemed incorrect.
discussion
we conducted a relatively open-ended, multi-phase annotation study, in which we aimed to assess the feasibility of iteratively constructing a reasonable gold standard reference based on umls domain knowledge.
the results presented in the previous section show that this is a feasible undertaking.
on the other hand, they confirm that conceptual annotation is extremely challenging; the main difficulty is mapping textual mentions to ontological terms , a time-consuming and labor-intensive task.
this is evidenced by the fact that interannotator agreement increases to acceptable levels  when the comparison is limited to conceptual knowledge available to the annotators.
we discuss some of the challenges in more depth below.
mapping text to ontological concepts
the core notion in mapping textual mentions to ontological concepts is semantic equivalence, which entails that the ontological concept must express the meaning of the textual mention adequately, and should not correspond to a more general or more specific meaning than the textual mention.
however, this equivalence is often very difficult to establish.
in our study, the difficulty of establishing semantic equivalence is further compounded by the nature and size of the primary reference terminology we used for conceptual information, the umls metathesaurus.
in addition, we defined the task in an open-ended manner and allowed the annotators to use newer umls releases when the primary knowledge source  is inadequate, which increased the complexity of the task and lowered the interannotator agreement.
to ease the burden for the annotator, we provided umls concepts identified by metamap as reference, corresponding to the domain knowledge available to semrep.
while this was useful to a large extent, it did not prevent disagreements regarding what constitutes an adequately expressive, semantically equivalent mapping.
in some cases, the concepts identified by metamap may even be misleading, as we found out, since one annotator often considered a given concept adequate for the textual mention in question, while the other was dissatisfied with the mapping and identified a clearly better umls metathesaurus concept.
for example, metamap failed to completely map the noun phrase "the d <dig> receptor" to the existing umls concept "c0082341: dopamine d <dig> receptor"  and identified the concept "c0597357: receptor"  instead.
one annotator considered this more general concept adequate, while the other identified the former as the more adequate mapping.
a similar situation occurred with "minimally invasive surgery," where metamap identified "c0543467: operative surgical procedures" , while, in fact, there exists a more suitable concept "c0282624: surgical procedures, minimally invasive"  for the mention.
there were also cases where metamap failed to identify a concept corresponding to a textual mention, and it was difficult for the annotator to see that the textual mention may in fact have a umls counterpart.
for instance, only one annotator found and used the concept "c1254042: anatomical maturation " to correspond to the head of the more specific noun phrase "lung maturation."
despite these difficulties, however, evidence from the annotators indicates that concepts identified by metamap were overall helpful to annotators and lightened the annotation load considerably.
furthermore, interannotator agreement increased significantly  when only metamap-supplied concepts were used, supporting our basic intuition that selecting a concept from a predefined list is less demanding than searching for a semantically equivalent concept from a sizable terminology and assessing its adequacy, an essentially subjective task.
in other gold standard annotation studies, such as the craft corpus and the clef corpus, carefully selected, well-curated vocabularies or ontologies were used.
for example, the craft corpus made use of six obo ontologies.
however, finding the semantically equivalent concept for a textual mention is still found to be challenging.
in our annotation, consistent with the semrep methodology, we did not limit ourselves to specific umls vocabularies and aimed to be more general in our coverage.
however, this generality has its drawbacks, as expected.
while synonymous terms from different vocabularies are clustered together to form concepts in the umls metathesaurus, there are still a significant number of distinct concepts that are very close in meaning.
for example, for the noun phrase "mineral metabolism impairment," metamap identified two separate concepts: "c0678715: mineral metabolism"  and "c0011155: deficiency" .
neither annotator found these mappings adequate ; however, they found different concepts, similar in meaning, to correspond to the meaning of the mention.
one identified "c0687148: mineral deficiency" , while the other identified "c0154260: disorder of mineral metabolism" .
while these concepts are not exactly the same, the difference between their meanings is arguably very small.
a similar situation holds between "c0403716: calculus in renal pelvis"  and "c0022650: kidney calculi"  for the phrase "pelvic stones."
the problem seems even more acute when it comes to gene and gene product concepts, as exemplified earlier.
it seems necessary to take into account such overlap and similarity in meaning, both in computing interannotator agreement and in system evaluation.
we attempted to address the semantic equivalence of concepts to some extent by devising an equivalence criterion involving gene and gene products; however, this criterion is limited to a single, clear pattern, and is unable to accommodate more complex cases of equivalence.
one principled solution may be to allow annotators to annotate more than one concept for a given textual mention when a clear preference for a concept cannot be established and to give partial  credit for any of the concepts.
this would clearly increase annotation complexity, and it may be more feasible as a postprocessing step.
annotation by domain
our results also show that difficulty of annotation varies by the biomedical subdomain.
ontological predications relating to biomolecular entities and processes are most challenging to annotate, while those on the topic of population characteristics of disease seem to be the most straightforward.
we explain the difficulty of annotating biomolecular relations by observing the following:
a. molecular biology text is hardest to read and interpret for a non-expert and none of the annotators are experts in this subdomain.
furthermore, as friedman et al.
<cit>  have shown, biomolecular domain text constitutes a sublanguage, with very specific characteristics, such as complex and nested relations as well as more prevalent syntactic ambiguity.
one interesting, syntactically ambiguous case involved the fragment "... il-1beta-induced ros formation, nf-kappab activation, and mcp- <dig> secretion ...", where one annotator took the modifier "il-1beta-induced" as modifying "nf-kappab activation" and "mcp- <dig> secretion" as well as "ros formation", and annotated the predications given in , while the other annotator took the modifier to modify "ros formation" only, and did not annotate the predications in .
c0021753: interleukin- <dig> beta -stimulates-c0128897: monocyte chemoattractant protein- <dig>
c0021753: interleukin- <dig> beta -stimulates-c0079904: nf-kappa b
b. the coverage of the umls semantic network with respect to molecular biology is perhaps the least extensive.
in fact, we have extended the umls semantic network to create the semrep ontology in prior work  <cit>  specifically to redress this gap.
c. based on evidence from the annotation process, it seems more challenging for metamap to map textual mentions of biomolecular entities to umls metathesaurus than to map those of entities from other subdomains.
for example, the semantic type most frequently associated with concepts identified by annotators and not by metamap was amino acid, peptide, or protein.
in contrast to genomic concepts and relations, disorder and population group concepts and their relations are well covered in the umls and metamap has less difficulty in mapping text to these concepts.
in addition, text concerning disease characteristics is overall easier to interpret for a non-expert.
diachronic change in domain knowledge
another complicating factor in conceptual annotation is that the ontologies and vocabularies the concepts are derived from may change over time.
there are two alternatives to address this situation:  using a static snapshot of the knowledge source  re-annotating at each update of the knowledge source.
our methodology was similar to the first alternative: we adopted the 2006aa release of umls as the primary source for conceptual information, while also recognizing that newer releases of umls have wider coverage, especially with respect to new drug or gene/protein names, thus allowing the annotators to use these resources, if necessary.
since we enforced text-bounded annotations, we believe that it will be possible to update the gold standard semi-automatically at future updates of the knowledge sources, using metamap or a similar program.
CONCLUSIONS
we have presented the construction of a semantic predication gold standard from biomedical literature text using the conceptual annotation paradigm.
manual conceptual annotation is considered extremely challenging, and our results confirm this perception, while also confirming that reasonable interannotator agreement could be achieved iteratively, consistent with the findings of bada et al.
<cit> .
while the domain knowledge we used  reflects the application-specific aspect of our annotation, we believe that our analysis and discussion provide important insights for future efforts in this area.
the resulting gold standard constitutes the first resource, to our knowledge, in the biomedical domain that incorporates conceptual annotation of semantic relations in a wide variety of subdomains.
two sets of annotations and the adjudicated gold standard are made publicly available  <cit>  for research purposes.
a umls license is required.
the corpus size is relatively small and may be insufficient for training information extraction systems.
however, we believe it can serve as a benchmark to evaluate independently developed systems based on umls knowledge sources.
our goal is to use it for this particular purpose, as well as to guide future system development.
authors' contributions
hk conceived of the study, led the annotation effort, performed the analyses, and drafted the manuscript.
hk, gr, and mf contributed to annotation, guideline development, and helped draft the manuscript.
tcr directed the research and helped draft the manuscript.
all authors read and approved the final manuscript.
appendix
ontological predicate definitions
ontological predicate definitions and an example for each of the definitions are provided below.
note that each of the following can also be negated :
• administered_to: given to an entity, when no assertion is made that the substance or procedure is being given as treatment.
patients with single brain lesion received an extra  <dig> gy x  <dig> radiotherapy ....
c0034618: radiation therapy -administered_to-c0030705: patients
• affects: produces a direct effect on.
implied here is the altering or influencing of an existing condition, state, situation, or entity.
this includes has a role in, alters, influences, predisposes, catalyzes, stimulates, regulates, depresses, impedes, enhances, contributes to, leads to, and modifies.
bap <dig> and its caspase cleavage product regulate cell surface expression of tetraspanins and integrin-mediated cell survival.
c1424489: bcap <dig> gene -affects-c0007620: cell survival
• associated_with: has a relationship to .
ep <dig> plays a critical role in tumorigenesis in mouse mammary gland ...
c1419062: ptger <dig> gene -associated_with-c1326912: tumorigenesis
• augments: expands or stimulates a process.
nicotine induces conditioned place preferences over a large range of doses in rats.
c0028040: nicotine -augments-c0815102: place preference learning
• causes: brings about a condition or an effect.
implied here is that an agent, such as for example, a pharmacologic substance or an organism, has brought about the effect.
this includes induces, effects, evokes, and etiology.
neurocysticercosis  is one of the major causes of neurological disease ...
c0338437:neurocysticercosis -causes-c0027765: nervous system disorder
• coexists_with: occurs together with, or jointly.
food intolerance-related constipation is characterized by proctitis.
c0009806: constipation -coexists_with-c0033246: proctitis
• converts_to: changes from one form to another .
... plasma nitrite is readily oxidized to nitrate within plasma ...
c0028137: nitrites -converts_to-c0699857: nitrate
• complicates: causes to become more severe or complex, or results in adverse effects.
infections can trigger gbs and exacerbate cidp.
c0021311: infection -complicates-c0393819: polyradiculoneuropathy, chronic inflammatory demyelinating
• diagnoses: distinguishes or identifies the nature or characteristics of.
manometry showed a higher anal sphincter resting pressure ...
c0024751: manometry -diagnoses-c0429217: anal sphincter pressure
• disrupts: alters or influences an already existing condition, state, or situation.
produces a negative effect on.
overexpression of nf-kappab inhibits tumor cell apoptosis.
c0079904: nf-kappab -disrupts-c0162638: apoptosis
• inhibits: decreases, limits, or blocks the action or function of .
in recent studies, the bdnf expression was reduced by typical neuroleptics.
c0040615: antipsychotic agents -inhibits-c0107103: brain-derived neurotrophic factor
• interacts_with: substance interaction.
here we show that chymases, which are chymotryptic peptidases secreted by mast cells, hydrolyze hgf ...
c0055673: chymase -interacts_with-c0062534: hepatocyte growth factor
• isa: the basic hierarchical link in the umls semantic network.
if one item isa another item then the first item is more specific in meaning than the second item.
the sympathetic neurotransmitter norepinephrine has been found ...
c0028351: norepinephrine -isa-c0027908: neurotransmitters
• location_of: the position, site, or region of an entity or the site of a process.
we report a case of primary cardiac epithelioid hemangioendothelioma arising from the right atrium of a 2-month-old infant.
c1269890: entire right atrium -location_of-c0206732: hemangioendothelioma, epithelioid
• manifestation_of: that part of a phenomenon which is directly observable or concretely or visibly expressed, or which gives evidence to the underlying process.
this includes expression of, display of, and exhibition of.
recurrence of glomerulopathy underlying esrd was frequent for igan, fsg...
c1261469: end stage renal failure -manifestation_of-c1398810: glomerulopathy
• method_of: the manner and sequence of events in performing an act or procedure.
... because of the use of sscp as a screening method and sequencing only a part of tshr exon  <dig>
c0243031: single-stranded conformational polymorphism -method_of-c0220908:screening procedure
• occurs_in: has incidence in a group or population.
older populations are more prone to bone loss with weight loss ...
c0599877: loss; bone -occurs_in-c1518563: older population
• part_of: composes, with one or more other physical units, some larger whole.
this includes component of, division of, portion of, fragment of, section of, and layer of.
the probasal bodies and microtubules within the blepharoplast cavities...
c0026046: microtubules -part_of-c0230744: basal body of cilium or flagellum, not bacterial
• precedes: occurs earlier in time.
this includes antedates, comes before, is in advance of, predates, and is prior to.
... the risk of tissue plasminogen activator-induced hemorrhagic transformation following ischemic stroke in mice ...
c0948008: ischemic stroke -precedes-c1096400: haemorrhagic transformation stroke
• predisposes: to be a risk to a disorder, pathology, or condition.
... high ghrelin levels contribute to obesity in prader-willi syndrome  ...
c0911014: ghrelin -predisposes-c0028754: obesity
• prevents: stops, hinders or eliminates an action or condition.
ipsapirone and ketanserin protects against circulatory shock, intracranial hypertension, and cerebral ischemia during heatstroke.
c0123905: ipsapirone -prevents-c0151740: intracranial hypertension
• process_of: disorder occurs in  organism.
... no information is available in cad patients with normal glomerular filtration rate .
c0010054: coronary arteriosclerosis -process_of-c0030705: patients
• produces: brings forth, generates or creates.
this includes yields, secretes, emits, biosynthesizes, generates, releases, discharges, and creates.
human epcs express functional par- <dig> ..
c0038250: stem cells -produces-c0668084:receptor, par- <dig>
• stimulates: increases or facilitates the action or function of .
candesartan therapy significantly reduced inflammation and increased adiponectin levels ...
c0717550: candesartan -stimulates-c0389071: adiponectin
• treats: applies a remedy with the object of effecting a cure or managing a condition.
this study initially surveyed  <dig> patients with clinical stage ib or iia cervical adenocarcinoma treated with radical hysterectomy and pelvic lymphadenectomy.
c0677962: radical hysterectomy -treats-c0279672: cervical adenocarcinoma
• uses: employs in the carrying out of some activity.
this includes applies, utilizes, employs, and avails.
pre-emptive therapy with oral valganciclovir for cmv infections...
c0087111: therapeutic procedure -uses-c0909381: valganciclovir
• compared_with: comparative predicate.
compared with placebo, candesartan therapy significantly lowered plasma hscrp levels...
c0032042: placebos -compared_with-c0717550: candesartan
• higher_than: comparative predicate.
losartan was more effective than atenolol in reducing cardiovascular morbidity...
c0126174: losartan -higher_than-c0004147: atenolol
• lower_than: comparative predicate.
amoxicillin-clavulanate was not as effective as ciprofloxacin for treating uncomplicated bladder infection in women.
c0054066: amoxicillin-potassium clavulanate combination -lower_than-c0008809: ciprofloxacin
• same_as: comparative predicate.
candesartan is as effective as lisinopril once daily in reducing blood pressure.
c0717550: candesartan -same_as-c0065374: lisinopril
predicate types abbreviations and definitions
• verb: verbal predicates
• nom: nominalizations or other argument-taking nouns
• adj: adjectival predicates
• prep: prepositional predicates
• aux: auxiliary verb predicates
• part: past participial predicates
• mod/head: the predication is the result of a noun phrase construction
• spec: the predication is hypernymic and must be licensed by the umls metathesaurus hierarchy, that is, the subject must be a descendant of the object in the umls metathesaurus hierarchy
• infer: the predication is the result of inference based on two other existing predications
• complex: the indicator is a multi-word expression , but not a multi-word lexical item .
sample annotation
the following information  is provided to the annotators:
<dig> ab.
<dig> high-dose aspirin at a therapeutic dose does not affect cyp2e <dig> activity in humans.
c0444956|high dose|qnco|||high-dose|901|1|9
c0004057|aspirin|orch,phsu|||aspirin|901|11|17
c0302350|therapeutic|ftcn|||therapeutic|888|24|34
c1446466|dose|qnco|||dose|888|36|39
c1518422|not|ftcn|||not|1000|46|48
c0919426|cyp2e <dig> gene|gngm|1571|cyp2e1|cyp2e1|888| 57|62
c0439167|% activity|qnco|||activity|888|64|71
c0020114|human|humn|||humans|1000|76|81
in the sentence line above,  <dig> ab.
<dig> is a sentence identifier .
in the subsequent entity lines, the fields correspond to the following :
<dig>  concept identifier
<dig>  umls metathesaurus preferred name
<dig>  umls semantic type abbreviation
<dig>  entrez gene id   <cit>
<dig>  entrez gene name
<dig>  textual mention that maps to the entity
<dig>  metamap score for the entity  <cit>
<dig>  first character position  of text denoting entity  <cit>
<dig>  last character position  of text denoting entity  <cit>
from these provided annotations, the following two semantic predication annotations are expected:
c0004057|aspirin|orch,phsu|orch|||neg_interacts_with|c0919426|cyp2e <dig> gene|gngm,aapp|gngm|1571|cyp2e1|verb|affect|aspirin|cyp2e1
c0020114|human|grup,humn|humn|||location_of|c0919426|cyp2e <dig> gene|gngm,aapp|aapp|1571|cyp2e1|prep|in|humans|cyp2e1
fields 1- <dig> and 8- <dig> correspond to fields 1- <dig> of the entity lines for subject and object, respectively.
fields  <dig> and  <dig> correspond to the individual semantic types that license the predication.
fields 5- <dig> and 12- <dig> correspond to fields 4- <dig> of the entity lines for subject and object, respectively.
field  <dig> corresponds to the ontological predicate .
field  <dig> is the ontological predicate type .
field  <dig> is the textual mention corresponding to the ontological predicate, field  <dig> corresponds to the subject argument and field  <dig> to the object argument.

 large-scale information extraction from scientific literature is increasingly used to support advanced knowledge management and discovery systems. the utility of such systems depends on the quality of the extracted information. manually annotated gold-standard corpora are critical for evaluating the accuracy and usefulness of information extraction systems. in the biomedical domain, various corpora annotated for semantic phenomena have been constructed in recent years; annotations range from named entities, to semantic relations, such as protein-protein interactions, protein/gene/rna relationships, disease-treatment relations, clinical relations, biological events, and gene regulation events. more recently, the notion of "silver standard" has also been introduced, referring to harmonization of automated system annotations, as a proxy to labor-intensive gold standard annotation. the gold standard corpora have often focused on text drawn from a narrow subdomain, adopting a particular semantic representation, addressing a small set of semantic types and aiming to provide training and evaluation support for specialized ie systems. these corpora differ with respect to their level of granularity and whether there is an ontological basis to the entity and relationship types used. for example, one of the most popular corpora in recent years has been the genia event corpus, drawn from the scientific literature on transcription factors in human blood cells. it is based on the notion of biological events, uses a few dozen gene ontology event types and has been the basis for recent biological event extraction systems as well as two bionlp shared task competitions. the generally narrow focus of such corpora and their specific representation formalisms render them largely unsuitable for evaluating ie systems using different formalisms or resources. we have been developing a semantic interpreter, semrep, which extracts content from biomedical text in the form of semantic predications. a semantic predication is a logical subject-predicate-logical object triple whose elements are drawn from the umls knowledge sources; the subject and object pair corresponds to umls metathesaurus concepts and the predicate to a relation type in an extended version of umls semantic network. while the umls semantic network has not been designed as an ontology in a strict sense, the extended version that semrep uses serves as an ontological resource: it defines a domain model consisting of concept types, relation types and the relationships that can hold between concept types. each semantic predication extracted by semrep is an instantiation of an ontological predication. we refer to this extended version of the umls semantic network as the semrep ontology henceforth. semrep extracts a range of predicates relating to clinical medicine, substance interactions, genetic etiology of disease, and pharmacogenomics. for example, the program identifies the semantic predications in from the input text in. arguments of the semantic predications have the form conceptidentifier: conceptname. textual mentions corresponding to the arguments are in bold, and those corresponding to the predicates are underlined in. mri revealed a lacunar infarction in the left internal capsule. c0024485: magnetic resonance imaging -diagnoses-c0333559: infarction, lacunar c0152341: internal capsule -location_of-c0333559: infarction, lacunar semrep processing is supported by an underspecified syntactic analysis based on the umls specialist lexicon and the medpost part-of-speech tagger. metamap is used to map simple noun phrases to umls metathesaurus concepts. entrez gene serves as a supplementary source to the umls metathesaurus with respect to gene/protein terms, which are identified using abgene, in addition to metamap. indicator rules map syntactic phenomena, such as verbs, nominalizations, prepositions, and modifier-head structure in the simple noun phrase, to ontological predicates from the semrep ontology. semrep currently uses the 2006aa release of the umls knowledge sources, due to the prevalence of ambiguity in later releases. the lack of a suitable, manually annotated gold standard corpus has so far precluded a formal evaluation of semrep; system improvements and modifications have been informally evaluated through error analysis. a formal evaluation requires conceptual annotation with respect to the umls; that is, text fragments need to be mapped to concepts and relations in the umls, which provides a formal representation of domain knowledge. considering that the umls metathesaurus, the basis for conceptual information, consists of source vocabularies and more than million concepts, it is clear that such conceptual annotation is an extremely challenging task. large-scale conceptual annotation is not generally attempted in the biomedical domain. in fact, apart from the recent craft corpus and the clef corpus, we are not aware of any such annotation work. the ontology-based semantic annotation of the craft corpus concentrates on biomolecular entities and processes, including gene/gene products, chemicals, sequence types, molecular functions, and cellular components. ninety-seven full-text articles were annotated with concepts from eight terminologies, six of them from the obo library. they report that relationship annotation between concepts is ongoing work. intended for clinical ie research, the clef corpus annotates information about clinical entities and the relations between them, along with temporal information. it is limited to clinical notes and reports of deceased patients who had a diagnosis of neoplasms. clef builds on a relevant subset of umls concepts and relations as domain knowledge. some semantic types are conflated and relations renamed in accordance with the goals of the project. for instance, clef semantic type condition includes symptoms, complications, functions, diagnosis, and problems, conflating several umls semantic types. in contrast to these manual conceptual annotation efforts, jimeno et al. semi-automatically created a small corpus annotated with disease concepts from umls metathesaurus. their semi-automatic methodology involves domain expert assessment of disease concepts identified by metamap, a dictionary lookup method, and a statistical method. in this article, we present an annotation study in which we annotated sentences from medline abstracts with semantic predications. our annotation follows the conceptual annotation paradigm and adopts the entire umls as the domain model. furthermore, we do not limit ourselves to text from a specific subdomain or adopt specific terminologies, in contrast to more focused efforts such as clef and craft. while our methodology bears some similarities to the clef methodology in terms of the domain model and the guidelines, we use fine-grained umls semantic types, rather than coarse semantic groups, allowing for more flexibility. with respect to our ongoing research, the resulting gold standard reference is mainly intended to facilitate comparison between semrep releases and guide further system development by allowing annotators to add comments and notes in a standardized manner so that problem areas can be identified. several limitations of the gold standard reference may include its relatively small size, its sentence-bound annotation, and its binary, umls semantic network semantic relation formalism, arguably not as rich a semantic representation as predicate-argument structures with semantic roles. while small corpus size may present challenges for learning purposes, it serves well for our primary goal of evaluation. on the other hand, the semantic predication representation has been shown to be simple, intuitively accessible, and tractable for large-scale knowledge discovery applications. furthermore, this representation lends itself readily to the semantic web and linked data movements, which aim to encode knowledge in subject-predicate-object triples for large-scale automatic reasoning. considering the challenges posed by this task, we believe our corpus presents a good first step in a larger-scale conceptual/relational annotation. it can serve as a gold standard reference for evaluating umls-based relation extraction systems and the lessons learned can provide guidance for future efforts in this area. methods we conducted the annotation study in three phases: a) practice annotation phase, b) main annotation phase and c) adjudication phase. before explaining these phases in more detail, we briefly discuss two fundamental aspects of our study: annotators and interannotator agreement. annotators three annotators, all authors of this paper, were involved in the annotation process. the annotators have diverse backgrounds; annotator a is a linguist, b a computer scientist and c a physician/biomedical informatics researcher. all three annotators have natural language processing experience and are experts in the semrep methodology as well as the umls knowledge sources. using domain experts as annotators is often considered a good strategy to ensure validity and reliability of annotation. however, the tendency of domain experts to rely on inference due to their background knowledge has also been noted. when annotation is concerned with a relatively narrow biomedical subdomain, it is generally feasible to recruit domain experts as annotators. however, several aspects of our annotation study make it difficult to find such annotators. first, we do not focus on a narrow biomedical subdomain. the consequence is that we need to either recruit experts who are knowledgeable in almost all aspects of biomedicine, or to find tens of annotators who can annotate different sentences on topics of their expertise. neither option seemed feasible within the scope of our annotation. furthermore, the two non-physician annotators work with biomedical text on a full-time basis, and they expressed comfort with annotation after the practice phase. secondly, in our annotation study, umls expertise is perhaps as crucial as domain expertise, and all three annotators are intimately familiar with the umls knowledge sources. recruiting domain experts who are also familiar with the umls would clearly be even more challenging. we believe that our small team of annotators with their expertise in umls and our multi-phase annotation methodology allows us to strike a balance between annotation reliability and validity. interannotator agreement the common approach to calculating interannotator agreement on classification tasks is to use kappa statistic, defined as κ = - pr)/), where pr is the relative observed agreement between annotators, and pr is the chance agreement. calculating κ for semantic predication annotations is non-trivial, since this type of linguistic annotation is not a simple classification task. semantic predication annotation task can be decomposed into several subtasks for which agreement can be measured independently: finding relation indicators, finding textual mentions of arguments, mapping the relation indicators to ontological predicates, and mapping the textual mentions of arguments to concepts. with the exception of subtask, the space of possible annotations is either not clearly defined or very large, making the calculation of pr and therefore κ, challenging. for example, consider subtask. one way to calculate κ for this subtask is to impose constraints on what can be annotated as an indicator. for example, only verbs may be considered as indicators and the number of verbs in a sentence can be used to calculate chance agreement pr. we did not impose such constraints since we aimed for breadth of coverage in our annotation study. this leads to an explosion of possible indicator annotations and to the case in which pr essentially approaches zero. in such cases, it has been shown that κ approximates f-measure among pairs of annotators. based on these observations and in line with annotation studies that share similarities with ours, we adopted f-measure for interannotator agreement. between two sets of annotations, we calculated it as the f-score of one set of annotations, when the second is taken as the gold standard. practice phase for the practice annotation phase, we randomly selected sentences from medline abstracts published in the last years. these sentences were extracted from the database of semantic medline, a web application to manage the results of pubmed searches, and were manually checked by the first author to ensure correct sentence segmentation. all three annotators participated in the practice annotation phase, each annotating the same set of sentences. given the annotators' familiarity with semrep, minimal guidance for annotation was provided at this phase and the annotators were asked to annotate semantic predications expressed in the sentences, the textual mentions of their arguments and the predicate and the indicator type, preposition, nominalization, participle, etc.). to lighten the burden of finding an appropriate umls metathesaurus concept corresponding to a textual mention, umls metathesaurus concepts were extracted from these sentences using metamap and were provided to the annotators. the guidance at this phase consisted of the following items: a list of core ontological predicates relevant to semrep and their definitions from the umls. for ontological predicates that are not part of the official umls semantic network but are in the semrep ontology, we used our own definitions. we provide these definitions in the appendix. an inventory of predicate types, also provided in the appendix. a sample sentence annotation, provided in the appendix and illustrated in figure basic instructions consisted of the following: a) annotation should be restricted to semantic predications involving the core ontological predicates that are provided. other predicates, even though they may be legitimate, should be ignored. on the other hand, the annotators are not restricted to follow the ontological predications that exist in the semrep ontology. b) the umls concepts extracted by metamap are not necessarily the best possible mappings. in addition, metamap may be unable to find a mapping. when in doubt, the annotator should try to find a concept that better matches the text using the umls terminology services or entrez gene, keeping in mind that semrep currently uses the 2006aa version of the umls knowledge sources. c) the annotation should be text-bound. that is to say, domain knowledge and inference should play a minimal role in annotation and the annotator should be concerned with what is explicitly stated in the text. to ensure this, the annotator should explicitly indicate the textual mentions that provide the basis for the annotation, as well as the indicator type for the annotation. providing only minimal guidance, we aimed to identify the major challenges in annotating predications and find ways to deal with them in the main annotation phase. this first phase was collaborative: the annotators were free to discuss sentences, concerns and difficulties and develop solutions. we chose not to use a particular annotation tool, since none of the existing tools fully met our needs, especially with respect to access to terminological resources. we decided not to develop a study-specific, in-house annotation tool due to time constraints. instead, the annotators were instructed to simply type semantic predications in a text document, along with the textual mentions that trigger the predication components. then, based on the results of the first phase, annotators were provided scripts that recognized formatting, spelling errors, and inconsistencies in annotation. these scripts were used by annotators in subsequent phases and helped them resolve such errors. the practice annotation phase was completed in three weeks. after this phase concluded, the first author analyzed each annotation set to identify annotation patterns. the analysis results were then discussed among the annotators and served to refine the guidelines for the second phase. we present these refinements in the results section. we also computed a baseline interannotator agreement between pairs of annotators. to assess agreement with respect to textual mentions vs. conceptual information, we calculated interannotator agreement using two criteria: a. strict equivalence criterion, where for two semantic predications to be considered equal, their subject-predicate-object triples must match exactly. b. relaxed equivalence criterion, where the exact match of the textual mentions of the arguments and of the predicate that establishes their relationship are considered sufficient for equality. in other words, conceptual match is not required between predication elements. main annotation phase in the main annotation phase, two annotators annotated a set of randomly selected sentences drawn from medline abstracts from the past years. similar to the practice phase, sentences were drawn from the semantic medline database, were checked for integrity, and umls metathesaurus concepts extracted by metamap were provided for reference. the annotators worked independently in this phase, which concluded in eight weeks. based on our observations from the practice phase, we extended the equivalence criteria underlying interannotator agreement in two ways: a. predication equivalence: a pair of distinct semantic predications may be considered equivalent under specific conditions when one inverts the arguments of the other and the predicates correspond to certain types. for instance, a predication x-location_of-y may be considered equivalent to y-part_of-x predicates when arguments correspond to biomolecular entities. b. gene/gene product correspondence: a pair of concepts may be considered equivalent when one corresponds to a gene term and the other corresponds to its gene product. for instance, the concept "c0287531: dusp protein, human" is considered equivalent to "c1333257: dusp gene". we also assessed the effect on annotation of domain knowledge provided to the annotators. for this purpose, we distinguished between two types of domain knowledge, conceptual and relational, and measured interannotator agreement on a subset of predications based on the following availability criteria, illustrated in below: a. availability of conceptual knowledge: a predication fulfills this criterion if both the subject and object arguments were extracted by metamap and, thus, were provided to the annotators. b. availability of relational knowledge: a predication fulfills this criterion if it is sanctioned by the semrep ontology. in other words, it corresponds to an existing ontological predication. c. availability of conceptual and relational knowledge: a predication fulfills this criterion if it satisfies both and, the previous two criteria. for illustration, consider the sentence fragment in. relevant concepts identified by metamap are given in, and an annotated predication in.... udp-glc is required in the synthesis of proteoglycans.... c0041986: uridine diphosphate c0008556: chromatography, gas-liquid c0033692: proteoglycan c0041988: uridine diphosphate glucose -produces-c0033692: proteoglycan while the predication in is correct, it does not fulfill criterion above because metamap fails to identify the subject "c0041988: uridine diphosphate glucose," an argument absent in. it also fails to fulfill criterion because the corresponding ontological predication "biologically active substance-produces-amino acid, peptide, or protein" is not licensed by the current semrep ontology. since neither criterion nor are met, it follows that the predication does not fulfil criterion, either. as a result, the predication in is not included when interannotator agreement calculation is restricted by any of the available domain knowledge criteria above. adjudication finally, the third annotator examined the annotations provided by each annotator and adjudicated them to create the current gold standard. during this phase, annotator c was free to discuss the annotations with the actual annotator to understand his/her reasoning. this phase concluded in four weeks.  in this section, we provide detailed information regarding different phases of the annotation, including the number of semantic predications annotated, their distribution by ontological predicate types, and indicator types. we report strict and relaxed interannotator agreement using various equivalence criteria and domain knowledge perspectives. for the main annotation phase, we measure interannotator agreement at the ontological predicate and ontological predication level, highlighting some of the annotation difficulties with respect to biomedical subdomains. examining annotation differences in the practice phase, we refined our guidelines for the main annotation phase and extended interannotator agreement measures, which we also discuss in this section. practice phase in this first phase, sentences were annotated, with an average of semantic predications per sentence. we show the number of predications annotated by each annotator at this phase in table and the most frequently annotated ontological predicates with their annotation frequency in table indicator types that signal predications are provided in table which shows that verbal predicates are considered at similar rates among annotators, while there are larger gaps in consideration of other types. interannotator agreement was fair to moderate in this phase, as shown in table the highest agreement occurred between annotators a and c, consistent with the largely similar patterns they exhibited in the types of ontological predicates they annotated, as shown in table unsurprisingly, the interannotator agreement increases, albeit at different rates, when equivalence of textual mentions is also considered as the basis for agreement as well as the conceptual equivalence. refining the guidelines the results of the practice phase helped us identify several aspects of annotation that are challenging. these were discussed among the annotators and several additions and clarifications were made in the annotation guidelines. these included the following: selecting appropriate umls metathesaurus concepts as stated earlier, we provided the annotators umls metathesaurus concepts identified by metamap for reference. after the first phase, our analysis revealed that annotation behavior was diverse with regard to these metamap-provided concepts. one annotator almost exclusively relied on them, while the other two more extensively utilized the umls to find more adequate concepts. in addition, one of the annotators considered metathesaurus concepts from newer umls releases. in the light of discussions over these differences, annotators were instructed to use the following decision criteria in selecting an appropriate concept: if the concept identified by metamap adequately describes the textual mention, use it. a concept that is clearly more general than the one stated in the text relation holds between them) cannot be used as a replacement. if metamap does not associate any concept with the textual mention or if the associated concept seems inadequate, try to find a umls 2006aa concept that is appropriate. when there is no corresponding umls 2006aa concept, a concept from a newer umls release may be used, provided that it does not violate the decision criterion above. finding entrez gene terms semrep uses entrez gene as a supplementary vocabulary to umls metathesaurus when it identifies gene/protein terms. the mapping to entrez gene terms is achieved via pattern matching in semrep. when available, these symbols were also provided to annotators in the practice phase. similar to their reactions to metamap-supplied umls concepts, annotators also relied on entrez gene terms to varying degrees. one annotator tried to find entrez gene terms for all gene/protein mentions, whether or not a corresponding umls concept was found, and disambiguate model organisms using the context surrounding the textual mention, while another completely ignored entrez gene terms. we identified two issues regarding these terms: when is it required to annotate a textual mention with an entrez gene term? is disambiguation with respect to model organisms necessary? we resolved these issues by concluding that the umls is the primary knowledge source for semrep, and a gene/protein mention needs to be annotated with an entrez gene term only if no corresponding umls concept is found. further, we decided that entrez gene terms should be limited to those in the homo sapiens taxon, since semrep currently only considers this taxon, and context beyond individual sentences may need to be taken into account for determining the model organism. semantic network ontological predicates the practice phase revealed some confusion with respect to the difference between an ontological predicate and its textual mention. for instance, for the fragment "... an association of 5-httlpr with intensity dependence of auditory-evoked potentials...", two annotators annotated the semantic predication "c0170657: serotonin transporter -associated_with-c0015215:auditory evoked potentials." the ontological predicate associated_with is used in semrep in a restricted sense, referring to only gene-disease association. although the semantic type of the object in the predication is not a disorder, the annotators used this ontological predicate because they were influenced by the choice of textual mention "association." the difference was made explicit in the guidelines. we further clarified the definitions of several ontological predicates, taking into account how they are conceptualized in the semrep ontology. for instance, we noted that comparative predicates are limited to substance and therapeutic procedure semantic types for the time being, while process_of is limited to disorder subjects. we also distinguished interacts_with/affects and inhibits/disrupts predicate pairs more explicitly: interacts_with and inhibits relations hold between substances, while affects and disrupts take processes as objects. hypernymic relations there was disagreement over what constitutes a hypernymic relation. problematic annotations included "c0050940: lansoprazole -isa-c0599473: enantiomer " for the noun phrase "lansoprazole enantiomers" and "c1443775: epidermal growth factor receptor inhibitor -isa-c0450442:agent " for the fragment "two oral egfr inhibitors... are small-molecule agents...." we concluded our discussion on this topic by distinguishing between taxonomic relations that pertain to structural aspects and other kinds of taxonomic relations, including those pertaining to functional aspects. it was clarified in the guidelines that structural taxonomic relations are not hypernymic. extending interannotator agreement calculation as briefly mentioned earlier, based on analysis of the agreement results in the practice phase, we extended equivalence criteria for interannotator agreement calculation. we describe these extensions in more detail in this section. equivalence of semantic predications we found that two distinct semantic predications derived from the same textual mentions may be too close in meaning to select one over the other. for instance, we identified cases where one annotator used a predication with the predicate location_of, while the other preferred one with the predicate part_of and inverse arguments. for the textual fragment "... alleles in two cell lines", one annotator annotated "c0002085: alleles -part_of-c0007600: cell line ", while the other annotated "c0007600: cell line -location_of-c0002085: alleles." such similarity in meaning is partly rooted in the umls semantic network. for instance, the ontological predication "virus-location_of-biologically active substance" is a valid ontological predication, as is one with inversion of these arguments and part_of as predicate. in addition, some indicator expressions can be ambiguous with respect to their meaning; for instance, the preposition "in" may map to either predicate in this example. examining instances of the location_of/part_of variation in the practice phase, we concluded that at the biomolecular level, the difference in meaning is blurred to the extent that two semantic predications may be considered equivalent for interannotator agreement calculation. however, we also noted that there are clear exceptions. for example, consider the sentence fragment "... truncated bid, which translocates to mitochondria...". while the predication "c0026237: mitochondria -location_of-c1144558: tbid protein " seems acceptable for this fragment, its counterpart with the predicate part_of -part_of-c0026237: mitochondria ") does not. to handle such exceptions, we manually judged the cases in which there was agreement due to this equivalence criterion for correctness as a post-processing step to interannotator agreement calculation and corrected the agreement score accordingly. a situation similar to location_of/part_of equivalence concerns produces/part_of predicates and the verbal indicator "express" and its derivations. a relation indicating gene expression events does not currently exist in the semrep ontology, leading one annotator to use part_of consistently for such events, and the other to use produces. in the fragment "the expressions of c-myc, ki- mmp-... in cancer tissues", one annotator chose "c1334508: mki gene -part_of-c0040300: body tissue " and the other "c0040300: body tissue -produces-c1334508: mki gene." we treat these cases as equivalent as well. interannotator agreement using these two equivalence criteria is denoted as pe below. correspondence of gene and gene products in the molecular biology literature, gene names are often used to denote gene products, leading to term ambiguity. for instance, in the fragment "tnfr1/fas engagement results in the cleavage of cytosolic bid to...", without knowing the context, it is difficult to determine whether tnfr fas or bid refer to genes or gene products. this ambiguity extends to the umls metathesaurus as well. for instance, bid maps to "c1332410: bid gene " and "c0531588: bid protein." on the other hand, tnfr is mapped to "c0255808: tumor necrosis factor receptor 1a " as well as "c1363984: tnfrsf1a gene, while the equivalence in the case of tnfr is ignored. although the two concepts corresponding to the textual mention are considered related terms in umls, it is difficult to establish their correspondence from their concept names alone. correspondence of this type is denoted as gp below. limiting agreement calculation by available domain knowledge to assess the difficulty of annotation due to the open-ended, exploratory nature of our annotation, we also limited interannotator agreement calculation based on the availability of conceptual knowledge to the annotator as well as that of relational knowledge and both, as mentioned earlier. our intuition in limiting calculation to a subset of annotated predications was that there would be more substantial agreement when the annotator chooses concepts and ontological predications from a predefined set. main annotation phase two annotators were involved in the main annotation phase. the average number of semantic predication annotations per sentence was slightly lower than that in the practice phase; one annotator was relatively consistent between the practice and main annotation phases, while the other annotated significantly fewer in the main phase, as shown in table the interannotator agreement results for the main annotation phase are presented in table with the improved guidelines provided to the annotators, the basic strict agreement between the annotators increased from to, while the basic relaxed agreement increase was higher: from to. additionally, we computed interannotator agreement using the extended equivalence criteria. adding the predication equivalence criterion increased the agreement by approximately 3%, while gene/gene product correspondence criterion provided a small increase of % overall. limiting the comparison to conceptual knowledge provided, the agreement reaches substantial levels with an increase of approximately 12%. relational domain knowledge alone had less effect on agreement. with both conceptual and relational knowledge provided to annotators, the agreement rises more than 15% in comparison to the basic case. we only show relaxed agreement in the basic case, where the increase from the strict counterpart is about %. we note that this trend was observable in the cases of predication filtering, as well. we consider strict agreement with base + pe + gp equivalence as the main agreement criterion in the rest of the paper. pe: predication equivalence, gp: gene/gene product correspondence, ck: conceptual knowledge, rk: relational knowledge, crk: conceptual and relational knowledge when we consider the indicator types that signal predications, modifier-head constructions appear more frequently as indicators than in the practice phase, as shown in table in addition, due to a clarification regarding how the indicator type for hypernymic relations should be annotated, we observed an increase in the frequency of the indicator type spec, which essentially indicates that the hypernymic relation between the concepts should be licensed by the umls metathesaurus hierarchy in addition to being indicated by a textual clue. the ontological predicate distribution in the main annotation phase is presented in table comparing the distribution to that in the practice phase, one noticeable trend is a relative increase in the number of process_of, part_of, treats and affects predicates and a decrease in inhibits and interacts_with predicates, possibly due to clarifications over definitions of the predicates after the practice phase. affects and inhibits predicates are not shown in table and table respectively, since they occur less frequently in the respective phases. however, we note that the increase in the number of affects predicates from the practice phase to the main annotation phase is %, whereas the decrease in that of inhibits is 8%. it is also worth mentioning that the high frequency of inhibits in the practice phase seems artificial, since most of those annotations were the result of a single instance that involved complex coordination. during this phase, we also computed interannotator agreement at the ontological predicate level to assess whether some types are more difficult to annotate than others. among the most frequent predicates, there is less agreement on predicates relating biomolecular entities or processes. on the other hand, predicates concerning disorders, anatomical parts and population groups yield overall higher agreement. further examining ontological predicates with highest and lowest interannotator agreement, these findings are confirmed. among ontological predicates annotated more than times, the highest disagreement rates were associated with those involving biomolecular entities and processes. on the other hand, the highest agreement rate was for process_of and prevents, both of which are disease-related ontological predicates. similar topical trends were observable at the ontological predication level, as well. among those ontological predications annotated more than times between annotators, the highest disagreement rates were found in those concerning the biomolecular entities and processes, while disagreement was lowest in predications involving disorders and population groups. these results clearly establish bio-molecular relations and disorder/population relations as two extremes in the spectrum in terms of ease of annotation. aapp: amino acid, peptide, or protein; celf: cell function; cell: cell; dsyn: disease or syndrome; gngm: gene or genome; humn: human; inpo: injury or poisoning; mamm: mammal; neop: neoplastic process; rcpt: receptor; sosy: sign or symptom annotators were allowed to provide comments for their annotations, and we examined the comments corresponding to the predications on which there was disagreement. the most frequent comments in disagreement cases are given below. the number of disagreement instances and the corresponding interannotator agreement scores are given in parentheses. the interannotator agreement score for a specific type of comment was calculated by considering all the predications marked with that comment, regardless of whether there was disagreement or not. a. unsatisfactory umls concept extracted by metamap: the umls concept identified by metamap is replaced with a more appropriate concept from the umls 2006aa release. b. no indicator rule: there is no indicator rule mapping from the textual mention to the ontological predicate in semrep. c. relational domain knowledge unavailable: the annotated semantic predication does not correspond to an ontological predication in the semrep ontology. d. newer umls release concept: a non-2006aa umls concept is preferred. e. complete metamap miss: a 2006aa umls concept that metamap fails to identify is used. adjudication phase in the adjudication phase, annotator c examined the annotations provided by a and b, resolving differences and determining the gold standard. as well as selecting one annotation from one set over another from the other set, the adjudicator could also override annotations from both sets or keep annotations from both sets. the final semantic predication count is an average of predication per sentence. the maximum number of predications per sentence is the frequency distributions by ontological predicate types and indicator types after reconciliation are presented in table and table respectively. while interannotator agreement at this stage is not meaningful since the annotation sets were available to the adjudicator, we measured it to assess whether the adjudicator clearly prefers one set of annotation to the other. the adjudicator agreed with annotator b at a much higher rate. this seems likely due to the fact that the adjudicator took b annotations as the basis and only added an a annotation when b annotation was deemed incorrect. discussion we conducted a relatively open-ended, multi-phase annotation study, in which we aimed to assess the feasibility of iteratively constructing a reasonable gold standard reference based on umls domain knowledge. the results presented in the previous section show that this is a feasible undertaking. on the other hand, they confirm that conceptual annotation is extremely challenging; the main difficulty is mapping textual mentions to ontological terms, a time-consuming and labor-intensive task. this is evidenced by the fact that interannotator agreement increases to acceptable levels when the comparison is limited to conceptual knowledge available to the annotators. we discuss some of the challenges in more depth below. mapping text to ontological concepts the core notion in mapping textual mentions to ontological concepts is semantic equivalence, which entails that the ontological concept must express the meaning of the textual mention adequately, and should not correspond to a more general or more specific meaning than the textual mention. however, this equivalence is often very difficult to establish. in our study, the difficulty of establishing semantic equivalence is further compounded by the nature and size of the primary reference terminology we used for conceptual information, the umls metathesaurus. in addition, we defined the task in an open-ended manner and allowed the annotators to use newer umls releases when the primary knowledge source is inadequate, which increased the complexity of the task and lowered the interannotator agreement. to ease the burden for the annotator, we provided umls concepts identified by metamap as reference, corresponding to the domain knowledge available to semrep. while this was useful to a large extent, it did not prevent disagreements regarding what constitutes an adequately expressive, semantically equivalent mapping. in some cases, the concepts identified by metamap may even be misleading, as we found out, since one annotator often considered a given concept adequate for the textual mention in question, while the other was dissatisfied with the mapping and identified a clearly better umls metathesaurus concept. for example, metamap failed to completely map the noun phrase "the d receptor" to the existing umls concept "c0082341: dopamine d receptor" and identified the concept "c0597357: receptor" instead. one annotator considered this more general concept adequate, while the other identified the former as the more adequate mapping. a similar situation occurred with "minimally invasive surgery," where metamap identified "c0543467: operative surgical procedures", while, in fact, there exists a more suitable concept "c0282624: surgical procedures, minimally invasive" for the mention. there were also cases where metamap failed to identify a concept corresponding to a textual mention, and it was difficult for the annotator to see that the textual mention may in fact have a umls counterpart. for instance, only one annotator found and used the concept "c1254042: anatomical maturation " to correspond to the head of the more specific noun phrase "lung maturation." despite these difficulties, however, evidence from the annotators indicates that concepts identified by metamap were overall helpful to annotators and lightened the annotation load considerably. furthermore, interannotator agreement increased significantly when only metamap-supplied concepts were used, supporting our basic intuition that selecting a concept from a predefined list is less demanding than searching for a semantically equivalent concept from a sizable terminology and assessing its adequacy, an essentially subjective task. in other gold standard annotation studies, such as the craft corpus and the clef corpus, carefully selected, well-curated vocabularies or ontologies were used. for example, the craft corpus made use of six obo ontologies. however, finding the semantically equivalent concept for a textual mention is still found to be challenging. in our annotation, consistent with the semrep methodology, we did not limit ourselves to specific umls vocabularies and aimed to be more general in our coverage. however, this generality has its drawbacks, as expected. while synonymous terms from different vocabularies are clustered together to form concepts in the umls metathesaurus, there are still a significant number of distinct concepts that are very close in meaning. for example, for the noun phrase "mineral metabolism impairment," metamap identified two separate concepts: "c0678715: mineral metabolism" and "c0011155: deficiency". neither annotator found these mappings adequate; however, they found different concepts, similar in meaning, to correspond to the meaning of the mention. one identified "c0687148: mineral deficiency", while the other identified "c0154260: disorder of mineral metabolism". while these concepts are not exactly the same, the difference between their meanings is arguably very small. a similar situation holds between "c0403716: calculus in renal pelvis" and "c0022650: kidney calculi" for the phrase "pelvic stones." the problem seems even more acute when it comes to gene and gene product concepts, as exemplified earlier. it seems necessary to take into account such overlap and similarity in meaning, both in computing interannotator agreement and in system evaluation. we attempted to address the semantic equivalence of concepts to some extent by devising an equivalence criterion involving gene and gene products; however, this criterion is limited to a single, clear pattern, and is unable to accommodate more complex cases of equivalence. one principled solution may be to allow annotators to annotate more than one concept for a given textual mention when a clear preference for a concept cannot be established and to give partial credit for any of the concepts. this would clearly increase annotation complexity, and it may be more feasible as a postprocessing step. annotation by domain our results also show that difficulty of annotation varies by the biomedical subdomain. ontological predications relating to biomolecular entities and processes are most challenging to annotate, while those on the topic of population characteristics of disease seem to be the most straightforward. we explain the difficulty of annotating biomolecular relations by observing the following: a. molecular biology text is hardest to read and interpret for a non-expert and none of the annotators are experts in this subdomain. furthermore, as friedman et al. have shown, biomolecular domain text constitutes a sublanguage, with very specific characteristics, such as complex and nested relations as well as more prevalent syntactic ambiguity. one interesting, syntactically ambiguous case involved the fragment "... il-1beta-induced ros formation, nf-kappab activation, and mcp- secretion...", where one annotator took the modifier "il-1beta-induced" as modifying "nf-kappab activation" and "mcp- secretion" as well as "ros formation", and annotated the predications given in, while the other annotator took the modifier to modify "ros formation" only, and did not annotate the predications in. c0021753: interleukin- beta -stimulates-c0128897: monocyte chemoattractant protein- c0021753: interleukin- beta -stimulates-c0079904: nf-kappa b b. the coverage of the umls semantic network with respect to molecular biology is perhaps the least extensive. in fact, we have extended the umls semantic network to create the semrep ontology in prior work specifically to redress this gap. c. based on evidence from the annotation process, it seems more challenging for metamap to map textual mentions of biomolecular entities to umls metathesaurus than to map those of entities from other subdomains. for example, the semantic type most frequently associated with concepts identified by annotators and not by metamap was amino acid, peptide, or protein. in contrast to genomic concepts and relations, disorder and population group concepts and their relations are well covered in the umls and metamap has less difficulty in mapping text to these concepts. in addition, text concerning disease characteristics is overall easier to interpret for a non-expert. diachronic change in domain knowledge another complicating factor in conceptual annotation is that the ontologies and vocabularies the concepts are derived from may change over time. there are two alternatives to address this situation: using a static snapshot of the knowledge source re-annotating at each update of the knowledge source. our methodology was similar to the first alternative: we adopted the 2006aa release of umls as the primary source for conceptual information, while also recognizing that newer releases of umls have wider coverage, especially with respect to new drug or gene/protein names, thus allowing the annotators to use these resources, if necessary. since we enforced text-bounded annotations, we believe that it will be possible to update the gold standard semi-automatically at future updates of the knowledge sources, using metamap or a similar program.  we have presented the construction of a semantic predication gold standard from biomedical literature text using the conceptual annotation paradigm. manual conceptual annotation is considered extremely challenging, and our results confirm this perception, while also confirming that reasonable interannotator agreement could be achieved iteratively, consistent with the findings of bada et al.. while the domain knowledge we used reflects the application-specific aspect of our annotation, we believe that our analysis and discussion provide important insights for future efforts in this area. the resulting gold standard constitutes the first resource, to our knowledge, in the biomedical domain that incorporates conceptual annotation of semantic relations in a wide variety of subdomains. two sets of annotations and the adjudicated gold standard are made publicly available for research purposes. a umls license is required. the corpus size is relatively small and may be insufficient for training information extraction systems. however, we believe it can serve as a benchmark to evaluate independently developed systems based on umls knowledge sources. our goal is to use it for this particular purpose, as well as to guide future system development. authors' contributions hk conceived of the study, led the annotation effort, performed the analyses, and drafted the manuscript. hk, gr, and mf contributed to annotation, guideline development, and helped draft the manuscript. tcr directed the research and helped draft the manuscript. all authors read and approved the final manuscript. appendix ontological predicate definitions ontological predicate definitions and an example for each of the definitions are provided below. note that each of the following can also be negated: • administered_to: given to an entity, when no assertion is made that the substance or procedure is being given as treatment. patients with single brain lesion received an extra gy x radiotherapy.... c0034618: radiation therapy -administered_to-c0030705: patients • affects: produces a direct effect on. implied here is the altering or influencing of an existing condition, state, situation, or entity. this includes has a role in, alters, influences, predisposes, catalyzes, stimulates, regulates, depresses, impedes, enhances, contributes to, leads to, and modifies. bap and its caspase cleavage product regulate cell surface expression of tetraspanins and integrin-mediated cell survival. c1424489: bcap gene -affects-c0007620: cell survival • associated_with: has a relationship to. ep plays a critical role in tumorigenesis in mouse mammary gland... c1419062: ptger gene -associated_with-c1326912: tumorigenesis • augments: expands or stimulates a process. nicotine induces conditioned place preferences over a large range of doses in rats. c0028040: nicotine -augments-c0815102: place preference learning • causes: brings about a condition or an effect. implied here is that an agent, such as for example, a pharmacologic substance or an organism, has brought about the effect. this includes induces, effects, evokes, and etiology. neurocysticercosis is one of the major causes of neurological disease... c0338437:neurocysticercosis -causes-c0027765: nervous system disorder • coexists_with: occurs together with, or jointly. food intolerance-related constipation is characterized by proctitis. c0009806: constipation -coexists_with-c0033246: proctitis • converts_to: changes from one form to another.... plasma nitrite is readily oxidized to nitrate within plasma... c0028137: nitrites -converts_to-c0699857: nitrate • complicates: causes to become more severe or complex, or results in adverse effects. infections can trigger gbs and exacerbate cidp. c0021311: infection -complicates-c0393819: polyradiculoneuropathy, chronic inflammatory demyelinating • diagnoses: distinguishes or identifies the nature or characteristics of. manometry showed a higher anal sphincter resting pressure... c0024751: manometry -diagnoses-c0429217: anal sphincter pressure • disrupts: alters or influences an already existing condition, state, or situation. produces a negative effect on. overexpression of nf-kappab inhibits tumor cell apoptosis. c0079904: nf-kappab -disrupts-c0162638: apoptosis • inhibits: decreases, limits, or blocks the action or function of. in recent studies, the bdnf expression was reduced by typical neuroleptics. c0040615: antipsychotic agents -inhibits-c0107103: brain-derived neurotrophic factor • interacts_with: substance interaction. here we show that chymases, which are chymotryptic peptidases secreted by mast cells, hydrolyze hgf... c0055673: chymase -interacts_with-c0062534: hepatocyte growth factor • isa: the basic hierarchical link in the umls semantic network. if one item isa another item then the first item is more specific in meaning than the second item. the sympathetic neurotransmitter norepinephrine has been found... c0028351: norepinephrine -isa-c0027908: neurotransmitters • location_of: the position, site, or region of an entity or the site of a process. we report a case of primary cardiac epithelioid hemangioendothelioma arising from the right atrium of a 2-month-old infant. c1269890: entire right atrium -location_of-c0206732: hemangioendothelioma, epithelioid • manifestation_of: that part of a phenomenon which is directly observable or concretely or visibly expressed, or which gives evidence to the underlying process. this includes expression of, display of, and exhibition of. recurrence of glomerulopathy underlying esrd was frequent for igan, fsg... c1261469: end stage renal failure -manifestation_of-c1398810: glomerulopathy • method_of: the manner and sequence of events in performing an act or procedure.... because of the use of sscp as a screening method and sequencing only a part of tshr exon c0243031: single-stranded conformational polymorphism -method_of-c0220908:screening procedure • occurs_in: has incidence in a group or population. older populations are more prone to bone loss with weight loss... c0599877: loss; bone -occurs_in-c1518563: older population • part_of: composes, with one or more other physical units, some larger whole. this includes component of, division of, portion of, fragment of, section of, and layer of. the probasal bodies and microtubules within the blepharoplast cavities... c0026046: microtubules -part_of-c0230744: basal body of cilium or flagellum, not bacterial • precedes: occurs earlier in time. this includes antedates, comes before, is in advance of, predates, and is prior to.... the risk of tissue plasminogen activator-induced hemorrhagic transformation following ischemic stroke in mice... c0948008: ischemic stroke -precedes-c1096400: haemorrhagic transformation stroke • predisposes: to be a risk to a disorder, pathology, or condition.... high ghrelin levels contribute to obesity in prader-willi syndrome... c0911014: ghrelin -predisposes-c0028754: obesity • prevents: stops, hinders or eliminates an action or condition. ipsapirone and ketanserin protects against circulatory shock, intracranial hypertension, and cerebral ischemia during heatstroke. c0123905: ipsapirone -prevents-c0151740: intracranial hypertension • process_of: disorder occurs in organism.... no information is available in cad patients with normal glomerular filtration rate. c0010054: coronary arteriosclerosis -process_of-c0030705: patients • produces: brings forth, generates or creates. this includes yields, secretes, emits, biosynthesizes, generates, releases, discharges, and creates. human epcs express functional par-.. c0038250: stem cells -produces-c0668084:receptor, par- • stimulates: increases or facilitates the action or function of. candesartan therapy significantly reduced inflammation and increased adiponectin levels... c0717550: candesartan -stimulates-c0389071: adiponectin • treats: applies a remedy with the object of effecting a cure or managing a condition. this study initially surveyed patients with clinical stage ib or iia cervical adenocarcinoma treated with radical hysterectomy and pelvic lymphadenectomy. c0677962: radical hysterectomy -treats-c0279672: cervical adenocarcinoma • uses: employs in the carrying out of some activity. this includes applies, utilizes, employs, and avails. pre-emptive therapy with oral valganciclovir for cmv infections... c0087111: therapeutic procedure -uses-c0909381: valganciclovir • compared_with: comparative predicate. compared with placebo, candesartan therapy significantly lowered plasma hscrp levels... c0032042: placebos -compared_with-c0717550: candesartan • higher_than: comparative predicate. losartan was more effective than atenolol in reducing cardiovascular morbidity... c0126174: losartan -higher_than-c0004147: atenolol • lower_than: comparative predicate. amoxicillin-clavulanate was not as effective as ciprofloxacin for treating uncomplicated bladder infection in women. c0054066: amoxicillin-potassium clavulanate combination -lower_than-c0008809: ciprofloxacin • same_as: comparative predicate. candesartan is as effective as lisinopril once daily in reducing blood pressure. c0717550: candesartan -same_as-c0065374: lisinopril predicate types abbreviations and definitions • verb: verbal predicates • nom: nominalizations or other argument-taking nouns • adj: adjectival predicates • prep: prepositional predicates • aux: auxiliary verb predicates • part: past participial predicates • mod/head: the predication is the result of a noun phrase construction • spec: the predication is hypernymic and must be licensed by the umls metathesaurus hierarchy, that is, the subject must be a descendant of the object in the umls metathesaurus hierarchy • infer: the predication is the result of inference based on two other existing predications • complex: the indicator is a multi-word expression, but not a multi-word lexical item. sample annotation the following information is provided to the annotators: ab. high-dose aspirin at a therapeutic dose does not affect cyp2e activity in humans. c0444956|high dose|qnco|||high-dose|901|1|9 c0004057|aspirin|orch,phsu|||aspirin|901|11|17 c0302350|therapeutic|ftcn|||therapeutic|888|24|34 c1446466|dose|qnco|||dose|888|36|39 c1518422|not|ftcn|||not|1000|46|48 c0919426|cyp2e gene|gngm|1571|cyp2e1|cyp2e1|888| 57|62 c0439167|% activity|qnco|||activity|888|64|71 c0020114|human|humn|||humans|1000|76|81 in the sentence line above, ab. is a sentence identifier. in the subsequent entity lines, the fields correspond to the following: concept identifier umls metathesaurus preferred name umls semantic type abbreviation entrez gene id entrez gene name textual mention that maps to the entity metamap score for the entity first character position of text denoting entity last character position of text denoting entity from these provided annotations, the following two semantic predication annotations are expected: c0004057|aspirin|orch,phsu|orch|||neg_interacts_with|c0919426|cyp2e gene|gngm,aapp|gngm|1571|cyp2e1|verb|affect|aspirin|cyp2e1 c0020114|human|grup,humn|humn|||location_of|c0919426|cyp2e gene|gngm,aapp|aapp|1571|cyp2e1|prep|in|humans|cyp2e1 fields 1- and 8- correspond to fields 1- of the entity lines for subject and object, respectively. fields and correspond to the individual semantic types that license the predication. fields 5- and 12- correspond to fields 4- of the entity lines for subject and object, respectively. field corresponds to the ontological predicate. field is the ontological predicate type. field is the textual mention corresponding to the ontological predicate, field corresponds to the subject argument and field to the object argument. 
the success of such practical applications crucially depends on the quality of extracted relations, which can be assessed against a gold standard reference. most such references in biomedical text mining focus on narrow subdomains and adopt different semantic representations, rendering them difficult to use for benchmarking independently developed relation extraction systems. in this article, we present a multi-phase gold standard annotation study, in which we annotated sentences randomly selected from medline abstracts on a wide range of biomedical topics with semantic predications. semantic relations increasingly underpin biomedical text mining and knowledge discovery applications. we measured interannotator agreement and analyzed the annotations closely to identify some of the challenges in annotating biomedical text with relations based on an ontology or a terminology. mapping text to ontological concepts emerges as the main challenge in conceptual annotation. while the resulting gold standard is mainly intended to serve as a test collection for our semantic interpreter, we believe that the lessons learned are applicable generally. the umls metathesaurus served as the main source for conceptual information and the umls semantic network for relational information. we obtain fair to moderate interannotator agreement in the practice phase. while interannotator agreement in the practice phase confirms that conceptual annotation is a challenging task, the increasing agreement in the main annotation phase points out that an acceptable level of agreement can be achieved in multiple iterations, by setting stricter guidelines and establishing semantic equivalence criteria. with improved guidelines and additional semantic equivalence criteria, the agreement increases by 12% in the main annotation phase. annotating predications involving biomolecular entities and processes is particularly challenging. in addition, we find that agreement increases to when the agreement calculation is limited to those predications that are based only on the explicitly provided umls concepts and relations.
 semantic relations increasingly underpin biomedical text mining and knowledge discovery applications. the success of such practical applications crucially depends on the quality of extracted relations, which can be assessed against a gold standard reference. most such references in biomedical text mining focus on narrow subdomains and adopt different semantic representations, rendering them difficult to use for benchmarking independently developed relation extraction systems. in this article, we present a multi-phase gold standard annotation study, in which we annotated sentences randomly selected from medline abstracts on a wide range of biomedical topics with semantic predications. the umls metathesaurus served as the main source for conceptual information and the umls semantic network for relational information. we measured interannotator agreement and analyzed the annotations closely to identify some of the challenges in annotating biomedical text with relations based on an ontology or a terminology.  we obtain fair to moderate interannotator agreement in the practice phase. with improved guidelines and additional semantic equivalence criteria, the agreement increases by 12% in the main annotation phase. in addition, we find that agreement increases to when the agreement calculation is limited to those predications that are based only on the explicitly provided umls concepts and relations.  while interannotator agreement in the practice phase confirms that conceptual annotation is a challenging task, the increasing agreement in the main annotation phase points out that an acceptable level of agreement can be achieved in multiple iterations, by setting stricter guidelines and establishing semantic equivalence criteria. mapping text to ontological concepts emerges as the main challenge in conceptual annotation. annotating predications involving biomolecular entities and processes is particularly challenging. while the resulting gold standard is mainly intended to serve as a test collection for our semantic interpreter, we believe that the lessons learned are applicable generally. 