Relevance Irrelevant Irrelevant Irrelevant Relevant Unknown Relevant Increase Relevant Increase Relevant Decrease Diﬀerent members of bcl-2 family may promote or inhibit apoptosis by synthesizing anti- and proapoptotic proteins The ATF6 and IRE1 pathways cooperatively caused apoptosis via induction of CHOP, activation of XBP1 and phosphorylation of JNK, and the PERK-eIF2a pathway counteracted the proapoptotic processes This result suggests that toxic products such as reactive oxygen species and aldehydes liberated by the action of polyamine oxidase on the acetylated polyamines formed by SSAT may enhance tumor development These results indicate that survivin is important for optimal development of bovine blastocysts and conﬁrm that survivin expression suppresses apoptosis of pre-implantation embryos Finally, we showed that after 7–9 days of incubation, MCH also inhibits proliferation of non-stimulated PBMC Taken together, LDM induces apoptosis in a p53-dependent manner when given at low doses, but in a p53-independent manner when given at high doses Soluble Flt-1 abolished hypoxia/VEGF-induced hyperpermeability Explanation Members of a gene family, not a gene, are causing apoptosis phenotype Pathways, not genes, are causing apop- tosis phenotype Metabolites, not proteins that generate them, are causing tumor development phenotype ‘Survivin expression’ does not indicate if there is a state change in amount ‘Incubation’ indicates MCH was added exogenously ‘Given at low doses’ indicates LDM was added exogenously Soluble forms of receptors create inactive complexes PMID 17404013 17464326 17675337 17075833 17537530 17534142 17311300 772 Nucleic Acids Research, 2009, Vol. 37, No. 3 Table 1. Examples of relevance and direction annotation Direction Sentence for annotation. The purpose of this corpus was to identify challenges in the annotation process and to reﬁne guide- lines that would help the annotators in choosing their evaluations. Annotating perturbations requires at times thorough knowledge of experimental biology, which can only be captured and organized within a solid framework. Therefore we sought to perform a preliminary analysis on a test corpus to improve on subsequent annotations. The design corpus was not used for any other purpose. This corpus was limited to sentences that included disease- related gene–phenotype relationships. Using the semantic relationship nomenclature of Tsai et al. (8), we selected reports in which the ‘agent’ that deliberately performs an action is represented by a gene or protein, and the ‘patient’ that is the recipient of the action corresponds to disease phenotypes. The information we sought stands in contrast to associative relationships, such as ele- vated protein levels correlating with disease activity. To create the design corpus, our initial query matched Medline sentences containing ordered triplets of a gene or protein name, a causative verb and a phenotype related to cancer or diabetes. Each member of the triplet was sepa- rated within the sentence by a maximum of three words. The retrieval was performed using Linguamatics I2E version 3.0 (Cambridge, UK). This software package has the ability to retrieve sentences from text that include word patterns established by user queries. The queries may include both syntactic and semantic constructs. Semantically, term classes can be deﬁned combining exter- nal ontologie and adding term morphological variations and regular expression patterns. Syntactically, it recog- nizes part-of-speech and shallow parsing constructs such as noun phrases, verbs and prepositions. In terms of scope, queries can be conﬁned to diﬀerent document or text sections, including abstracts, titles or sentences. For example, a sentence-level query may comprise a term class protein, a list of verbs (I2E can automatically generate morphological variants) and a list of phenotype objects. The gene/protein thesaurus was internally devel- oped and based on BioThesaurus (9). Forty verbs that signal causality (e.g. inhibition, stimulation) were com- piled manually, as was a set of disease-related phenotypes relevant to cancer (e.g. tumorigenesis, vascularization) and diabetes (e.g. serum glucose levels, weight gain). A set of 100 retrieved sentences and relationships were annotated by three PhD-level evaluators for relevance and direction of perturbation (see Table 1 for examples). ‘Relevance’ annotation was used to mark relationships that should be eliminated from the corpus for being irrel- evant to the intent of the retrieval query, e.g. if the gene was not acting as an agent. ‘Direction’ indicated the type of perturbation (increased, decreased or unknown) rela- tive to the starting state. For example, if a gene is added back to cells that carry a deletion in that gene to restore the wild-type state, it is noted as an increase, because the abundance of the gene was increased relative to the start- ing state. Sentences were annotated as ‘unknown’ if there was no stated perturbation, or direction could not be inferred at the sentence level. It is worth noting that although an unknown direction could be frequently resolved by reviewing the abstract or the full text of the article, we strictly limited our scope to evidence found at the sentence level. While most sentences contained straightforward descriptions of perturbations (e.g. ‘mutations in gene A’ or ‘protein B inhibition’), the broad range of perturbations in the literature, combined with the complex grammatical structures found in biomedical text, made some sentence– relationship pairs diﬃcult to annotate consistently, mainly due to diﬀerences in knowledge of experimental