descriptions and settings by the evaluators. When inter- annotator agreement proves to be challenging, other groups have adopted strategies to further improve the ﬁnal gold standard annotation set (10). For this work, sentences deemed diﬃcult to evaluate were set aside for later annotation by discussion and group consensus. Therefore, the gold standard was comprised of annota- tions with three-way agreement from the individual assess- ment, plus the consensus annotations. When each evaluator’s set of ‘straightforward’ independent annota- tions (n = 237 out of 300 annotations, 79%) was com- pared to the gold standard, agreement averaged 92.9%. Sentence–relationships for which there was no agreement or only two-way agreement were discussed and annotated by consensus (14%, n = 14). While overall inter-annotator agreement cannot be measured with the annotation pro- cedure devised, the agreement metrics described are help- ful as proxies to the level of ambiguity of the task. Pairs that are only evaluated by consensus diﬀer from the rest largely in the knowledge required from the evaluator to elucidate the annotation, rather than in the factual ambi- guity of the sentence. Hence, discussion and consensus assessment of pairs yields a better annotation set than individual annotation. An agreement of 92.9% can be con- sidered high for the task. Only 7% (n = 7) of sentence– relationship pairs were marked irrelevant. Following the same guidelines, we created the test corpus. Each of the three evaluators assessed diﬀerent sets of 500 sentence–relationship pairs, 250 for diabetes and 250 for cancer. Sentences deemed not straightforward were left without annotation (n = 126, 8.4%) and later annotated by consensus. Only 6.3% of all relationships (n = 95) were considered irrelevant, demonstrating the high precision of the retrieval query. Overall, the proce- dure used to construct the corpus assured high quality to the 1405 sentence–relationship annotation. Genes were mapped to standard nomenclature using our protein/ gene thesaurus. We measured the accuracy of this map- ping using 250 sentences from the corpus. Accuracy was 89%. We performed detection of increased, decreased or unknown perturbations using machine learning techni- ques. For that purpose, we constructed a vector of fea- tures for each relationship–sentence pair using the test corpus above. The vector was composed of diﬀerent sets of features. One of them represented token presence as a binary vector of token weights wi, d = {w1,. . .,w|T|}, where T is the set of sentence tokens: a weight has value 1 if the token is present in the sentence and value 0 otherwise, an approach called set of words (SOW) (11). Tokens were created by stemming and tokenization of the sentence words. Hyphenated names were considered both as single and separate tokens in the SOW in order to capture aﬃxes like ‘anti-.’ Since proximity to the gene name can be important in determining a token’s role in describing a perturbation, the sentence was further divided in several sections relative to the gene name’s position. Each section was characterized by a set of features using SOW. The sections considered were: n tokens before the gene name, where n = {5, 10, all}, and n tokens after the gene name, where n = {10, all}. We noted that many perturbation Nucleic Acids Research, 2009, Vol. 37, No. 3 773 descriptions were adjacent or very close to the gene name (e.g. ‘overexpression of p53’). Hence, we included a feature with the distance between the gene name and the beginning of the sentence (e.g. distance of 0 or 1 may indicate that the perturbation is unknown, e.g. ‘TNF-a induces apoptosis . . .’ or ‘The p53 gene . . .’). We also cre- ated a set of features using a small perturbation ontology developed independently over a disease-agnostic set of retrieved sentences. If a member of the ontology was pres- ent in the sentence a feature was added with value 1, oth- erwise with value 0. All the feature sets described were integrated in a single feature vector for each sentence. An algorithm based on the principles of maximum entropy (12, http://homepages.inf.ed.ac.uk/s0450736/ maxent_toolkit.html) was trained using 80% of the test corpus, randomly sampled from the cancer and diabetes sets, and tested over the remaining 20% (results were aver- aged over 10 runs). The training allowed the algorithm to predict the presence of a perturbation and its direction. Performance measures were evaluated and compared favorably to an SVM algorithm (13). To create the analysis corpus, we extended the scope of our methodology used to retrieve the test corpus by elim- inating the disease-speciﬁc phenotype constraints in the retrieval query (hence, phenotypes were not included in the query). We applied the query to Medline diabetes abstracts after 1996 and retrieved 359 385 relationships related to diﬀerent conditions and phenotypes. This output was run through the machine-learning algorithm to create a wide-scope, disease-agnostic set of 191 240 per- turbation predictions. To create a diabetes-speciﬁc subset, only sentences from Medline abstracts containing the word diabetes in their MeSH descriptors were included. RESULTS There were signiﬁcant diﬀerences both in technique and direction between the cancer and diabetes perturbations in the test corpus, with decreased perturbations more preva- lent in cancer. Table 2 shows the performance of the Table 2. Perturbation extraction performance Count Precision (%) Recall (%) F-measure (%) Perturbation No ontology Only ontology Straightforward Increased No ontology Only ontology Straightforward Decreased No ontology Only ontology Straightforward 690 436 254 79.2 76.4 77.2 83.3 75.3 76.6 65.7 77.7 79.0 78.5 79.8 80.4 79.8 79.6 59.5 78.9 71.1 69.9 50.9 68.8 65.1 63.6 58.4 67.2 79.4 77.6 67.0 81.0 72.9 72.9 56.8 72.7 71.2 70.0 66.4 72.9 Precision, recall and macro-averaged F-measure were calculated using four diﬀerent combinations of feature sets: ‘full’, ‘no ontology’, ‘only ontology’ and ‘straightforward’. Baseline frequencies were 32.8% for increased perturbations and 18.1% for decreased perturbations.