Online assessment of protein interaction information extraction systems

Background. Manual database (DB) curation efforts extracting protein-protein interactions (PPIs) from publications are not able to cover the entire scientific literature on those interactions. Neither is high-throughput proteomics likely to reproduce this data in the foreseeable future. Existing text mining systems can extract a subset of the PPI data curated by DBs from publications. With the last three BioCreative challenges (II, II.5, and III), community evaluations of these PPI text mining systems have been made. Community challenges contribute to scientific exchange, define annotation standards, and generate lasting corpora to develop text mining approaches. At the same time, the FEBS Letters experiment on Structured Digital Abstracts (SDAs) evaluated the utility of incorporating authors in the curation process. Together, the joint results provide the basis for comparing the extraction of PPI data from literature using text mining, authors, or curators. Results. Some tasks of the challenges were PPI article selection, mapping interacting proteins to DB identifiers, detecting interacting protein pairs, as well as reporting the experimental methods used by the authors. In total, BioCreative II, II.5, and III attracted 52 research groups world-wide. The BioCreative Meta-Server (BCMS) collated the annotations from BioCreative II participants and was modified for the two following challenges to hold the challenges online. The MINT, IntAct, and BioGRID IMEx databases curated the gold standard annotations for 3,632 full-text articles and provided 19,642 classifications of abstracts as PPI-relevant or not. We introduce a new evaluation function FAPβ for scoring ranked lists, composed of the harmonic mean between Fβ-score and Average Precision (AP). It penalizes disproportionate result sets, and does not require a threshold. The evaluation showed that automated ranking of PPI articles could reduce the time needed to select relevant articles by one third. The best mapping systems achieved a F1-score of 29% and an AP of 26%. Conclusions. The BioCreative gold standard corpora form long-lasting assets, documented by hundreds of registered participants and citations. Article ranking can facilitate PPI article selection when compared to keyword-based queries. Better mapping systems are needed to improve the extraction quality of (normalized) interaction pairs, with the main mapping error arising from organism disambiguation issues. However, a relevant number of curator annotations are based on long-range discourse relations that are not captured by current approaches. Improving experimental method concept extraction will require mapping the scientific jargon to their ontology terms. The FAP-score identifies the best submissions and can establish cutoffs for high-recall/low-precision approaches. While automated systems did not generate annotations that match those of authors or curators, providing ensemble results from a meta-service and minimal human intervention point to new avenues for assisting curators and authors in retrieving relevant identifiers. List of Abbreviations AP Average Precision AUC Area Under Curve BCMS BioCreative Meta-Server BioNLP Natural Language Processing in Biology CRF Conditional Random Field FAP F-measured Average Precision FEBS Federation of European Biochemical Societies FN False Negative FP False Positive HMM Hidden Markov Model IAA Inter-Annotator Agreement IE Information Extraction IR Information Retrieval JPA Joint Probability of Annotation GN Gene Normalization GO Gene Ontology KGT Knowledge Generation from Text Max. Ent. Maximum Entropy (i.e., multinomial logistic regression) MCC Matthew’s Correlation Coefficient MeSH Medical Subject Headings (thesaurus) MEMM Maximum Entropy Markov Model NER Named Entity Recognition NLM National Library of Medicine NLP Natural Language Processing PoS Part-of-Speech PSI-MI Proteomics Standards Initiative Molecular Interactions (ontology) PPI Protein-Protein Interaction PR Precision/Recall ROC Receiver Operator Characteristic SDA Structure Digital Abstract SVM Support Vector Machine TAP Threshold Average Precision TF-IDF Term Frequency times Inverted Document Frequency TN True Negative TP True Positive UMLS Unified Medical Language System