Moussab/ORKG-training-evaluation-set huggingface dataset:
Research field: Natural Language Processing
Title: Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task
Abstract: Manually curating chemicals, diseases and their relationships is significantly important to biomedical research, but it is plagued by its high cost and the rapid growth of the biomedical literature. In recent years, there has been a growing interest in developing computational approaches for automatic chemical-disease relation (CDR) extraction. Despite these attempts, the lack of a comprehensive benchmarking dataset has limited the comparison of different techniques in order to assess and advance the current state-of-the-art. To this end, we organized a challenge task through BioCreative V to automatically extract CDRs from the literature. We designed two challenge tasks: disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. To assist system development and assessment, we created a large annotated text corpus that consisted of human annotations of chemicals, diseases and their interactions from 1500 PubMed articles. 34 teams worldwide participated in the CDR task: 16 (DNER) and 18 (CID). The best systems achieved an F-score of 86.46% for the DNER task—a result that approaches the human inter-annotator agreement (0.8875)—and an F-score of 57.03% for the CID task, the highest results ever reported for such tasks. When combining team results via machine learning, the ensemble system was able to further improve over the best team results by achieving 88.89% and 62.80% in F-score for the DNER and CID task, respectively. Additionally, another novel aspect of our evaluation is to test each participating system’s ability to return real-time results: the average response time for each team’s DNER and CID web service systems were 5.6 and 9.3 s, respectively. Most teams used hybrid systems for their submissions based on machining learning. Given the level of participation and results, we found our task to be successful in engaging the text-mining research community, producing a large annotated corpus and improving the results of automatic disease recognition and CDR extraction. Database URL: http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/
Extracted statement — data source: PubMed
Research field: Natural Language Processing
Title: Overview of the BioCreative VI Precision Medicine Track: mining protein interactions and mutations for precision medicine
Abstract: Abstract The Precision Medicine Initiative is a multicenter effort aiming at formulating personalized treatments leveraging on individual patient data (clinical, genome sequence and functional genomic data) together with the information in large knowledge bases (KBs) that integrate genome annotation, disease association studies, electronic health records and other data types. The biomedical literature provides a rich foundation for populating these KBs, reporting genetic and molecular interactions that provide the scaffold for the cellular regulatory systems and detailing the influence of genetic variants in these interactions. The goal of BioCreative VI Precision Medicine Track was to extract this particular type of information and was organized in two tasks: (i) document triage task, focused on identifying scientific literature containing experimentally verified protein–protein interactions (PPIs) affected by genetic mutations and (ii) relation extraction task, focused on extracting the affected interactions (protein pairs). To assist system developers and task participants, a large-scale corpus of PubMed documents was manually annotated for this task. Ten teams worldwide contributed 22 distinct text-mining models for the document triage task, and six teams worldwide contributed 14 different text-mining systems for the relation extraction task. When comparing the text-mining system predictions with human annotations, for the triage task, the best F-score was 69.06%, the best precision was 62.89%, the best recall was 98.0% and the best average precision was 72.5%. For the relation extraction task, when taking homologous genes into account, the best F-score was 37.73%, the best precision was 46.5% and the best recall was 54.1%. Submitted systems explored a wide range of methods, from traditional rule-based, statistical and machine learning systems to state-of-the-art deep learning methods. Given the level of participation and the individual team results we find the precision medicine track to be successful in engaging the text-mining research community. In the meantime, the track produced a manually annotated corpus of 5509 PubMed documents developed by BioGRID curators and relevant for precision medicine. The data set is freely available to the community, and the specific interactions have been integrated into the BioGRID data set. In addition, this challenge provided the first results of automatically identifying PubMed articles that describe PPI affected by mutations, as well as extracting the affected relations from those articles. Still, much progress is needed for computer-assisted precision medicine text mining to become mainstream. Future work should focus on addressing the remaining technical challenges and incorporating the practical benefits of text-mining tools into real-world precision medicine information-related curation.
Extracted statement — data source: PubMed
Research field: Natural Language Processing
Title: Overview of the NLM-Chem BioCreative VII track: Full-text Chemical Identification and Indexing in PubMed articles
Abstract: The BioCreative NLM-Chem track calls for a community effort to fine-tune automated recognition of chemical names in biomedical literature. Chemical names are one of the most searched biomedical entities in PubMed and – as highlighted during the COVID-19 pandemic – their identification may significantly advance research in multiple biomedical subfields. While previous community challenges focused on identifying chemical names mentioned in titles and abstracts, the full text contains valuable additional detail. We organized the BioCreative NLM-Chem track to call for a community effort to address automated chemical entity recognition in full-text articles. The track consisted of two tasks: 1) Chemical Identification task, and 2) Chemical Indexing prediction task. For the Chemical Identification task, participants were expected to predict with high accuracy all chemicals mentioned in recently published full-text articles, both span (i.e., named entity recognition) and normalization (i.e., entity linking) using MeSH. For the Chemical Indexing task, participants identified which chemicals should be indexed as topics for the article's topic terms in the NLM article and indexing, i.e., appear in the listing of MeSH terms for the document. This manuscript summarizes the BioCreative NLM-Chem track. We received a total of 88 submissions in total from 17 teams worldwide. The highest performance achieved for the Chemical Identification task was 0.8672 f-score (0.8759 precision, 0.8587 recall) for strict NER performance and 0.8136 f-score (0.8621 precision, 0.7702 recall) for strict normalization performance. The highest performance achieved for the Chemical Indexing task was 0.4825 f-score (0.4397 precision, 0.5344 recall). The NLM-Chem track dataset and other challenge materials are publicly available at https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/. This community challenge demonstrated 1) the current substantial achievements in deep learning technologies can be utilized to further improve automated prediction accuracy, and 2) the Chemical Indexing task is substantially more challenging. We look forward to further development of biomedical text mining methods to respond to the rapid growth of biomedical literature. Keywords— biomedical text mining; natural language processing; artificial intelligence; machine learning; deep learning; text mining; chemical entity recognition; chemical indexing
Extracted statement — data source: PubMed
Research field: Natural Language Processing
Title: BioCreAtIvE Task 1A: gene mention finding evaluation
Abstract: Abstract Background The biological research literature is a major repository of knowledge. As the amount of literature increases, it will get harder to find the information of interest on a particular topic. There has been an increasing amount of work on text mining this literature, but comparing this work is hard because of a lack of standards for making comparisons. To address this, we worked with colleagues at the Protein Design Group, CNB-CSIC, Madrid to develop BioCreAtIvE (Critical Assessment for Information Extraction in Biology), an open common evaluation of systems on a number of biological text mining tasks. We report here on task 1A, which deals with finding mentions of genes and related entities in text. "Finding mentions" is a basic task, which can be used as a building block for other text mining tasks. The task makes use of data and evaluation software provided by the (US) National Center for Biotechnology Information (NCBI). Results 15 teams took part in task 1A. A number of teams achieved scores over 80% F-measure (balanced precision and recall). The teams that tried to use their task 1A systems to help on other BioCreAtIvE tasks reported mixed results. Conclusion The 80% plus F-measure results are good, but still somewhat lag the best scores achieved in some other domains such as newswire, due in part to the complexity and length of gene names, compared to person or organization names in newswire.
Extracted statement — Evaluation metrics: Recall
Research field: Natural Language Processing
Title: Overview of the CHEMDNER patents task
Abstract: A considerable effort has been made to extract biological and chemical entities, as well as their relationships, from the scientific literature, either manually through traditional literature curation or by using information extraction and text mining technologies. Medicinal chemistry patents contain a wealth of information, for instance to uncover potential biomarkers that might play a role in cancer treatment and prognosis. However, current biomedical annotation databases do not cover such information, partly due to limitations of publicly available biomedical patent mining software. As part of the BioCreative V CHEMDNER patents track, we present the results of the first named entity recognition (NER) assignment carried out to detect mentions of chemical compounds and genes/proteins in running patent text. More specifically, this task aimed to evaluate the performance of automatic name recognition strategies capable of isolating chemical names and gene and gene product mentions from surrounding text within patent titles and abstracts. A total of 22 unique teams submitted results for at least one of the three CHEMDNER subtasks. The first subtask, called the CEMP (chemical entity mention in patents) task, focused on the detection of chemical named entity mentions in patents, requesting teams to return the start and end indices corresponding to all the chemical entities found in a given record. A total of 21 teams submitted 93 runs, for this subtask. The top performing team reached an f-measure of 0.89 with a precision of 0.87 and a recall of 0.91. The CPD (chemical passage detection) task required the classification of patent titles and abstracts whether they do or do not contain chemical compound mentions. Nine teams returned predictions for this task (40 runs). The top run in terms of Matthew’s correlation coefficient (MCC) had a score of 0.88, the highest sensitivity ? Corresponding author
Extracted statement — Evaluation metrics: Recall
Research field: Natural Language Processing
Title: Overview of the BioCreative VI Precision Medicine Track: mining protein interactions and mutations for precision medicine
Abstract: Abstract The Precision Medicine Initiative is a multicenter effort aiming at formulating personalized treatments leveraging on individual patient data (clinical, genome sequence and functional genomic data) together with the information in large knowledge bases (KBs) that integrate genome annotation, disease association studies, electronic health records and other data types. The biomedical literature provides a rich foundation for populating these KBs, reporting genetic and molecular interactions that provide the scaffold for the cellular regulatory systems and detailing the influence of genetic variants in these interactions. The goal of BioCreative VI Precision Medicine Track was to extract this particular type of information and was organized in two tasks: (i) document triage task, focused on identifying scientific literature containing experimentally verified protein–protein interactions (PPIs) affected by genetic mutations and (ii) relation extraction task, focused on extracting the affected interactions (protein pairs). To assist system developers and task participants, a large-scale corpus of PubMed documents was manually annotated for this task. Ten teams worldwide contributed 22 distinct text-mining models for the document triage task, and six teams worldwide contributed 14 different text-mining systems for the relation extraction task. When comparing the text-mining system predictions with human annotations, for the triage task, the best F-score was 69.06%, the best precision was 62.89%, the best recall was 98.0% and the best average precision was 72.5%. For the relation extraction task, when taking homologous genes into account, the best F-score was 37.73%, the best precision was 46.5% and the best recall was 54.1%. Submitted systems explored a wide range of methods, from traditional rule-based, statistical and machine learning systems to state-of-the-art deep learning methods. Given the level of participation and the individual team results we find the precision medicine track to be successful in engaging the text-mining research community. In the meantime, the track produced a manually annotated corpus of 5509 PubMed documents developed by BioGRID curators and relevant for precision medicine. The data set is freely available to the community, and the specific interactions have been integrated into the BioGRID data set. In addition, this challenge provided the first results of automatically identifying PubMed articles that describe PPI affected by mutations, as well as extracting the affected relations from those articles. Still, much progress is needed for computer-assisted precision medicine text mining to become mainstream. Future work should focus on addressing the remaining technical challenges and incorporating the practical benefits of text-mining tools into real-world precision medicine information-related curation.
Extracted statement — Evaluation metrics: Recall
Research field: Natural Language Processing
Title: Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction
Abstract: While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research activity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an automatic system able to summarize scientific results, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.
Extracted statement — Concept types: Score
Research field: Networking and Internet Architecture
Title: A Stochastic Approach of Dependency Evaluation for IoT Devices
Abstract: Internet of things (IoT) is an emerging technique that offers advanced connectivity of devices, systems, services, and human beings. With the rapid development of hardware and network technologies, the IoT can refer to a wide variety and large number of devices, resulting in complex relationships among IoT devices. The dependencies among IoT devices, which reflect their relationships, are with reference value for the design, development and management of IoT devices. This paper proposes a stochastic model based approach for evaluating the dependencies of IoT devices. A random walk model is proposed to describe the relationships of IoT devices, and its corresponding Markov chain is obtained for dependency analysis. A framework as well as schemes and algorithms for dependency evaluation in real-world IoT are designed based on traffic measurement. Simulation experiments based on real-life data extracted from smart home environments are conducted to illustrate the efficacy of the approach.
Extracted statement — Type of considered dependencies: Service
Research field: Numerical Analysis/Scientific Computing
Title: Out of one, many: Exploiting intrinsic motions to explore protein structure spaces
Abstract: Reconstructing the energy landscape of a protein holds the key to characterizing its structural dynamics and function [1]. While the disparate spatio-temporal scales spanned by the slow dynamics challenge reconstruction in wet and dry laboratories, computational efforts have had recent success on proteins where a wealth of experimentally-known structures can be exploited to extract modes of motion. In [2], the authors propose the SoPriM method that extracts principle components (PCs) and utilizes them as variables of the structure space of interest. Stochastic optimization is employed to sample the structure space and its associated energy landscape in the defined varible space. We refer to this algorithm as SoPriM-PCA and compare it here to SoPriM-NMA, which investigates whether the landscape can be reconstructed with knowledge of modes of motion (normal modes) extracted from one single known structure. Some representative results are shown in Figure 1, where structures obtained by SoPriM-PCA and those obtained by SoPriM-NMA for the H-Ras enzyme are compared via color-coded projections onto the top two variables utilized by each algorithm. The results show that precious information can be obtained on the energy landscape even when one structural model is available. The presented work opens up interesting venues of research on structure-based inference of dynamics. Acknowledgment: This work is supported in part by NSF Grant No. 1421001 to AS and NSF Grant No. 1440581 to AS and EP. Computations were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University, VA (URL: http://orc.gmu.edu).
Extracted statement — Has method: SoPriM-NMA
Research field: Numerical Analysis/Scientific Computing
Title: Out of one, many: Exploiting intrinsic motions to explore protein structure spaces
Abstract: Reconstructing the energy landscape of a protein holds the key to characterizing its structural dynamics and function [1]. While the disparate spatio-temporal scales spanned by the slow dynamics challenge reconstruction in wet and dry laboratories, computational efforts have had recent success on proteins where a wealth of experimentally-known structures can be exploited to extract modes of motion. In [2], the authors propose the SoPriM method that extracts principle components (PCs) and utilizes them as variables of the structure space of interest. Stochastic optimization is employed to sample the structure space and its associated energy landscape in the defined varible space. We refer to this algorithm as SoPriM-PCA and compare it here to SoPriM-NMA, which investigates whether the landscape can be reconstructed with knowledge of modes of motion (normal modes) extracted from one single known structure. Some representative results are shown in Figure 1, where structures obtained by SoPriM-PCA and those obtained by SoPriM-NMA for the H-Ras enzyme are compared via color-coded projections onto the top two variables utilized by each algorithm. The results show that precious information can be obtained on the energy landscape even when one structural model is available. The presented work opens up interesting venues of research on structure-based inference of dynamics. Acknowledgment: This work is supported in part by NSF Grant No. 1421001 to AS and NSF Grant No. 1440581 to AS and EP. Computations were run on ARGO, a research computing cluster provided by the Office of Research Computing at George Mason University, VA (URL: http://orc.gmu.edu).
Extracted statement — Has method: SoPriM-PCA
Research field: Nutritional Epidemiology
Title: Dietary patterns and cardiometabolic risk factors among adolescents: systematic review and meta-analysis
Abstract: <jats:title>Abstract</jats:title><jats:p>This study systematised and synthesised the results of observational studies that were aimed at supporting the association between dietary patterns and cardiometabolic risk (CMR) factors among adolescents. Relevant scientific articles were searched in PUBMED, EMBASE, SCIENCE DIRECT, LILACS, WEB OF SCIENCE and SCOPUS. Observational studies that included the measurement of any CMR factor in healthy adolescents and dietary patterns were included. The search strategy retained nineteen articles for qualitative analysis. Among retained articles, the effects of dietary pattern on the means of BMI (<jats:italic>n</jats:italic> 18), waist circumference (WC) (<jats:italic>n</jats:italic> 9), systolic blood pressure (<jats:italic>n</jats:italic> 7), diastolic blood pressure (<jats:italic>n</jats:italic> 6), blood glucose (<jats:italic>n</jats:italic> 5) and lipid profile (<jats:italic>n</jats:italic> 5) were examined. Systematised evidence showed that an unhealthy dietary pattern appears to be associated with poor mean values of CMR factors among adolescents. However, evidence of a protective effect of healthier dietary patterns in this group remains unclear. Considering the number of studies with available information, a meta-analysis of anthropometric measures showed that dietary patterns characterised by the highest intake of unhealthy foods resulted in a higher mean BMI (0·57 kg/m²; 95 % CI 0·51, 0·63) and WC (0·57 cm; 95 % CI 0·47, 0·67) compared with low intake of unhealthy foods. Controversially, patterns characterised by a low intake of healthy foods were associated with a lower mean BMI (−0·41 kg/m²; 95 % CI −0·46,−0·36) and WC (−0·43 cm; 95 % CI −0·52,−0·33). An unhealthy dietary pattern may influence markers of CMR among adolescents, but considering the small number and limitations of the studies included, further studies are warranted to strengthen the evidence of this relation.</jats:p>
Extracted statement — Age group: Adolescents