dataesr/scientific-paragraphs-categorization huggingface dataset:
Phase 1: Iterative Improvement of the Adverse Drug Reaction Information Extraction Module Phase 1 consisted of an iterative process of validations and improvements of the tool. The review on the corpus was constituted by manual annotation of the messages by 2 experts in pharmacovigilance. The manual annotation process used a tool developed by Expert System for in-house testing purposes  [20]. This tool did not integrate functionalities for blind manual annotation. No gold standard was established at this step. All along the process, both annotators could see the drugs, AEs, and ADRs extracted by the tool, and their task consisted of validating or invalidating these extractions and, if needed, complementing the annotations performed by the system. The annotators were asked to (1) check all drugs and all AEs in the post, irrespective of whether they have been involved in an ADR relationship, and (2) identify all drug-AE relationships in the same sentence.
Evaluation Dataset For phase 2, the selection of datasets is given in Figure  3. On October 28, 2016, Kappa Santé Detec't database contained about 23 millions of posts, and approximately 21 million posts were published after January 1, 2007. Messages about drugs, for which marketing authorization has been withdrawn before the cut-off date, were considered as some AEs might appear at long term or patients might discuss about old drugs and their safety. The software developed by Kappa Santé for purpose of Web discussions collection offered the possibility of preidentifying the posts containing at least one pharmaceutical or molecule name. We considered the dataset of all posts containing at least one preidentified compound or brand name and combined it with a complementary set of posts randomly sampled from the remainder of the Kappa Santé Detec't database.
Adverse Drug Reactions From Patient Reports in Social Media Manual Annotation Platform We developed a Web application dedicated to manual annotations valuable as gold standard, for the project purposes in phase 2. This application was based on Java Servlets and JavaScript libraries and connected in Java Database Connectivity to the dataset of posts selected for the gold standard annotation. We used a self-completion mechanism to attach portions of message to Racine Pharma and MedDRA terms. We used drag-and-drop operations to fill in a table containing the manual review of the co-occurrences, where each line was dedicated to 1 co-occurrence. For each co-occurrence, the drug and the AE were dragged-and-dropped in the first and the second column, respectively. In the third column, the manual annotator was given a drop-down menu that presented 3 possibilities for defining the co-occurrence: explicit positive ADR, explicit negative ADR, or no ADR.
Sample Size Calculation Recall can be estimated from a random sample of ADR concepts annotated by the gold standard, whereas precision can be estimated from a random sample of ADR concepts extracted by the ADR information extraction module. However, creating the gold standard on the whole evaluation dataset before sampling in it for the recall was an overwhelming task. At the same time, a vast majority of messages did not contain any information on the drug concept. Thus, we expected a very low proportion of ADR concepts identified by the ADR information extraction module in the evaluation dataset. Therefore, the recall estimated from a random sample of ADR concepts annotated by the gold standard was mathematically approximated by the recall estimated on ADR concepts sampled for the precision.
Results At the time of publishing this evaluation protocol, several steps of this project would have been completed. The corpus for evaluation has been constituted, and the NLP tools have identified and extracted the information about ADR inside this corpus. We selected the ADR concepts and constituted the samples of the entities necessary to set up the gold standard. The 2 annotators completed the annotations process. Data analyses for assessing the interannotator agreement and the performance of the ADR information extraction module as compared with the gold standard are ongoing, and the study results are expected internally before the end of 2018.
Regarding the gold standard, 2 experts in medical terminologies trained in drug safety performed manual annotations. However, contrary to the phase 1 manual review, none of them could be considered as a pharmacovigilance expert. The medical informatics community needs shared open corpora to evaluate their methods and tools. Recent efforts have led to making several datasets more accessible and the evaluation of the methods more standardized, for example, the Multiparameter Intelligent Monitoring in Intensive Care corpus  [47]  and the Informatics for Integrating Biology and the Bedside challenges  [48]. Drug safety could benefit from open-access validation datasets made available for research purposes  [25, 35]. However, several obstacles remain. For example, the ADR-PRISM consortium got approval to reuse posts for research purposes, but the approval is restricted to the project.
). The analysis results are publicly available as online interactive analytics graphs. The review clearly depicted increased use of KE approaches for DS. The collected data illustrate the use of KE for various DS aspects, such as Adverse Drug Event (ADE) information collection, detection, and assessment. Moreover, the quantified analysis of using KE for the respective DS core activities highlighted room for intensifying research on KE for ADE monitoring, prevention and reporting. Finally, the assessed use of the various data sources for DS special topics demonstrated extensive use of dominant data sources for DS surveillance, i. e. , Spontaneous Reporting Systems, but also increasing interest in the use of emerging data sources, e. g. , observational healthcare databases, biochemical/genetic databases, and social media.
g. , using disproportionality-based statistical methods  (Montastruc et al. , 2011). The current era of "data explosion" affects the entire spectrum of health, including DS. While traditionally postmarketing DS surveillance relied on SRSs as well as clinical studies and the scientific literature, advances in Information and Communication Technologies (ICT) recently enabled the exploitation of new/emerging data sources, such as observational healthcare databases, biochemical and genetic databases, social media, internet search logs, etc. To this end, various computational analysis methods have been proposed for 1 Due to the numerous abbreviations used in the article, we provide as Appendix an abbreviation index to facilitate the reader. 2 ADEs: Side-effects that may or may not have causal relationship with the drug, including the events caused by drug misuse (e.
Thus, the main research question of the current study is: "What are the main KE methods applied in the DS domain, upon which knowledge models and data sources are they applied, what is their contribution/added 7 MedDRA R (the Medical Dictionary for Regulatory Activities) is a reference, international terminology in the domain of DS, developed under the auspices of the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH). The MedDRA R trademark is owned by IFPMA on behalf of ICH. 8 A mapping review aims "to map out and categorize existing literature on a particular topic, identifying gaps in research literature from which to commission further reviews and/or primary research"  (Grant and Booth, 2009). value for DS, and what are the potential gaps, challenges and opportunities for further research?".
METHODS A systematic search was performed by querying two reference bibliographic repositories: PubMed 9  and Web of Science 10. The study comprised of the article retrieval step and two consecutive review stages (Figure  1 ); the first aimed to filter irrelevant articles with the domains of KE and DS based on their title and abstract, and the second was devoted to evaluating the remaining papers' full-text in detail, and map them based on specific analysis criteria. The review was conducted by the authors of the paper. In the retrieval stage, we defined and executed two queries (provided as Supplementary Material) and imported the obtained citations in BibReview 11  , a tool that was used throughout our study enabling collaborative review of bibliographic data  (Lamy et al. , 2015). The queries included two core parts (linked with the logical operator AND), each comprising of synonym terms describing the domains of interest, i.
most important data source categories, and the KE core activities based on the reviewed articles. Interestingly, "signal detection, " "MoA analysis and identification, " and "Drug interactions" are the three most elaborated DS special topics, exploiting a number of heterogeneous data sources, e. g. , SRSs, ADE databases, etc. One of the key KE foundations is the reuse of established/reference knowledge structures (i. e. , ontologies, standard terminologies, etc. ). This facilitates semantic interoperability between different systems and widens the spectrum upon which KE approaches are applicable to. Figure  7  presents the most widely adopted terminologies/ontologies in the reviewed articles 18. The Unified Medical Language System (UMLS), MedDRA, the Anatomical Therapeutic Chemical (ATC) Classification System, the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT), and the International Classification of Diseases (ICD) are the most widely used terminologies, while the Ontology for Adverse Events (OAE), the Vaccine Ontology (VO), and GO are the most widely referred ontologies.
The types of data sources employed in the reviewed articles vary significantly, highlighting the complexity of the domain and the need for advanced data integration and representation schemes based on KE  (Koutkias and Jaulent, 2015). Figure  8A  presents the distribution of data source 18 Not all referenced terminologies/ontologies are presented for readability purposes. The full list of referenced terminologies/ontologies can be found in the detailed analysis provided as Supplementary Material in the form of a spreadsheet. categories, while Figure  8B  presents the most popular data sources 19 , as employed in the reviewed articles. SRSs [e. g. , the FDA Adverse Event Reporting System (FAERS) and the Vaccine 19 Not all data sources identified are presented for readability purposes. The full list of data sources identified can be found in the fully detailed analysis provided as Supplementary Material in the form of a spreadsheet.
Adverse Event Reporting System (VAERS)], drug information databases (e. g. , DrugBank), ADE databases [mainly the Side Effect Resource (SIDER)], genetic and biochemical information data sources [e. g. , GO and the Kyoto Encyclopedia of Genes and Genomes-GenomeNet (KEGG)], as well as scientific literature repositories (i. e. , PubMed/MEDLINE) are the most prominent ones. The selected articles were also critically reviewed to identify challenges or weaknesses and, consequently, gaps in the applied research practices. As shown in Figure  9 , in many of the reviewed articles the research significantly depended on manual work (e. g. , data curation, annotation, etc. ), conducted by a small group of experts. Furthermore, despite elaborating on KE representation schemes like ontologies, many studies did not evaluate the proposed models regarding quality, e.
Data sources Table  2  presents the usage of data sources for specific DS applications, citing also the respective articles. We organize data sources in two main types: (a) those established or dominant in the domain of DS, such as SRSs, clinical trial databases, and bibliographic databases, and (b) emerging or quite new, such as observational healthcare databases, biochemical/genetic information databases, and social media platforms. Established data sources SRSs constitute the dominant data source for DS. They have been widely used in the reviewed articles for signal identification (mostly through NLP) as well as monitoring and validation. Interestingly, in order to improve the mining capacity of FAERS for signal detection and promote semantic interoperability between FAERS and other data sources, NLP techniques and normalization procedures were applied to FAERS data using reference terminologies, i.
e. , MedDRA, RxNorm, and the National Drug File-Reference Terminology (NDF-RT)  (Wang et al. , 2014). ADE databases [mostly SIDER, the Comparative Toxicogenomics Database (CTD), and MetaADEDB], clinical trials data (from ClinicalTrials. gov), drug information databases (e. g. , DrugBank) and bibliographic databases [i. e. , PubMed/MEDLINE and the Semantic MEDLINE Database (SemMedDB)  (Kilicoglu et al. , 2012) , a database of semantic relationships extracted from MEDLINE] have been employed for signal detection and MoA investigation. Emerging data sources Observational healthcare databases and Electronic Health Records (EHRs) in particular, gained a major interest recently for DS research. In the scope of KE for DS, structured EHRs were used for signal detection, combining the use of ontologies and NLP approaches, as well as for developing medication-related Clinical Decision Support Systems (CDSSs).
Unstructured EHR data, i. e. , free-text clinical notes, were also used for ADR identification. Recent advances in high-throughput sequencing technologies enable the integration of biological information to support SP by focusing on gene-drug-disease interaction networks. An increasing number of these frameworks incorporate genetic data (most often genomic polymorphisms as described in PharmGKB) for drug-drug interactions (DDIs) and ADR in silico prediction, stressing the need to integrate such data to complement in vivo and in vitro investigations on pharmacogenomics. Information on pathways (e. g. , from KEGG), proteins (e. g. , from UniProt) and their annotations with GO were the most prominent data sources for ADE identification and the analysis of the respective MoA. Interestingly, the use of biomolecular functional network data improved ADR predictions  (Huang et al.
Category Application in drug safety Established data sources SRS • Signal detection  (Sarntivijai et al. , 2012; Tao et al. , 2012; Cheng et al. , 2013; Boyce et al. , 2014; Cheng and Zhao, 2014; Courtot et al. , 2014; Iyer et al. , 2014; Wang et al. , 2014; Cai et al. , 2015 Cai et al. , , 2017;; Dupuch and Grabar, 2015; Liu and Chen, 2015; Koutkias and Jaulent, 2016; Liu et al. , 2016 Liu et al. , , 2018   (Henegar et al. , 2006; Gottlieb et al. , 2012; Courtot et al. , 2014; Iyer et al. , 2014)  • Monitoring  (Marcos et al. , 2013)  ADE databases • Signal detection  (Huang et al. , 2011; Gurulingappa et al. , 2012; Cheng et al. , 2013; Cheng and Zhao, 2014; Iyer et al. , 2014; Shang et al. , 2014; Cai et al. , 2015; Herrero-Zazo et al. , 2015; Jiang et al. , 2015; Koutkias and Jaulent, 2015, 2016; Bravo et al.
, 2016; Eshleman and Singh, 2016; Kawazoe et al. , 2016; Lowe et al. , 2016; Noor et al. , 2016; Abdelaziz et al. , 2017 ; Knowledge Base Workgroup of the Observational Health Data Sciences and Informatics (OHDSI) Collaborative, 2017; Nguyen et al. , 2017) • MoA identification/analysis  (Huang et al. , 2011; Gottlieb et al. , 2012; Xu and Wang, 2013; Cai et al. , 2015; Herrero-Zazo et al. , 2015; Guo et al. , 2016; Noor et al. , 2016; Abdelaziz et al. , 2017; Personeni et al. , 2017; Piñero et al. , 2017)  • Validation  (Gurulingappa et al. , 2012)  Drug information databases • Signal detection  (Tari et al. , 2010; Huang et al. , 2011; Boyce et al. , 2014; Cheng and Zhao, 2014; Iyer et al. , 2014; Cai et al. , 2015; Herrero-Zazo et al. , 2015; Koutkias and Jaulent, 2015, 2016; Noor et al.
, 2016; Zhang et al. , 2016; Abdelaziz et al. , 2017 ; Knowledge Base Workgroup of the Observational Health Data Sciences and Informatics (OHDSI) Collaborative, 2017) • MoA identification/analysis  (Lin et al. , 2010; Tari et al. , 2010; Huang et al. , 2011; Gottlieb et al. , 2012; Cai et al. , 2015; Herrero-Zazo et al. , 2015; Noor et al. , 2016; Zhang et al. , 2016; Abdelaziz et al. , 2017)  Bibliographic databases • Signal detection  (Tari et al. , 2010; Gurulingappa et al. , 2012; Boyce et al. , 2014; Shang et al. , 2014; Zhang et al. , 2014 Zhang et al. , , 2016;; Bravo et al. , 2016; Koutkias and Jaulent, 2016; Lowe et al. , 2016; Noor et al. , 2016 ; Knowledge Base Workgroup of the Observational Health Data Sciences and Informatics (OHDSI) Collaborative, 2017; Voss et al. , 2017) • MoA identification/analysis  (Tari et al.
, 2015 Henriksson et al. , , 2016;; Sarker and Gonzalez, 2015; Iqbal et al. , 2017)  Biochemical and genetic information databases • Signal detection  (Arikuma et al. , 2008; Tari et al. , 2010; Boyce et al. , 2014; Cai et al. , 2015; Kawazoe et al. , 2016; Noor et al. , 2016; Abdelaziz et al. , 2017 ; Knowledge Base Workgroup of the Observational Health Data Sciences and Informatics (OHDSI) Collaborative, 2017) • MoA identification/analysis  (Arikuma et al. , 2008; Tari et al. , 2010; Gottlieb et al. , 2012; Hur et al. , 2012; Cai et al. , 2015; Noor et al. , 2016; Abdelaziz et al. , 2017; Piñero et al. , 2017).
OntoADR • Combined with OAE to investigate MoA of Tyrosine Kinase Inhibitors  (Sarntivijai et al. , 2016)  • Secondary use of EHRs and observational studies data (e. g. , signal detection and automatic report generation)  (Declerck et al. , 2015; Yuksel et al. , 2016)  • Searching, coding, and information retrieval of ADE information  (Bousquet et al. , 2014; Souvignet et al. , 2016)  (WHO) Adverse Reaction Terminology (WHO-ART) were used to semantically categorize and interrelate (e. g. , group) concepts regarding signals or ADE reports and also as common reference terminologies for integration purposes or NLP tasks. In USoriginated studies, RxNorm was used as a reference terminology for drugs, but to a smaller extent compared to ATC overall. An interesting application of SNOMED-CT was for enhancing the semantics provided by WHO-ART  (Alecu et al.
Knowledge elicitation Knowledge elicitation activities are typically related with rulebased inferencing combined with ontological reasoning methods. For instance, a conceptual model relying on the Drug Interaction Ontology (DIO) to identify DDIs was developed based on two rule-based inferencing modules (Pathway object constructor and Drug interaction detector)  (Arikuma et al. , 2008). Drug-Drug Interactions Ontology (DINTO) combined Description Logic (DL)  (Baader et al. , 2004)  based reasoning with rules formed in the Semantic Web Rules Language (SWRL) to identify DDIs and investigate their MoA  (Herrero-Zazo et al. , 2015) , upon a conceptual model exploiting Pharmacokinetics and Pharmacodynamics related knowledge. The Drug Enzyme Interaction (DEI) ontology was combined with a rule-base to investigate drug MoAs  (Zhang et al.
, 2016). Similarly, ProLog was used to encode rules regarding drug metabolism and conduct reasoning to identify potential DDIs  (Tari et al. , 2010). In addition, SPARQL queries following specific patterns regarding temporal inference were used to identify ADRs upon HL7 messages integrated in one large Resource Description Framework (RDF) graph  (Kawazoe et al. , 2016). Rules referring to four levels of interaction mechanisms, namely, pharmacokinetic, pharmacodynamic, pharmacogenetic, and multi-pathway interaction, were employed to identify DDIs and their underlying MoAs upon a large RDF knowledge graph integrating 15 DDI databases  (Noor et al. , 2016). Inferencing methods based on graph theory were also extensively applied. Graph clustering coefficient analysis was used to identify similar ADE clusters  (Lin et al.
Knowledge extraction The most widely used knowledge extraction approach refers to the use of NLP techniques applied on unstructured data, i. e. , free-text, originated from biomedical literature, social media, clinical notes, etc. , using various computational methods (including ML-based). Relying on core NLP methods, the DrugNerAR system demonstrated its ability to identify drug mentions in biomedical literature for DDI identification  (Segura-Bedmar et al. , 2010) , and drug-gene relationships, extracted from MEDLINE  (Xu and Wang, 2013). NLP was also used upon bibliographic data sources, storing a structured representation of plain text in a "parse tree database" for further elaboration and reasoning to identify DDIs  (Tari et al. , 2010). Notably, an alternative approach targeting social media took into account the three previous and the three next tokens to analyse each token in its context for identifying ADR mentions  (Nikfarjam et al.
, 2015). Context-based semantic analysis across sentences improved the identification of ADRs in patient forums, using the NegEx tool and drug indications to filter out negated ADEs and drug indications, respectively  (Liu and Chen, 2015). NLP was also applied on clinical notes to identify DDIs based on drug-gene relationships extracted from SemMedDB  (Zhang et al. , 2014) , while SemMedDB was also exploited for Literature Based Discovery aiming at signal assessment  (Shang et al. , 2014). Similarly, NLP was applied on the clinical notes of a large dataset, taking into account contextual information (i. e. , temporal information and categorization in factual, hypothetical or negated sentence), to detect ADEs specific to antipsychotics and antidepressants  (Iqbal et al. , 2017). NLP was also applied on WikiPedia to identify drugs and conditions in the title of its articles, as well as links to other pages related to drugs, conditions and ADRs, aiming to construct a lexicon of ADR terms  (Lowe et al.
Knowledge integration WHO-ART and SNOMED-CT were mapped based on synonymy in the UMLS Metathesaurus to automatically generate definitions of WHO-ART terms in a DL formalism, i. e. , the Web Ontology Language (OWL), aiming to identify WHO-ART terms that may be grouped together  (Alecu et al. , 2008). As the same medical condition may be coded with different terms in DS databases, it was assumed that such approach would enable to group similar terms and improve signal generation. As a next step in the same line of work, SNOMED-CT was used to convert MedDRA to an OWL ontology, namely, OntoADR, which combined the semantics of MedDRA and SNOMED-CT  (Bousquet et al. , 2014) , through a relational database implementation  (Souvignet et al. , 2016). Koutkias and Jaulent investigated the limitations of computational signal detection methods when applied on single data sources, and elaborated on multiple heterogeneous signal detection methods, data sources and other drug-related resources under a common, integrated framework  (Koutkias and Jaulent, 2015).
The framework relied on the Pharmacovigilance Signal Detection Ontology (PV-SDO) and a multiagent system, implementing a comprehensive workflow comprising of method selection and execution, as well as outcomes' aggregation, filtering, ranking and annotation  (Koutkias and Jaulent, 2016). Declerck et al. proposed an ontology-based abstraction layer called Common Information Model-CIM  (Declerck et al. , 2015). CIM was populated through software "bridges" based on mappings of local EHR databases to CIM, thus accommodating the dependencies of the overall framework on the local EHR data schemas. Furthermore, various data sources (SPLs, ADE information, clinical trials data, etc. ) were integrated in a single knowledge graph based on common-terms matching and mappings to reference terminologies, in order to provide a unified and semantically enhanced knowledge base for information regarding drug products  (Boyce et al.