dataesr/scientific-paragraphs-categorization huggingface dataset:
Toponym Resolution Metrics When a gold standard corpus and a toponym resolver are aligned on the same geographical database, here the database GeoNames, the standard metrics of precision, recall and F-measure can be used to measure the performance of the resolver. For this challenge, we report all results by using two common variations of these metrics: strict and overlapping measures. In the strict measure, resolver annotations are considered matching with the gold standard annotations if they hit the same spans of text; whereas in overlapping measure, both annotations match when they share a common span of text. We computed the P and R for toponym detection with the standard equations: P recision = T P/(T P + F P ) and Recall = T P/(T P + F N ), where TP (True Positive) is the number of toponyms correctly identified by a toponym detector in the corpus, FP (False Positive) the number of phrases incorrectly identified as toponyms by the detector, and FN (False Negative) the number of toponyms not identified by the detector.
Baseline System We released an end-to-end system to be used as a strong baseline. This system performs sequentially the detection and the disambiguation of the toponyms in raw texts. To detect the toponyms the system uses a feedforward neural network described in  (Magge et al. , 2018). The disambiguation of all toponyms detected is then performed using a common heuristic, the population heuristic. Using this heuristic, the system always disambiguates a toponym by choosing the place which has the highest population in GeoNames. The baseline system can be downloaded from the Codalab website of the competition. We also made available to the participants a Rest service to search a recent copy of GeoNames, the documentation and the code to deploy the service locally can be found on the Codalab website.
system which, according to their ablation study  (Wang et al. , 2019) , proved to be effective 9. Note that the performance of the first system is close to our IAA for toponym detection. Toponym Disambiguation: All systems relied on handcrafted features to disambiguate toponyms. Their features described the lexical context of the toponyms and their importance. The importance of the toponyms was estimated by the frequencies of the candidates in the training data or by their populations. While the two top ranked systems combined such features with machine learning, SVM for UniMelb and a gradient boosting algorithm for DM NLP, others just encoded them into hard rules leading to suboptimal disambiguation.
Conclusion In this paper we presented an overview of the results of SemEval 2019 Task 12 which focuses on toponym resolution in scientific articles. Given an article from PubMed, the task consists of detecting all mentions of place names, or toponyms, in the article and mapping them to their corresponding entry in GeoNames, a database of geospatial locations. All systems resolved the toponyms in our corpus sequentially, detecting the toponyms before disambiguating them. Among the 21 systems submitted for toponym detection, neural network based approaches were the most popular and the most efficient to detect toponyms with scores approaching the Inter-Annotator agreement. One key to success for the top ranked systems was to design two different algorithms to detect toponyms in the body and in the tables of the articles.
3 reference sequence for the PRPS1 gene. Databases, such as ExAC Genome browser (http://exac. broad insti tute. org), dbSNP135 (National Center for Biotechnology Information [NCBI], Bethesda, Maryland, USA, http://www. ncbi. nlm. nih. gov/proje cts/SNP/), Clin  Var (www. ncbi. nlm. nih. gov/clinvar), and HGMD professional (www. hgmd. cf. ac. uk) , were also screened. Pathogenic variants of interest were verified by Sanger sequencing using forward and reverse primer pairs. The PRPS1 variant was submitted to the corresponding LOVD database at http://datab ases. lovd. nl/ share d/varia nts/00004 38212 (patient ID00207340). MLPA and Sanger sequencing of GJB2 and GJB6 were performed to screen for hearing loss.
| Genetic testing NGS revealed the detection of the hemizygous variant on chromosomic position X:106,882,604, c. 202A>T, p. (Met68Leu) (Figure  1 ). This missense variant was not found in ExAC, dbSNP, Clin Var, and HGMD databases. Neither had it been reported in our French cohort of 3,412 patients suffering from inherited peripheral neuropathies (personal data). Familial segregation was also in accordance with an asymptomatic carrier mother. Therefore, this variant is located in exon 2 corresponding to the N-terminal domain of the protein. This variant is located in a crucial region, the flag region of the dimer interface (Figure  2 ). This change is predicted to lead to an altered affinity between the different subunits in the dimer, thereby may prevent the hexamer formation. This substitution may result in the total absence of the PRPS1 hexamer.
Bisphenol F (BPF) is one of several Bisphenol A (BPA) substituents that is increasingly used in manufacturing industry leading to detectable human exposure. Whereas a large number of studies have been devoted to decipher BPA effects, much less is known about its substituents. To support decision making on BPF's safety, we have developed a new computational approach to rapidly explore the available data on its toxicological effects, combining text mining and integrative systems biology, and aiming at connecting BPF to adverse outcome pathways (AOPs). We first extracted from different databases BPF-protein associations that were expanded to protein complexes using protein-protein interaction datasets. Over-representation analysis of the protein complexes allowed to identify the most relevant biological pathways putatively targeted by BPF.
Then, automatic screening of scientific abstracts from literature using the text mining tool, AOP-helpFinder, combined with data integration from various sources (AOP-wiki, CompTox, etc. ) and manual curation allowed us to link BPF to AOP events. Finally, we combined all the information gathered through those analyses and built a comprehensive complex framework linking BPF to an AOP network including, as adverse outcomes, various types of cancers such as breast and thyroid malignancies. These results which integrate different types of data can support regulatory assessment of the BPA substituent, BPF, and trigger new epidemiological and experimental studies.
, 2018)  with considerable additional value for emerging toxicological knowledge  (Pollesch et al. , 2019). Adverse outcome pathway networks can be initiated by 1 or more stressors that can be environmental chemicals. Computational approaches allow to explore and identify key information, and therefore can be used to develop AOP networks. In their study, Oki and Edwards, generated computationally predicted AOPs by integrating multiple data sources from HTS studies by using Frequent Itemset Mining  (Oki and Edwards, 2016). In this study, we describe a computational approach to establish linkages between environmental stressors and health effects, using available information from the literature and databases. Different sources of information were considered such as PubMed, ToxCast, CompTox, and AOP-wiki, and integrated to develop individual AOP and AOP networks.
MATERIALS AND METHODS Overall strategy. Linkage between BPF (see §2), biological events and toxic effects were investigated using a systems toxicology approach (Figure  1 ). This multistep approach is based on integration of existing knowledge from various sources of information. In the first step, specific data on BPF-protein associations were extracted from chemical biology databases. Second, these BPF-protein associations were expanded to protein complexes. By using a high confidence interactome  (Li et al. , 2017) , we were able to decipher protein complexes associated with BPF ( §3). Then, protein-pathway information was integrated into these protein complexes to statistically order linkage between the BPF and biological pathways ( §4). Finally, to have a better understanding of the mode of action of BPF leading to the identified pathways, literature searches were performed, and relevant information were integrated ( §5 and 6).
Therefore, these steps allowed to suggest a plausible AOP induced by BPF leading to thyroid cancer, that has been extended to an AOP network ( §7). Bisphenol F. To get the most complete biological picture of BPF, the 3 isomers for BPF were analyzed (Figure  2 ): the 2,2 0 -isomer of BPF (CAS rn 2467-02-9) (2,2-BPF), the 2,4 0 -isomer of BPF (CASRN 2467-03-0) (2,4-BPF), and 4,4 0 -isomer of BPF (CAS rn 620-92-8) (4,4-BPF). The commercially available BPF mixed isomers was also taken into consideration (CAS rn 1333-16-0). Information from the different sources were compiled using the common name "BPF," and various synonyms (Supplementary Table  1 ). These synonyms were extracted from the CompTox database, and only the valid ones were retained (19 for 4,4-BPF, 12 for 2,2-BPF, 11 for 2,4-BPF, and 13 for BPF mixture).
Linking BPF to protein complexes. To identify proteins known to be associated with BPF, we compiled information from 2 publicly available databases. First, we extracted information from the ToxCast database (https://actor. epa. gov/dashboard; dashboard accessed on December 2018)  (Judson et al. , 2010). The ToxCast database is based on high-throughput technologies, and aggregate information for thousands of chemical substances. This U. S. Environmental Protection Agency infrastructure contains information for 9076 chemicals that have been tested on 359 assays with a total of 1192 endpoints  (Judson et al. , 2010; Kavlock et al. , 2012). Then to collect more proteins associated with BPF, the Comparative Toxicogenomics database (CTD) (as of December 2018)  (Davis et al. , 2018)  was used. CTD contained 1 898 228 chemical-protein curated associations mined from peer-reviewed scientific literature, between 13 108 chemicals and 47 581 proteins for 593 organisms.
All proteins were mapped to HUGO name identifiers and Entrez gene id to facilitate further data integration. Because in a biological system, proteins tend to function in groups or complexes, an important step was to enrich the list of compiled proteins using information from a high confidence human protein interactome that is protein-protein interactions (PPIs) based on experiments and inferred model organism data  (Li et al. , 2017). The InWeb 3. 0 tool (www. cbs. dtu. dk/services/VirtualPulldown-1. 1b/web/) was used to identify the first-order interacting proteins. Such strategy is based on a neighbor's pull down approach  (Lage et al. , 2007). The version that we used, contained a total of 507 142 unique PPIs involving 14 441 human proteins, as of December 2018. As a result, the list of relevant proteins associated with BPF was extended by inclusion of their first-order PPI partners, considering a significant pull down score threshold of 0.
25  (Lage et al. , 2007). Biological enrichment of the protein complexes. To identify pathways and toxic effects related to BPF, information on biological pathways were integrated into the protein complex linked to BPF. To catch as much as possible information, the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways-based database was used as a source of information  (Kanehisa et al. , 2019). To perform the over-representation analysis (ORA), the R package WebGestaltR (version 3), was used. The protein complex was therefore tested for significant pathways associations using a test based on a hypergeometric distribution. A significance level of 0. 05 after Benjamini-Hochberg correction for multiple testing of p values was used to select the most relevant associations. Exploration of the literature to contribute to the development of AOP.
The literature search was done in 2 ways with the aim to capture as much as possible information to link BPF to AOPs. As AOPs are in development, and limited number has been validated, we decided to combine our approach with a manual exploration of the PubMed database. First, an automatic search was performed using the recently developed AOP-helpFinder tool  (Carvaillo et al. , 2019) , that is a hybrid approach that combined text mining procedure and graph theory, to identify linkage between BPF and biological events present in already defined AOPs. The TOXLINE database was screened to compile scientific publications mentioning BPF in a toxicological context (https://toxnet. nlm. nih. gov/cgi-bin/sis/ htmlgen? TOXLINE) (as of July 2017). To capture as much as possible of the existing information, several synonyms for BPF where used (Supplementary Table  1 ) that were extracted from the CompTox database (https://comptox.
epa. gov/dashboard) (Williams et al. , 2017). Then, to prepare the biological data, AOPs were considered. Based on the AOP-wiki database (as of August 2017) (https://aopwiki. org), a dictionary was developed. The AOPwiki database contains information related to AOPs for which mechanistic representations of toxicological effects over various  level of the biological organization were reported. The developed dictionary contains information related to MIE/KE and AO. Both MIE and KE were integrated in the same "list" that contains 1318 events. The AO list contains 61 events. These 2 categories were analyzed separately using the text mining tool, mainly due to their wording composition. Biological events associated with MIE and KE were more complex sentences compared with AO that are more likely to be defined by 1 or few words.
Finally, the literature screening approach was performed using the AOP-helpFinder tool. The text mining part is used to identify comentioned words (eg, a BPF isomer and a biological event) in an abstract from the scientific literature. The graph theory allowed to order the findings by calculating scores. Based on this tool, 2 different scores were calculated: a position score that determine the position of the co-occurred terms in an abstract. The more the AOP-related words are placed toward the end of the abstract, the more they can be considered as a result and not a working hypothesis. The second score, the weighted score takes into consideration the complexity of the AOP-related terms (from 1 word to 21 words)  (Carvaillo et al. , 2019). Then, the PubMed database (as of February 2019) was screened manually to decipher targeted information between identified proteins (through the biological enrichment of the protein complexes, as described above) involved in a disease of interest.
Integrating biological data. To get a more comprehensive and complete mechanistic view of BPF associated with the identified AOPs, we included as much as possible information (molecular targets, biological events, toxic effects) at different levels of the biological organization by using several databases. The U. S. EPA's ToxCast program has screened thousands of chemicals for biological activity, primarily using high-throughput in vitro bioassays. The ToxCast dashboard (https://actor. epa. gov/dashboard/) (as of March 2019) was used. Relevant data were also extracted from other sources of information such as the CompTox database (https://comptox. epa. gov/dashboard/) (as of March 2019) and the PubChem bioassays database (https://pubchem. ncbi. nlm. nih. gov/) (as of March 2019). This step allows us to build an individual AOP induced by BPF.
Generation of AOP network. To characterize an AOP network induced by BPF, we assembled shared events (MIE/KE/AO) between the putative AOP developed by the systems biology approach, and available information using the AOP-wiki database (as of April 2019). As a result, the structure of the AOP network is described by at least 2 or more AOPs that share at least 1 event. The development of the AOP network was done by screening the AOP-wiki database to extract events shared with the proposed individual AOP for BPF.
Generation of Protein Complexes for BPF Compounds Using the ToxCast and CTDs, we extracted proteins for which the chemicals show biological activities. Therefore, we were able to compile information regarding 9 proteins for 2,4 0 -BPF, 12 proteins for 2,2 0 -BPF, and 111 proteins for 4,4 0 -BPF (Figure  3 ). As a result, 114 unique proteins were identified, and among them 6 proteins were common to the 3 compounds (AR, ESR1, POU2F1, VRD, NR1I2, and NFE2L2). No information was obtained for the BPF mixed isomer. Therefore, the data analyses were concentrated on these 114 unique proteins that are connected to at least 1 of 3 BPF isomers. Using the list of identified proteins, BPF isomers were linked to protein complexes by determining first-order PPI partners for each of the 114 proteins. In the generated network, 307 proteins (including the 114 input proteins) were connected to their respective partners, with a total of 496 edges (see Supplementary Table  2 ).
Pathways-based Analysis of the Protein Complexes To identify pathways associated with the 3 studied BPF, the protein complexes were enriched by ORA using the KEGG database. Among the 307 proteins present in the network, 288 were mapped to unique entrezGeneID, and therefore used for the ORA (Supplementary Table  3 ). Among these 288 proteins, 158 have annotations in the KEGG database. Consequently, the ORA of these 158 proteins revealed several statistically significant pathways (see Supplementary Table  4 ) and were mapped to a number of categories such as the function of the synapses or cell signaling. Some of these appear to be more specific and potentially connected in terms of mechanisms of action. Several lipid metabolism-related pathways were retrieved with the KEGG analysis. For example, pathways such as "linoleic acid metabolism" (with a corrected p value of 8.
Moreover, among the identified pathways, 4 were associated with endocrine-related cancers (prostate, endometrial, breast, thyroid) (Table  1 ). According to the CompTox database (access as of March 2019), no cancer information was associated yet to the BPF compounds. Therefore, we decided to explore the most significant link (BPF-thyroid cancer) using an integrative approach including text mining, literature searches, and additional databases exploration.
Knowledge-Driven Analysis to Link BPF to Biological Events As a next step, to be able to further support the association of BPF with AOPs, an unsupervised analysis of the existing literature was performed using the tool AOP-helpFinder, followed by a manual targeted literature analysis, and data integration. The main idea was to decipher potential linkages, based on existing knowledge, between BPF and the previously identified endocrine-related cancers by systems biology (Table  1 ). First, an automatic screening of the literature was carried out using the newly developed AOP-helpFinder tool  (Carvaillo et al. , 2019). This tool was run on the 190 publications mentioning BPF extracted from the TOXLINE database, using the developed AOP dictionary  (Carvaillo et al. , 2019). This search led to the identification of 1 AO that is cancer, and 8 KEs (Supplementary Table  5 ).
, 2010; Manole et al. , 2001). Based on these text mining-based results and the ones from the systems biology analysis, we decided to further explore thyroid cancer, as this outcome was the most significant 1 identified by ORA. Therefore in a second literature-based step, associations between the stressor BPF, the biological events identified previously (cell proliferation and thyroid cancer), and the proteins found by ORA (CCND1; CDH1; MAPK1; MYC; NCOA4; PPARG; RXRG) (Table  1 ) were explored by manual  curation of available scientific publications using the PubMed database. This targeted analysis of the literature allowed to identify 9 publications (Supplementary Table  6 ) that were read by scientific experts to characterize a putative individual AOP (Figure  4 ). Among the possible MIE linked to BPF, we identified estrogen-activated pathways (such as estrogen receptor alpha [ER-a] activation), inhibition of catalase and antagonism of PPARc (ie, one of the proteins found by ORA).
To strengthen the links between BPF and PPARc, and to validate the potential relationship between BPF and cell proliferation, several databases were explored (ToxCast, CompTox, and PubChem bioassays). A human fluorescence assay from the ToxCast database (as of March 2019), revealed BPF binding to PPARc with an apparent affinity (AC50) of 38. 5 mM. Therefore, BPF may modulate PPARc signaling pathways involved in various biological functions including cell proliferation and differentiation. Among BPA derivatives, bisphenol-A-diglycidyl-ether is a known PPARc antagonist  (Sato et al. , 2016). From the PubChem bioactivities database, the 3 BPF compounds were defined as human PPARc antagonists (BioAssay 743199) (as of March 2019). In the CompTox database, the 3 BPF compounds were associated with cell proliferation (Tox21 cell-based assay on human) below their cytotoxicity limit (AC50 of 18.
AOP Network Characterization Related to BPF As a final step, a putative AOP network was developed using the proposed individual AOP (Figure  4 ) and additional information from the AOP-wiki database (Figure  5 ). Based on available information from the AOP-wiki database and comparison with the events in the proposed AOP (Figure  4 ), 2 MIEs, 4 KEs, and 4 AOs were retrieved. These links are very coherent; for example, among the AO, endometrial, and breast cancers were also retrieved among the 5 identified pathways (endocrine-related cancers) using the KEGG database (Table  1 ). Promotion (proliferation of a tumoral clone) of endometrial and breast cancer cells has also been associated with the specific activation of ER-a (MIE1181/KE1065 and one of the MIE of the AOP in Figure  4 ) and with KEs that are indirectly linked to cell proliferation (KE870: progression of the G1 phase, increased cell proliferation; KE1182: increase, cell proliferation, epithelial cell; and KE1189: increase, proliferation, endothelial cells) through at least 2 signaling pathways (the transcriptional activation of cyclin D1 by ER-a  (Sutherland et al.
, 2019). In this study, we have attempted to combine the different strategies described above to characterize AOPs which could be linked to BPF isomers. Indeed, biological enrichment of a protein complex (using PPIs) associated with the BPF isomers, literature mining (with AOP-helpFinder and manual curation) and data integration from various sources, allowed to improve our assessment of biological pathways and biological events   6 ). connected to BPF. By using such a higher-level combination of methods, we were not only able to link BPF to an individual AOP but we were also able to associate these isomers to a complex AOP network. These networks are particularly relevant to represent multiple outcomes of a substance and also of a mixture of substances. By combining different modules of text mining/ scoring and of systems biology approaches, we provide a strategy to accelerate connection of a substance to AOs and identify potential critical steps of its putative mode of action.
One advantage of such approaches is the ability to gather disparate types of information (chemical-protein associations, PPIs, protein-signaling pathway annotations) from various sources (literatures, databases), and to integrate them with the aim to identify previously uncharacterized links  (Audouze and Grandjean, 2011). The development of such models is now feasible due to recent advances in both experimental (via highthroughput technologies) and computational areas, eg, though advanced developed methods to identify association between chemicals and health disorders  (Audouze et al. , 2010). A recent study showed that AOP networks allowed to develop new AOPs, reflecting the power of generating AOP networks to better understand mechanistic pathways  (Knapen et al. , 2018). Adverse outcome pathway networks can also be used for assay development and refinement, as shown in a study that created an AOP network for reproductive and developmental toxicity in fish, based on 5 relevant existing AOPs  (Knapen et al.