dataesr/scientific-paragraphs-categorization huggingface dataset:
Contextualized Embeddings We use two off-the-shelf ELMo models 7  : ELMo small a general-domain model trained on the 1 Billion Word Benchmark corpus  (Chelba et al. , 2013). This is the small version of ELMo that produces 256-dimensional embeddings. ELMo original the original ELMo model. This is a general-domain model trained on a mix of Wikipedia and newswire data. It produces 1024-dimensional embeddings. Additionally, we also build embeddings by finetuning each model on the i2b2 corpus. The finetuning is achieved by resuming the training of the ELMo language model on the new data (i2b2). At each epoch, the validation perplexity is monitored and ultimately the best model is chosen: ELMo small finetuned the result of fine-tuning ELMo small for 10 epochs. ELMo original finetuned the result of fine-tuning ELMo original for 5 epochs.
Using In-domain Resources It is natural to wonder how our results fare against models trained on large in-domain corpora. Fortunately, there are two such corpora in the clinical domain: MIMIC III (2016) a collection of medical notes from a large database of Intensive Care Unit encounters at a large hospital  (Johnson et al. , 2016)  11. This is a large (1 billion tokens) indomain corpus. PubMed (2018) a collection of scientific article abstracts in the biomedical domain 12. This is a large (4 billion tokens) corpus from a close but somewhat different domain. Both  Zhu et al. (2018)  and Si et al. (  2019 ) trained the ELMo (original) on MIMIC, with the former resorting to only a part of MIMIC mixed with some curated Wikipedia articles. Table  4  reports their results, to which we add the performance of strategies using word2vec embeddings trained on MIMIC and PubMed, and an opensource ELMo model trained on PubMed 13.
We can see yet again that word2vec embeddings perform less well than ELMo models trained on the same corpora. We also see that combining the two kinds of embeddings still brings some improvement (see ELMo (PubMed) + + MIMIC). And more importantly, we observe that by using only general-domain resources, we perform very close to the ELMo models trained on a large indomain corpus (MIMIC) with a maximum difference in F1 measure of ⇡ 1. 5 points. 11 The MIMIC-III corpus can be downloaded at https: //mimic. physionet. org/gettingstarted/ access/. 12 The PubMed-MEDLINE corpus can be downloaded at https://www. nlm. nih. gov/databases/ download/pubmed_medline. html. 13 Since we did not train this model ourselves, we are not sure whether the training corpus is equivalent to the PubMed corpus we use for training word2vec embeddings.
Wikipedia can be seen as a global database of accumulated human knowledge with an immense variety of topics. Moreover the way Wikipedia articles are citing each other encodes scientific, social, historical, and many other aspects. In principle one should be able to extract from Wikipedia direct or indirect relations between cancers, drug cancers and countries. We focus our study on cancer which is one of the major cause of human mortality and which consequently have important social and political impacts all around the world. We aim to measure these impacts through the prism of  Wikipedia. Thus we develop a complementary approach to the existing statistical approaches [2, 4, 5], the Wikipedia network analysis based on the Google matrix and PageRank algorithm invented by Brin and Page in 1998 for World Wide Web search engine information retrieval  [6, 7].
In this section the node influence is defined through the PageRank ranking and the PageRank sensitivity. In the section "Results" we present the influence of cancer devoted pages in Wikipedia and extract a cancer ranking which is compared to cancer rankings extracted from GBD study  [5]  and  GLOBOCAN [2]  databases. We also use the reduced Google matrix to construct a reduced network of cancers and we determine the interaction of cancers with countries and cancer drugs. We corroborate the results obtained from the network structure of Wikipedia articles with various disease burden and/or epidemiological studies. To our knowledge, it is the first time that such correspondence is established. Finally we compare cancer prescriptions obtained from May 2017 English Wikipedia network analysis with approved medications reported in National Cancer Institute  [25]  and DrugBank  [29].
Consequently we observe a new cluster of thoracic region cancers comprising Skin, Breast, Lung cancers and Melanoma. Let us connect to each cancer types the two preferentially linked cancer drugs, i. e. , for each cancer type cr, the two cancer drugs d 1 and d 2 giving the two highest value ðG rr þ G qrnd Þ d;cr. Using DrugBank database  [29] , we easily check that indeed each drug is currently used to treat the cancer type to which it is connected. Also, closely connected cancer types share the same medication, e. g. , Skin cancer and Melanoma are treated by Vemurafenib and Dabrafenib which are enzyme inhibitor of BRAF gene  [57] , Leukemia and Non-Hodgkin lymphoma are treated by the antibody Rituximab targeting B-lymphocyte antigen CD20  [58]. On the other hand non connected cancer types can in some cases share the same medication, the monoclonal antibody Trastuzumab typically used for Breast cancer is now also considered as a drug for Stomach cancer since these two cancer types overexpress the HER2 gene  [59].
and Table 3): Lung cancer is mostly sensitive to Irinotecan, Etoposide, Carboplatin, Breast cancer to Raloxifene, Trastuzumab, Docetaxel, Leukemia to Mercaptopurine, Imatinib, Rituximab, etc. Following the National Cancer Institute  [25]  and/or DrugBank  [29]  databases, these associations cancer-drug are indeed approved. Gestational trophoblastic disease, HPV vaccines to Vulvar and Vaginal cancers, Fluorouracil to Anal cancer, Doxorubicin to Soft-tissue cancers, etc. Again the National Cancer Institute  [25]  and DrugBank  [29]  databases report these possible drug-cancer associations. Let us consider directly the reduced Google matrix associated to the top 20 cancer types and top 20 cancer drugs according to May 2017 English Wikipedia PageRank list (Table  3 ). This reduced Google matrix G R and its G rr , G pr and G qrnd components are shown in Fig  5.
We observe that hidden links gives also accurate medication, see drugs associated to Non-Hodgkin lymphoma and Bladder cancer in Table  6. Conversely for each cancer drug d of the 20 most influent cancer drugs in 2007 English Wikipedia we determine the three most connected cancer types cr, i. e. , the three cancer types with the highest value of ðG rr þ G qrnd Þ cr;d. In Table  7  we show for which cancers a drug is prescribed according to May 2017 English Wikipedia. Again the results are globally in accordance  with National Cancer Institute  [25]  and DrugBank  [29]  databases. We note that hidden links here correspond mainly to clinical trials, e. g. , Imatinib is an approved drug for treatment of certain forms of Leukemia, but experiments were or will be done for Breast cancer and Prostate cancer.
Conclusion Using PageRank and CheiRank algorithms, we investigate global influences of 37 cancer types and 203 cancer drugs through the prism of Human knowledge encoded in the English edition of Wikipedia considered as a complex network. From the ranking of Wikipedia articles using PageRank algorithm we extract the ranking of the most influent cancers according to Wikipedia. This ranking is in good agreement with rankings, by either mortality rates or yearly new cases, extracted from WHO GLOBOCAN 2018 [2] and Global Burden of Diseases study 2017  [5]  databases. 3 ), we give the three strongest cancer ! drug links, i. e. , for a given cancer type cr we select the three cancer drugs d with the highest values ðG rr þ G qr Þ d;cr. Drug with red background indicates a pure hidden cancer ! drug link, i.
Inferred relations between cancer types and countries obtained from Wikipedia network analysis are in accordance with global epidemiology literature. The PageRank sensitivity of countries to cancer types gives also a complementary tool corroborating epidemiological analysis. As far as we know, it is the first study highlighting correspondence between Wikipedia network analysis and disease burden or epidemiological studies. Inferred interactions between cancers and cancer drugs allows to determine drug prescriptions by Wikipedia for a specific cancer. These Wikipedia prescriptions appear to be compatible with approved medications reported in National Cancer Institute  [25]  and DrugBank  [29]  databases.
This work was supported by the  French "Investissements d'Avenir" program , project  ISITE-BFC , contract  ANR-15-IDEX-0003  (JL), by the  Bourgogne Franche-Comte  ´Region 2017-2020  APEX  project, see http://perso. utinam. cnrs. fr/lages/apex/ (JL) and in part by the  Programme Investissements d'Avenir   ANR-11-IDEX-0002-02 , reference  ANR-10-LABX-0037-NEXT , project  THETRACOM , (DLS). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The contemporary air-sea flux of CO 2 is investigated by the use of an air-sea flux equation, with particular attention to the uncertainties in global values and their origin with respect to that equation. In particular, uncertainties deriving from the transfer velocity and from sparse upper ocean sampling are investigated. Eight formulations of air-sea gas transfer velocity are used to evaluate the combined standard uncertainty resulting from several sources of error. Depending on expert opinion, a standard uncertainty in transfer velocity of either ~5% or ~10% can be argued and that will contribute a proportional error in air-sea flux. The limited sampling of upper ocean fCO 2 is readily apparent in the Surface Ocean CO 2 Atlas databases. The effect of sparse sampling on the calculated fluxes was investigated by a bootstrap method, that is, treating each ship cruise to an oceanic region as a random episode and creating 10 synthetic data sets by randomly selecting episodes with replacement.
, 2016). In parallel, the development of the SOCAT community has provided uniform, quality-controlled data sets for surface ocean fCO 2  (Pfeil et al. , 2013)  containing progressively more data values. Recent papers by  Woolf et al. (2016)  and  Goddijn-Murphy et al. (2015)  describe the principles and practical steps involved in reanalyzing upper ocean CO 2 data from SOCAT synthesis data files  (Bakker et al. , 2014; Bakker et al. , 2016; Pfeil et al. , 2013)  to prepare them for synoptic-scale flux calculations. The version of FluxEngine (Version 2) and data used in this study are included in  Holding et al. (2018) , which also provides links to the software. In this paper, we present calculations referenced to 2010, using an enlarged data set on upper ocean CO 2 (Version 4 of  SOCAT; 18.
(2016). Ice fractions were drawn from the OSI-SAF/SSMI sea ice data record. As noted already, analysis in this study is restricted to "global" air-sea fluxes and to four oceanic areas. The four regions are illustrated in the supporting information (Figure  S2 ). A relatively small region south of 60°S is defined as "Southern," while the main oceanic expanse and marginal seas are assigned as "Pacific," "Atlantic," or "Indian" according to International Hydrographic Office (IHO) definitions. The sum of these four regions provides a global value. This global value excludes inland seas such as the Mediterranean Sea but includes the deep ocean and their marginal seas. Generally, through widespread international effort and collaboration the sampling of upper ocean CO 2 measurements has increased over the last few years, as reflected in Figure  2.
Uncertainties are larger when we resolve particular oceanic regions (Figure  4 ) and specific calendar months (Figure  5 ), but most results remain fairly robust. The accumulation and quality controlling of data in the SOCAT databases contributes greatly to reducing uncertainties to a useful level even in many remote regions. One noteworthy result is an apparently robust estimate of the net annual downward flux south of 60°S (the IHO definition of the Southern Ocean) of ~-0. 13 Pg C/year (see Figures  4  and 5 ). That result contrasts with an upward flux calculated by  Takahashi et al. (2009)  south of 58°S but is far more consistent with other analyses of the global carbon cycle (e. g. ,  Gruber et al. , 2009). Excluding older data would be desirable, but we find that excluding data prior to 2005 makes the results less credible (section 3.
This paper focuses on determining the structural similarity of two molecules, i. e. , the similarity of the interconnection of all the elementary cycles in the corresponding molecular graphs. In this paper, we propose and analyze an algorithmic approach based on the resolution of the Maximum Common Edge Subgraph (MCES) problem with graphs representing the interaction of cycles molecules. Using the ChEBI database, we compare the effectiveness of this approach in terms of structural similarity and computation time with two calculations of similarity of molecular graphs, one based on the MCES, the other on the use of different fingerprints (Daylight, ECFP4, ECFP6, FCFP4, FCFP6) to measure Tanimoto coefficient. We also analyze the obtained structural similarity results for a selected subset of molecules.
Motivation This article focuses on algorithmic approaches to compute the structural similarity of pairs of molecules in large molecular databases. Indeed, in organic chemistry, when a new molecule is designed, it is necessary to determine chemical reactions that can be used to synthesize this target molecule from available compounds. Finding such chemical reactions usually consists in searching in a reaction database (such as REAXYS  [1]  or ChEBI  [2] ) for a molecule that is structurally close to the target molecule. As it is sometimes proposed in various existing approaches (see  [3]  and refs), we assume here that: (i) a molecule is represented by a specific graph encoding it structure, and that (ii) two molecules have a similar structure if they have a similar interconnection of the elementary cycles (typically carbon cycles) of their molecular graphs.
When consider solving the MCES problem to measure the structural similarity of molecular graphs, two limitations could occur. First, the required computation time is exponential with respect to the number of vertices of the two graphs, which is a major limitation when considering comparing one molecule with all molecules in a database. Second, considering molecular graphs could provide a similarity measure that is not sufficiently focused on structural similarity (i. e. , the interconnection of elementary cycles of the two molecular graphs). Note that the last approach consists in measuring distances of weighted editions between two molecular graphs, an edition being an operation of adding or deleting a vertex or an edge in the graph, or changing the label of a vertex. These approaches are notoriously used in the field of bioinformatics  [17, 18].
Even if the complexity of such approach can be realistic for molecules of reasonable sizes, it is not convenient to obtain the graph of cycles for all the molecules of a database as Chebi or as Reaxys. Thus, let us consider a molecular graph G = (V, E, w V , w E ) that may be disconnected. We define the structural graph of a molecular graph as the maximum subgraph of G without any vertex with a degree lower than 2 in the subgraph. We first compute a minimum cycle basis B = {c 1 , c 2 ,. , c k } of G by executing the Horton algorithm  [27]  on each 2-connected components of G. Then, to obtain a canonical basis, for each pair c i , c j of cycles in B, and for any elementary cycle c = c i � c j in G and not in B such that |c| = max(|c i |, |c j |), we add c in B. Finally, z j is the set of all the cycles with a length less than or equal to parameter j in B.
Data The target database of molecules considered in the present work is a freely available dictionary of small molecular entities called Chemical Entities of Biological Interest ChEBI  [2]. This database contains 90, 130 molecules. We first uniformly select a set M of 10, 000 molecules among the molecules in ChEBI containing at least three cycles (so that the structural similarity has a meaning). We consider a subset M S of 500 molecules in M whose molecules are also chosen uniformly. We focus on three similarity methods: MCES on molecular graphs (MG), MCES on graphs of cycles (GC), and Tanimoto Coefficient (TC). For Tanimoto coefficient, we used Daylight Fingerprint, ECFP4, ECFP6, FCFP4, and FCFP6. Note that to make sure that the MG and GC methods compute the similarity on the structural part of molecules, we remove all the leaves and isthmus in all molecular graphs for MG before computing similarity.
We then calculate the similarity with MG for all the pairs of molecules in M S and the similarity with GC and TC for all the pairs of molecules in M. Our first goal is to evaluate and compare the performances of the approaches MG, GC and TC on M S from three points of view: the execution time required to calculate the measure of similarity for each pair of molecules, the capacity of each approach to distinguish the subset of pairwise of structurally similar molecules (i. e. , the ones having similar core structures) and finally the capacity of discriminating real similar, medium similar and not similar pairs of molecules. We also analyze the sets of pairwise structurally similar molecules obtained from the MG method in M. Finally, we select seven molecules in M S , pairwise not similar and which have different properties of similarity: some have several homologous molecules in the ChEBI database and others not, some are similar to few non-homologous molecules, others to many.
For each molecule, we consider the distributions of computed similarities in each of the three approaches for the 90, 130 molecules in ChEBI, and we compare them to the pairwise similar subsets of molecules, induced by these three methods. The computation of MG and GC has been done on a cluster Intel(R) Xeon CPU E5-260 v3 @2. 40GHz with 64G of RAM. To find a maximum clique in a graph to solve MCES, we use a linear program resolved by SCIP  [29]  (Solving Constraint Integer Programs). Because of the number of molecules in the database (90130 molecules, knowing that many other databases are larger) and since the similarity calculation between two graphs may have an exponential runtime due to the NP-completeness of the problem MCES (indeed, finding an MCES requires to compute a maximum clique in the product graph of the linegraphs induced by the two considered graphs  [6] ), we have to set a maximum similarity computation time for each pair of molecules.
This time depends on the number of vertices of the considered graphs (MG or GC), which is why the computation time GC is small compared to that of MG since a molecular graph contains more vertices than its graph of cycles; this is especially true when the molecules are similar since in this case, the maximum clique is large. For example, if the maximum time for each similarity is 20 seconds, then the whole computation requires ±20 days on the cluster to compare one molecule to all the other ones in ChEBI. As a consequence of the time limitation, some similarities are not computed for some pairs of molecules in the MG context. The source code is available in the S1 Appendix. For the Tanimoto Coefficient (TC), we used the RDKit software provided by GitHub and SourceForge (http://www. rdkit. org).
1 second/pair respectively with GC and TC. The computation times of the different fingerprints (Daylight, ECPF, FCFP) are equivalent, therefore we present the time for Daylight fingerprint only. It should be noted that we developed GC and MG in C and the software RDKit used for TC is developed in C++. What is important to note here is the low speed of execution of MG compared to GC and TC. There is no pre-processing time needed on MCES with MG. Table  2  presents the pre-processing time for M. Over 99. 98% fingerprints were computed in less than 0. 1 seconds each whereas 8. 74% of graph of cycles need more than 0. 1 seconds. pair. Looking at MG vs TC, we find that dots are aligned around the upper diagonal part. This means that TC and MG provide quite close similar values. There are few dots located in the left upper and right lower part.
Amphotericin B has a particular cyclic structure, its minimum cycle basis contains 3 cycles with a particular cycle of length 36 (this cycle belongs to the structural part). The corresponding graph of cycles thus contains 3 vertices (Fig  15). The GC distribution of similarities concerns all the molecules of the database. This distribution given in Fig  16  shows 11 molecules fully similar to the target one (degrees of similarity equal to 1), and another distinguished set of molecules being partially similar to it (degrees of similarity equal to 0. 6 or to 0. 7). The other molecules can be considered as different from the target molecule (similarity lower than 0. 5). Thus, the calculation using graphs of cycles discriminates the molecules into three classes, which the molecular graph approach does not do.
Stefi Nouleho Ilemo  is funded by the  LABEX DIGICOSME   11LABX-0045  from the  French ANR. The funder  Labex Digicosme  provided support in the form of salaries for author SNI but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of this author are articulated in the 'author contributions' section.
This paper illustrates our conviction that data processing relying on nightly batches is a thing of the past. It shows how a complex analytics platform for huge amounts of data can be created from scratch in a matter of minutes on the cloud, thus negating the need for dedicated on-premise machines that would do the job slower at a higher cost. We also present the CloudDBAppliance project, where we aim at building a Cloud database platform that will be available as a service, and will integrate the operational database with the tools for further data analysis, eliminating the hassle of exporting the data for processing. a.
TESTING A FINANCE USE-CASE ON A CLUSTER ON THE CLOUD We simulated a use-case from the finance industry based on ActiveViam's experience working with the largest banks in the world. Finance is one of the most demanding verticals for analytical capabilities. We loaded millions of transactions, prices and sensitivities recorded for hundreds of historical dates, amounting to 50 TB of data in memory across a 1,600-core cluster on Google Cloud. We wanted to see if a combination of Google Cloud hardware and services and ActivePivot in-memory technology could achieve high-enough performances to perform live analytics on such a volume of data during the course of a regular work day, without relegating any task to batch processing. We tested the scenario from start to finish, including setup, loading, queries and calculations.
Setup and Loading The exceptional storage and networking infrastructure used by public cloud providers make it possible to decouple storage from compute. They are fast enough to load large datasets in minutes and operate in-memory analytical platforms on demand. At least that is the theory, because not every software platform can be operated on-demand. Inmemory databases from Oracle, Microsoft and SAP for instance are now available in the Cloud, but still follow a monolithic architecture where storage is attached to and managed by the database, and that database is always on. To take full advantage of the cloud model and effectively eliminate the need for batch processing, another approach is needed that decouples storage from compute and allows customers to use memory resources on demand.