dataesr/scientific-paragraphs-categorization huggingface dataset:
A key requirement for this is robustness and reliability, which is now achieved by performing computation in a fault-tolerant way on hardware which may not be fault-tolerant. A more in-depth overview can be found in  Stickley and Aragon-Calvo (2015). Spark is now for many years the leading solution in this domain. Hosted by the Apache foundation, the code is opensource and its community very active. In very general terms, "Apache Spark is a cluster computing platform designed to be fast and general"  (Karau et al. , 2015). Although available for personal usage on laptops, it will reveal its full potential when running on a cluster on some large amount of data 11  within a distributed file system. To introduce the necessary vocabulary, we outline some of the features of the Spark solution and refer the reader to  Karau et al.
(2015) ;  Tang et al. (2018) ;  Peloton et al. (2018)  for a more thorough technical introduction. In Spark, the user program, called the driver, launches multiple workers. Each worker can hold several executors (processes launched for an application) that execute a number of tasks. 12 Workers read data by blocks (possibly from a distributed file system) and can persist computed objects in memory. In a typical work flow the Spark scheduler sends a set of tasks to the workers in order to compute locally over the nodes holding the data or in memory if they have been cached. The reduced outputs are then returned to the driver that can perform some final combination. Although quite an evolved process, on a practical level the user only needs to interact with RDDs (Resilient Distributed Datasets,  Zaharia et al.
(2012) ) which are abstractions representing a distributed collection of objects. They provide a low level access to Spark features. Scientists will typically prefer to work with DataFrames (from the SQL module), which are built on top of RDD but with data organized into named columns and which also include the benefits of the Spark SQL's execution engine  (Sect. 4. 3. 1). Using Spark presents a number of advantages that are particularly well suited to data analysis tasks in the physical sciences, as we discuss below. Simple parallelisation. The use of the mapReduce paradigm allows a coarse level of parallelisation without ever writing complicated code or directives. This is best illustrated by an example. Suppose you are looking for the maximum value of a huge set of numbers. The driver requires workers to compute their maximum value.
The results from this map step are then returned and combined in the reduce step, here by taking the maximum of their values. Such a behaviour could also be implemented with other mechanisms such as MPI, but would require writing some extra piece of code while, quite remarkably, it is builtin feature in Spark. Optimization. In imperative languages like for example C, C++ or Fortran, each instruction generates some low level commands that are immediately executed. Spark uses the mechanism of lazy evaluation, meaning that most instructions (called transformations) do not immediately trigger execution, but instead updates the construction of a Direct Acyclic Graph or DAG which in physicists language represents a pipeline. It is often a common surprise to newcomers to see how fast some Spark commands seem to perform, but this is only because the pipeline has not yet been closed by specifying an action, so nothing was actually run (see an example in Sect.
4. 3. 2). Then, the python front-end to Spark can be seen as a way of declaratively specifying a calculation that is then passed to the Spark layer for evaluation. An interesting aspect of this is that knowledge of the full pipeline structure allows Spark to perform some precise optimizations based on a set of logical/physical rules before running it. As long as one makes use of Spark functions, the user is relieved of the burden of code optimisation such as accessing different parts of data in different places. Cache and interactive analysis. Maybe the most interesting feature for the end-user analysis is the ability of putting the data in cache. While quite a complex operation behind the hood, on the user side this is as simple as calling the cache() function. Even when the cluster does not allow all the data to be held in memory, putting some part in cache and spilling the rest over disk can still be interesting (see  Fig.
3 of SparkFITS18). A key point is that in general, you do not need to put all the variables in cache, only those that will be accessed several times. The memory requirements are then drastically reduced. Once the relevant variables are cached, all computations can be carried out efficiently enough that an interactive analysis becomes generally possible. For demonstration purposes in the following sections, we have put 110 GB of data in memory on a small cluster, but one can achieve easily 1 TB on large centres like the National Energy Research Scientific Computing Center (NERSC)  13. Performances and scaling. Spark, as a largely used industrial standard, gives generally excellent performances for analysing large datasets. We do not claim that Spark will always give the ultimate performance as depending on the specific domain some implementations with MPI or using relational databases, carefully tuned by experts, may outperform it.
Working with Spark As we saw previously, Spark's basic abstraction is the RDD which represents a coarse-grained distributed collection of objects which allow in-memory fault-tolerant computations. Dataframes were later introduced in 2015 within the SQL module  (Armbrust et al. , 2015). They add to the RDD the knowledge of its data structure which allows a higher level of optimization. Although one can retrieve an RDD from a dataframe, one should always try to use the latter in the analyses, especially for python applications. Indeed the RDD performances are much worse in python than in the native scala language. This is not any more the case when working with dataframes, for which similar performances are obtained in most cases as shown in Sect. 4. 3. 8. Most of astronomers use today the python language for their analyses and we emphasize that pyspark can be run within ipython or a jupyter notebook.
We will then essentially discuss the python interface although some comparison with scala performances will be later shown. In our opinion, the scala language remains an interesting option to be looked at and we highlight some of the reasons why it may be useful to astronomers below: • there exist some Spark kernels in scala to work in jupyter notebooks 14  , • the HEALPix package 15  can be build in java (and therefore used in scala) although not including all functionalities, • the JEP package 16  allows to make calls to external python modules as numpy or matplotlib, and the library Py4J 17  provides an efficient bridge between python and scala. Concerning data formats, spark-fits allows the reading of FITS distributed binary tables and also in recent versions of images. Some connectors also exist to read the HDF5 format (see references in SparkFITS18).
A full interactive analysis We present a suite of rather standard operations 18  an enduser can perform using the output of a 10 years LSST simulation presented in Sect. 4. 1. Commands were run interactively in the pyspark shell on a cluster described in Sect. 4. 2. They are explicitly shown as boxed inline code with their output following right after. We benchmark different options in some cases and focus more on performances in Sect. 4. 3. 8.
Infrastructure The cluster we used for this work, located at Université Paris-Sud in France 21  is rather modest in order to illustrate the fact that you do not need huge resources to achieve spectacular results. It consists of nine 36 GB machines, each with 18 cores, running over HDFS. Since the total memory fraction dedicated to the cache is set to 0. 6, the usable memory amount ( 170 GB) is largely sufficient to hold our full dataset. Then, we used the following set-up • 1 driver (4 GB RAM) • 8 executors (ie. workers) each using 17 cores and 30 GB of RAM. The python interactive shell is run as followed pyspark --driver-memory 4g \ --total-executor-cores 136 \ --executor-cores 17 \ --executor-memory 30g 4. 3. Spark analysis 4. 3. 1. Using dataframes Dataframes in Spark come from the Spark.
SQL module. Similar to the pandas one 22 , they can be viewed as a set of named columns over which you can perform operations, but in a distributed environment. Some native Spark functions act on them and should be used as much as possible since they have been very optimized. They are available through the following import command from pyspark. sql import functions as F Once substantial data reduction has been achieved, we can recover a standard pandas dataframe with the toPandas() method, which opens the door to further standard python analysis. It is worth mentioning that toPandas() returns to the driver memory, so one should keep in mind that the data reduction (reduce methods) must be performed before.
Reading the data We begin by loading the (set of) FITS files using spark-fits. CoLoRe FITS format stores separately the cosmological ( Z_COSMO) and RSD (DZ_RSD) redshifts, but since we only want to work on their sum, we construct the z column on the fly: gal = spark. read. format ( " fits " ) \. option ( " hdu " ,1 ) \. load ( " hdfs : path / to / fits / directory " ) \. select ( " RA " ," Dec " , \ ( F. col ( " Z_COSMO " ) + F. col ( " DZ_RSD " ) ) \. alias ( " z " ) ) This represents a (Spark) dataframe object. We show two ways of accessing columns in the select function, either simply through their names (strings) or, when some operations are to be performed, through F. col(. ) that returns Columns objects. Other dataframe standard ways to access columns are through gal. RA or gal['RA'].
count () \. orderBy ( F. asc ( " bin " ) ) Finally we may want to add the bin locations, drop the bin number and go back to the python world by recovering a standard pandas object: pd = h. select ( " bin " ,\ ( zmin + dz / 2 + h. bin * dz ). alias ( ' zbin ') ,\ " count " ) \. drop ( " bin " ) \. toPandas () The histogram is obtained in about 10 s (Table  1 ) which is impressive for running on 6 10 9 data records. We can now study for instance how the selection function varies with the PZ smearing which is shown on Fig. 3. In order to prepare for the next part, let us see how to build the histogram by calling an external function. Operations may be applied onto dataframes with a User Defined Function (UDF) as: But we find that there are performance issues since execution time goes from the previous 10 s to 2 minutes.
Performances We already discussed the user-time measured for each step. They are summarized in Table  1. But are we far from the best possible ones in Spark? The native Spark language is scala which generally leads to the best performances. So, we have recoded and run all the previous commands in scala and compare performances in Table  1. The initialization phase (load(HDU)) is slightly longer in scala than in python. Caching the 6 10 9 data records which is the most demanding part but only needs to be performed once, is obtained in our case in about 1. 5 mins in each case. Then, the statistics part is slightly more efficient in python. As long as native dataframe operations are used both implementations yield a similar 10 s value for histograms. The use of an external (UDF) function is severely penalized in python.
each FITS files is opened and read, 2. redshift uncertainties are added shooting random numbers, 3. the min and max values are computed and stored, 4. all min/max values are combined to obtain the lowest (highest) min (max). We implemented that process in python: it takes about 45 mins to process the 32 files. Which is to be compared to the 2. 8 + 12. 4 + 97. 7 + 1. 8 2 mins we obtained with Spark. This sequential comparison is unfair since one could also implement some MPI logic to read the files in parallel on each node, or even post-process the files to distribute them on each core. The point is not getting the very best performance but already an excellent one with very little effort. Furthermore, while in the imperative approach all the data in memory is lost when the program ends, using the cache in Spark offers the possibility to investigate several aspects of the data in a single session without ever reloading them.
It further depends on the type of query that is to be optimized. Spatial locality may be achieved straightforwardly in 2D with a Healpix indexing and in 3D with an octree 26. But some more evolved indexing may be required for instance when also considering the color or shape of galaxies. This is an important topic, but one that goes beyond the scope of this paper and will be discussed in a forthcoming one. We have addressed the question of interfacing pyspark to some external python code, which can be done reasonably well using (pandas) user-defined-functions. This however requires I/O's to be quite simple. A difficulty we encountered was that building some application always requires access to some external python libraries that must be available on each executor. While Spark offers some integrated mechanism to transmit binary modules to all workers, it is not a very efficient solution.
The administrator may install some set of common scientific libraries on each node, but it is not a very flexible solution. An elegant solution we found for NERSC was to build a dedicated shifter image (a docker variant) including Spark and the necessary libraries. This is a simple solution that can be put in place by each user, but running a docker-like image over each node is not always supported on every data centre. One can also interface some C, C++ or Fortran codes with scala and the JNA (Java Native Access) library. Some examples were reported in  Zhang et al. (2015) ;  Wiley et al. (2011)  and the interested reader may start with https://github. com/astrolabsoftware/Interfaces for a practical usage. On super-computers one may take benefit from combining the HTC and HPC approaches, a road that has still to be explored.
Acknowledgements We acknowledge the use of the HEALPix package (Górski et al. , 2005) and the CoLoRe fast simulation with support from  David Alonso ,  Anze Slosar  and  Javier Sanchez  that we kindly thank. The Spark work was performed at the  VirtualData center at Université Paris Sud  and we thank  Adrien Ramparison  for the upgrades and maintenance of the cluster. The CoLoRe simulation was run at the  National Energy Research Scientific Computing Center , a  DOE Office of Science User Facility  supported by the  Office of Science of the U. S. Department of Energy  under Contract No. DE-AC02-05CH11231.
Systems biology is the scientific field that focusses on studying complex systems (e. g. living organism, tissue, cell, etc. ) using systems-wide scale analysis that includes (high-throughput) datasets as representatives of an entire phenotype  [11]. This approach has been readily applied in cancer research  [12] , drugs research  [13] , immunology  [14]  and in toxicology  [15]. Systems biology approach can lead to a more holistic and dynamic view of complex biological processes. In order to ensure such view, all necessary data about a certain biological process needs to be either sampled by using high-throughput technologies (holistic view) in time series (dynamic interaction) or has to be collected as such information is dispersed among many published articles. Representing this knowledge as a network in the form of a map is easier to understand the relations/interactions between entities (e.
g. proteins, DNA, RNA, etc. ) than text-based-only database  [16]. The map represents biological mechanisms and through analysis of the of map, either as a static network (topology) or dynamic one (simulations) structural details and regulations can be revealed behind certain biological phenomena  [17] [18] [19] [20] [21]. In order to identify a putative mechanism behind nis regulation during repetitive KI prophylaxis, we have constructed manually a process description (PD) diagram that represents the biochemical reaction network that describes the thyroid hormone synthesis and iodide metabolism in the form of a map. Due to the amount of details or granularity of the biochemical reaction network this type of diagram enables mechanistic descriptions  [22]. After performing network analysis, a sub-network has been selected with nodes that correspond to major participants of the thyroid hormone synthesis and iodide metabolism.
g. CellDesigner software  [23]. The obtained network is a directed, sequential and mechanistic one  [22]. The next step is to analyse the network using Cytoscape software  [24]  and the BiNoM plugin to retrieve a sub-graph with connected nodes of interest. The CellDesigner software is not suitable to perform such analysis. This sub-graph will be then converted manually in to an activity flow (AF) diagram and subsequently reduced by removing nodes that are of no interest. The most reduced AF diagram can then be used for postulating a hypothesis often visualised in a schematic diagram. For a more detailed explanation see text in materials and methods. first step is to construct a PD network that summarises all possible information about iodide metabolism and thyroid hormone synthesis from published articles.
PD network construction To manually construct a biochemical network depicting thyroid hormone biosynthesis and iodide metabolism, all relevant infor mation from scientific articles was retrieved from the PubMed database using the molecule(s) of interest as key word(s), e. g. "nis" was used as a query or the combination of "iodide" and "nis" was used as query. Pendrin was used as separate query either in com bination with key words or not. No data-mining tools were used. The CellDesigner software (version 4. 4)  [23]  was used to represent molecular biological mechanisms, derived from the literature, resulting in a structured network representation (processdescriptive diagram) compliant with Systems Biology Markup Language (SBML) level 2 that is suitable for further computational analysis  [25].
Biochemical network We have manually constructed and curated a PD network that depicts the biosynthesis of thyroid hormones and the metabolism of iodide in a single thyrocyte (Fig. 2 ). The network is displayed as a "geographical" map, a way to easily comprehend the displayed information. The map summarises all known up-to-date informa tion about thyroid hormone synthesis and iodide metabolism. The manually constructed network contains 283 chemical species (including 93 proteins, 34 genes and 34 RNA) that are connected with 201 arches/reactions. In total 217 annotations were included to justify each reaction/edge derived from 111 unique scientific articles. The annotations can be found in the Supplementary In formation. The complete network can be downloaded either as the xml-file generated by the CellDesigner software or as a svg-file which can be opened by any internet browser.
Such an approach has been used before to hypothesise a possible mechanism that induces the epithelial to mesenchymal transition in colorectal cancer  [18]. Furthermore, the application of molecular network as maps in systems biology is becoming more common; there is a scientific community that uses maps of networks as basis for mathematical models, prediction analysis and for hypothesis generation  [53, 54]. To our knowledge this is currently the most up-dated map created (Fig. 2 ) with CellDesigner software  [23]  and it contains more details than similar maps that can be found in the KEGG and Reactome database  [55, 56]. The thyroid hormone signalling pathway from the KEGG database has 123 entities and 71 reactions, the map of reactome database: regulation of thyroid hormone activity contains 56 entities and 20 arches, respectively.
8MYA (95% HPD: 432. 1, 468. 1)  (Hedges et al. , 2015; dos Reis et al. , 2015). Of the ten species of bony fish that split off the spotted gar (Lepisosteus oculatus) lineage  (Amores et al. , 2011) , whose genomes are generally tetraploid, all had two versions of CLTC and at least one CLTCL1 gene, except for cave fish (Astyanax mexicanus) apparently lacking CLTCL1. Eight additional species of vertebrates with high genome coverage and reliable annotation had the CLTC gene but no identifiable CLTCL1 gene. CLTCL1 genes are present in the Caviomorpha and Sciuridae rodent suborders, and lost in the Muroidea suborder from the entire Mus genus  (Wakeham et al. , 2005)  and from rat (Rattus norvegicus). The Cetartiodactyla clade also appears to have lost CLTCL1, as CLTCL1 is absent from the four representative genomes in our dataset (pig (Sus scrofa), sheep (Ovis aries), cow (Bos taurus), Yangtze finless porpoise (Neophocaena asiaeorientalis)).
CLTCL1 Figure  1. Phylogenetic analysis of CLTC/CLTCL1 reveals independent loss of the gene encoding CHC22 clathrin from vertebrate lineages and complete conservation of the gene encoding CHC17 clathrin. Phylogenetic profiles of CLTC/CLTCL1 are shown, with gene presence in the corresponding genome indicated by a filled black circle. All sequences used have less than 5% unspecified residues ('X's in the relevant database). Divergent gene sequences with low sequence similarity but that still fall within the CLTC clade are shown as empty circles (see Materials and methods for similarity threshold and sequence IDs). Based on the profile and species tree the most parsimonious phylogenetic tree for loss and duplication events is inferred and shown as red stars and blue squares, respectively. DOI: https://doi.
Human population genetic analyses indicate purifying selection with ongoing diversification for CLTCL1 To follow up the indication that CLTC and CLTCL1 are subject to different degrees of purifying selection, we investigated their variation in human populations. We analyzed 2504 genomes from the 1000 Genomes Project database, phase 3 (1000  Auton et al. , 2015)  and identified alleles resulting from non-synonymous substitutions for CLTC and CLTCL1. This dataset included individuals from each of five human meta-populations: European (EUR, 503), East Asian (EAS, 504), Admixed American (AMR, 347), South Asian (SAS, 489) and African (AFR, 661). Individual populations with their abbreviations are listed in Supplementary file 1a. The reference sequences for chimpanzee (Pan troglodytes) and pseudo-references for two archaic humans, Altai Neanderthal and Denisovan, were also included to relate allelic variation to the ancestral state.
These two main alleles differ by a single methionine to valine substitution at position 1316 (M1316V) in the protein sequence (SNP ID rs1061325 with genomic location chr22:19184095 on hg19 assembly). The valine at position 1316 is predicted to have a functional effect on the protein since it was categorized as 'probably damaging' with a probability of 0. 975 by PolyPhen  (Adzhubei et al. , 2010)  and as 'damaging' by SIFT  (Kumar et al. , 2009). Sequences in chimpanzee and archaic humans have the M1316 allotype, suggesting that M1316 is likely to represent the ancestral state. To further investigate this, we inspected raw sequencing data from both Altai Neanderthal and Denisovan (Supplementary file 1d). We inferred the most likely genotype to be homozygous for the M1316 amino acid (minimum sequencing depths equal to 40 and 28, respectively).
We then extracted sequencing data for an additional 13 archaic and ancient humans (Supplementary file 1d). We found that the V1316 amino acid is present in Pleistocene hunter-gatherers and Neolithic farmers but not in other Neanderthals or Holocene hunter-gatherers in this limited data set. The equivalent residue in human CHC17 (encoded by CLTC) is also methionine, suggesting that methionine at this position likely pre-dated the initial duplication generating CLTCL1. For the non-human species analyzed (Figure  1 ), all CHC-encoding genes present would produce clathrins with M1316, further indicating its ancient and conserved role in CHC structure or function. To quantify the levels of nucleotide and allelic diversity for non-synonymous sites within human populations, several summary statistics of diversity were calculated.