dataesr/scientific-paragraphs-categorization huggingface dataset:
Genetic studies Whole exome sequencing was performed on patient A; following a filtering strategy that enriched for rare (MAF of < 1%) nonsynonymous/splice variants that segregated in a recessive manner, a short list of candidate genes was obtained but without any known gene associated with WM disease (Additional file 2). Independently, NGS-based screening of genes associated with mitochondrial disorders performed on patient B disclosed two heterozygous variants in KARS (NM_ 001130089. 1): c. 1124A > G/p. Tyr375Cys and c. 381C > G/p. Phe127Leu. These variants and their segregation in the family were confirmed by Sanger sequencing. Given the striking similarities of clinical and neuroimaging features between patient B and patient A, WES data from the latter were reconsidered and a likely causative role was assigned to the homozygous variant c.
1514G > A/p. Arg505His in KARS. Similarly, because of her clinical presentation an NGS-based screening of KARS was performed on patient C, leading to the identification of two heterozygous variants: c. 815 T > G/p. Phe272Cys and c. 1043G > A/p. Arg348His. All the identified KARS variants had an extremely low frequency in public databases (< 0. 01%), are predicted to be deleterious and hit phylogenetically conserved amino acid residues, supporting their causative role (Additional file 3). Notably, the c. 1514G > A/p. Arg505His found in patient A has been recently reported in compound heterozygosity with another missense change in two siblings with early-onset hearing loss and leukoencephalopathy and its pathogenicity has been experimentally proven  [10].
megaterium  (34)  and 10% in C. perfringens  (18) ). These differences in cortex N-deacetylation are particularly surprising, given that all these genomes contain several potential N-deacetylases. Indeed, six N-deacetylases have been annotated in the B. subtilis genome  (35). Analysis of C. perfringens and B. megaterium genomes using the IMG/JGI database  (36)  showed that these contained 8 and 13 potential N-deacetylases, respectively, based on the identification of a pfam01522 domain. Global levels of GlcNAc N-deacetylation in the spore cortex indicate that among the remaining potential 11 N-deacetylases of C. difficile, there is potentially one enzyme or more able to target this residue in spore cortex, which shows another specificity of C. difficile cortex compared with other species. The link between the number of potential N-deacetylases and N-deacetylation levels in spore cortex has yet to be investigated, and to our knowledge the specific impact of GlcNAc N-deacetylation in the spore cortex has not been studied in any species.
This work was supported in part by the  French Government 's  Investissement d'Avenir Program, Laboratoire d'Excellence "Integrative Biology of Emerging Infectious Diseases,"  Grant  ANR-10-LABX-62-IBEID. The authors declare that they have no conflicts of interest with the contents of this article. This article contains Figs. S1-S6 and Tables  S1-S6. 1 Funded by  Ministere de l'Enseignement Superieur, de la Recherche et de l'Innovation. 2To whom correspondence should be addressed: EA 4043-UniteBacteries Pathoge.
Acknowledgments This paper has been supported by  DFG , as part of the  Collaborative Research Center SFB876  (http://sfb876. tu-dortmund. de/). This work was partially supported by  National Funds through FCT/MEC (Portuguese Foundation for Science and Technology)  and co-financed by  ERDF  ( European Regional Development Fund)  under the PT2020 Partnership, within project  UID/CEC/04234/2013  ( CISTER Research Centre ); also by  ARTEMIS/0003/2012 -JU  grant nr. 333053 (CONCERTO) and  ARTEMIS/0001/2013 -JU grant  nr. 621429  ( EMC2 ) This material is based upon work funded and supported by  NSF  grants  OISE 1427824  and  CNS 1527727 , and the  Department of Defense  under Contract No. FA8721-05-C-0003  with  Carnegie Mellon University  for the operation of the Software Engineering Institute, a federally funded research and development center.
Sky surveys represent a fundamental data source in astronomy. Today, these surveys are moving into a petascale regime produced by modern telescopes. Due to the exponential growth of astronomical data, there is a pressing need to provide efficient astronomical query processing. Our goal is to bridge the gap between existing distributed systems and high-level languages for astronomers. In this paper, we present efficient techniques for query processing of astronomical data using ASTROIDE. Our framework helps astronomers to take advantage of the richness of the astronomical data. The proposed model supports complex astronomical operators expressed using ADQL (Astronomical Data Query Language), an extension of SQL commonly used by astronomers. ASTROIDE proposes spatial indexing and partitioning techniques to better filter the data access.
Introduction Astronomy is undergoing a large and unprecedented growth of astronomical data in both of volume and complexity. Advances in new instruments and extremely large sky surveys are creating massive datasets including billions of objects. The Sloan Digital Sky Survey (SDSS), Large Synoptic Survey Telescope (LSST) and ESA's GAIA mission  [1]  have been the largest and most accurate three-dimensional maps of the Galaxy ever obtained in the history of astronomy. Traditional relational database systems are no longer adequate, not only because of data explosion, but also because of computational complexity of astronomical queries. Recently, the shared-nothing type of parallel architecture, which uses commodity hardware, is becoming a de facto standard in massive data handling. In this context, the distributed in-memory computing framework Apache Spark  [2]  has emerged as a fast and general purpose engine for large-scale data processing in memory.
While Spark fits well the large scale nature of astronomical data, it does not provide native support of astronomical queries. to erroneous query results, as shown by our experiments in Section 6. We have also demonstrated that their query performances remain limited due to their data organization scheme which is less suitable for astronomical data. Hence, new techniques and tools are necessary for processing astronomical data. However, there is a gap between existing distributed systems and astronomical data handling using astronomical query language. Our goal is to fill this gap and introduce a new framework for the management of large volume of astronomical data. Efficient optimization techniques to reduce search space and lighten the cost of distance computation are necessary. In this work, we present ASTROIDE, a distributed data server tailored for the management of large volume of astronomical data and concentrate on query processing and optimization.
Precisely, the support of astronomical data involves extending several levels of Spark. At language level, the query language should be extended to enable astronomical functions. We also extend the Dataframe API with the same functions at the programing level. At the optimizer level, the parsing, the logical and the physical also need to be customized or extended. In ASTROIDE, queries are expressed in Astronomical Data Query Language (ADQL)  [5] , an SQL extension with astronomical functions. At the storage level, more appropriate techniques are needed for organizing and accessing astronomical data. ASTROIDE implements a data partitioner that achieves both spatial locality and load balancing. It also adopts a well-known sky pixelization technique, combined with a spherical space filling curve scheme, namely HEALPix  [6] , to achieve high performance query execution.
At the intermediate level, the query language should be parsed and optimized seamlessly. Various logical and physical optimization techniques for the ADQL execution are proposed, and integrated to Spark SQL thanks to the extensibility of its optimizer. This includes the control of the data partitioning mechanism and spatial indexing as well as the customization of the query plan. Our query optimizer is responsible for generating an efficient execution plan for the given ADQL query; it injects spatial-aware optimization, which avoids reading unnecessary data as much as possible; it evaluates and selects the best plan. Therefore, the optimizer has been extended with various rules tailored for astronomical data. In a nutshell, our system includes the following extensions over Spark: • Index and partitioning support for astronomical data.
Our system combines data partitioning with an indexing technique adapted to astronomical data (HEALPix pixelization), in order to speed-up queries processing time. • Astronomical operators. We design efficient and scalable cone search, kNN search, cross-match, and kNN join algorithms tailored to our physical data organization. • High-level data access. Our framework supports the data access and query using the most common astronomical query language ADQL. • Astronomical query optimization. We extend the Catalyst optimizer and exploit partition tuning for astronomical operators. We introduce new physical and logical optimizations to optimize query execution plans using transformation rules and strategies. Besides these contributions, we fully implemented ASTROIDE  1. Our experiments on real data show its efficiency and its scalability.
Related Work Early astronomical servers. Traditional DBMS technologies have been widely used for the management of astronomical data. For example, the VizieR service  [7]  developed by the Centre de Données de Strasbourg (CDS) is available to provide a tool for accessing astronomical data listed in published catalogs and executing cross-matching queries. Q3C  [8] , standing for Quad Tree Cube, provides a sky partitioning schema for PostreSQL and offers an SQL interface for the main astronomical queries. However, the performance of these systems are limited because they are mainly based on centralized architecture style. AscotDB  [9]  builds on the combination of several pieces of technology: the SciDB  [10] , a scalable database that stores data in distributed and multidimensional arrays, AStronomical COllaborative Toolkit (Ascot) for graphical data exploration, and Python for easy programmatic access.
It also indexes the spherical coordinates by using HEALPix. This have led the authors to propose a specific mapping schema between the spherical grid induced by HEALPix and the multidimensional array data structure on which SciDB is based. Besides, some recent works  [11]   [12]  propose customized solutions to execute some astronomical queries in a parallel architecture. In  [11] , the authors introduce a cross-matching function including a partitioning approach with HEALPix. OPEN SKYQUERY  [12]  allows querying and cross-matching distributed astronomical datasets. The authors propose zoning and partitioning algorithms for parallel query execution using RDBMS technologies. However, none of these efforts provide a scalable astronomical server with a full and unified programming interface like ADQL, as we target.
Approaches based on distributed frameworks. In  [13] , the authors report the ability of existing MapReduce management systems (Hive, HadoopDB) to support large scale declarative queries in the area of cosmology. But the query cases used in this work differs from our context because they don not cover the spatial queries, which are typical in astronomical applications. Recent works have addressed the support of spatial data and queries using a distributed data server. Their architecture have followed the development of the Hadoop ecosystem. We devise these works in four representative proposals. SpatialHadoop  [14]  is an extension to Hadoop that supports spatial data types and operations. It improves each Hadoop layer by adding spatial primitives. SpatialHadoop adopts a layered design composed of four layers: language, storage, MapReduce, and operations layers.
For the language layer, it adds an expressive high level SQL-like language for spatial data types and spatial operations. In the storage layer, SpatialHadoop adapts traditional spatial index structures, Grid, R-tree and R + -tree, to form a two-level index structure, called global and local indexing. SpatialHadoop enriches the MapReduce layer by adding two new components, SpatialFileSplitter and SpatialRecordReader. In the operations layer, SpatialHadoop focuses on three basic operations: range query, spatial join, and k nearest neighbor (kNN). [15]  is a scalable multi-dimensional data store for Location Based Services (LBSs), built as an extension of HBase. MD-HBase supports a multi-dimensional index structure over a range partitioned Key-value store, builds standard index structures like kd-trees and Quad-trees to support range and kNN queries.
MD-HBase GeoSpark  [4]  extends the core of Apache Spark to support spatial data types, indexes and operations. In other words, the system extends the resilient distributed datasets (RDDs) concept to support spatial data (SRDDs). GeoSpark provides native support for spatial data indexing (R-Tree and Quad-Tree) and query processing algorithms (range queries, kNN queries, and spatial joins over SRDDs) to analyze spatial data. More recently, SIMBA  [3]  have been proposed as an extension of Spark SQL (not only at the core level of Spark) to support spatial queries and analytics over big spatial data. SIMBA builds spatial indexes over RDDs. It offers a programming interface to execute spatial queries (range queries, circle range queries, kNN, Distance join, kNN join), and uses cost based optimization.
) are different from our context. Also, these systems do not provide a high level query language adapted to the astronomical context like ADQL. They do not offer astronomical functions tailored for the spherical coordinates system, and do not support specific astronomical operators such as cone search queries, cross-match queries that are supported by our system. They use conventional spatial indexing techniques, while we adopt specialized indexing method for astronomical data by using HEALPix. The Hierarchical Equal Area isoLatitude Pixelization (HEALPix)  [6]  is one of the most popular indexing methods for astronomical data, and is commonly used to index the catalogs. In our context, we used HEALPix as a linearization technique to transform a two dimensional data points (represented by spherical coordinates) into a single dimension value represented by a pixel identifier (HEALPix ID).
Data locality helps us to organize spatially close points in the same partition or in consecutive partitions, and thus optimizes query execution by grouping access to close objects and avoiding access to irrelevant partitions. • The software for HEALPix  [19]  is easily accessible and contains many functionalities that are useful in our context such as filtering neighbor pixels or those in a cone. Prior work has proposed other spatial indices that are suitable for celestial objects, including Hierarchical Triangular Mesh (HTM)  [20]. Comparison between the two indexing techniques is detailed in  [21]. Both HTM and HEALPix hierarchically partition the sky using a fixed number of cells, and each object is associated to the index of the cell that contains the object. The difference between the two schemes is that HEALPix partitions the sphere in equal-area cells whereas HTM partitions the sphere in regions with different sizes.
Astronomical Queries In the International Celestial Reference System (ICRS), the sky is projected on a celestial sphere. The position of a star is recorded as a two dimensional coordinates, namely the right ascension ra and the declination dec. The ranges of ra and dec are [0 • , 360 • ] and [-90 • , 90 • ] respectively. ASTROIDE focuses on three main basic astronomical queries, that require more calculations than ordinary search or join in legacy relational databases. They are formally defined hereafter. For a sake of consistency, we introduce the notations listed in Table  3. 2. • Spherical distance. SD(r, s) defines the spherical distance between r and s, which reflects their shortest distance over the celestial sphere. It is calculated with the Harvesine formula as the length of the great circle arc between two points on a sphere.
Formally: SD(r, s) = 2 arcsin sin 2 ∆δ/2 + cos δ 1 cos δ 1 sin 2 ∆α/2 • Cone Search is one of the most frequent queries in the astronomical domain. It returns a set of stars whose positions lie within a circular region of the sky. Given a dataset R and a cone c defined by a sky position p and a radius around that position, a cone search query returns all pairs of points r ∈ R within c. Formally: Cone-Search(R, c) = {r | r ∈ R, SD(r, c. p) ≤ c. } • Cross-Matching query aims at identifying and comparing astronomical objects belonging to different observations of the same sky region. Cross-matching takes two datasets R, S and a radius as inputs and returns all pairs of points (r,s) such as their spherical distance is lower than. XMATCH(R,S, ) ={(r, s) | (r, s) ∈ R × S, SD(r, s) ≤ } Cross-Matching is equivalent to a spatial distance join on two datasets R and S (R S) in database terms.
• Given a query point q, a dataset S and an integer k > 0, the k nearest neighbors from S denoted kNN(q,S) is a set of k objects such that: ∀o ∈ kN N (q, S), ∀s ∈ S -kN N (q, S), SD(o, q) ≤ SD(s, q) • kNN Join takes two datastes R,S and an integer k > 0 and returns each object r ∈ R with each of its kNNs from S. kNN-Join(R,S) = {(r, s) | r ∈ R, s ∈ kN N (r, S)} Moreover, we can extend these queries to encompass other attributes. For instance, a query example can be: Selection of sources within a certain angular distance from the specified center position (cone search) fitting as well limitations on a certain magnitude range (or a particular spectral type) ordered by magnitude. Astronomical queries are commonly expressed using the Astronomical Data Query Language (ADQL)  [5]. ADQL is a well-known language adapted to query astronomical data, and is promoted by the International Virtual Observatory Alliance (IVOA).
Spark Apache Spark  [2]  is rising in popularity as an alternative to disk oriented processing, due to its ability for executing most computations in memory and its expressive API for complex data processing. Apache Spark is a fast and general purpose engine for large scale data processing. Spark mainly offers an immutable distributed collection of objects for in-memory cluster computing called Resilient Distributed Dataset (RDDs). RDDs are splitted into multiple partitions and operated on in parallel across processing nodes. Spark SQL, a component on top of Spark Core, introduces a new data abstractions called Dataset (a distributed collection of data) and DataFrame. A DataFrame represents a dataset with a defined schema unlike RDDs. Internally, each DataFrame is represented as logical plan in Spark.
This logical plan is transformed to a physical plan by applying a set of strategies. Spark SQL provides also an extensible query optimizer called Catalyst  [2] , which makes adding new optimization techniques easy. Catalyst can be extended in two ways: by defining custom rules, and by adding optimization strategies. Catalyst mainly represents the logical plans as trees and sequentially applies a number of optimization rules to manipulate them. Recently, a cost based optimization module (CBO) has been introduced in this optimizer. This module analyzes all execution plans, assigns a cost to each plan and chooses the lowest cost for the physical execution. Spark SQL is agnostic of astronomical query dialect and optimization techniques. Nevertheless, it offers many advantages as: • A superior performance compared to other distributed systems • Its optimizer is powerful and easily extensible.
Overview of ASTROIDE The primary goal of ASTROIDE  [22]  is to provide a scalable and efficient query processing system for astronomical data. In view of the foregoing, we have based ASTROIDE on Spark framework. Our design takes full advantage of Spark features. Figure  2  shows the architectural components of ASTROIDE. Our framework is composed of two modules: data partitioning and querying system. The data partitioning module is responsible for managing the partitions in a way to ensure data locality, load balancing and task parallelization. Our partitioning algorithm runs as follows. First, we apply a linearization technique to map the two dimensional spherical coordinates representing positions of stars on the sky into a single dimensional ID using HEALPix scheme. As explained in Section 3.
Query optimizer The query optimizer is an important component since it is the responsible for generating a query execution plan that computes the query result efficiently. This module is an extension of the Spark Catalyst optimizer. The decision to exploit Catalyst  [2]  a backbone for our optimizer was driven by the fact that astronomical databases contain both astronomical and non-astronomical queries, as opposed to developing a new optimizer. Our optimizer still leverages all the benefits of Spark's optimizer like predicate pushdown, projection pruning, join reordering,. We distinguish two types of operators: relational operators and astronomical operators. Here, we integrate optimizations for astronomical operators and let Catalyst performs relational operators. In the remainder of the paper, we present the techniques of extending Catalyst to support astronomical queries.
Query2 Find the 6 closest sources to a star represented by a position p=(44. 97, 0. 09) in gaia catalog (kNN). SELECT TOP 6 ra,dec, DISTANCE(Point('ICRS', ra, dec), Point('ICRS', 44. 97, 0. 09)) AS dist FROM gaia ORDER BY dist; The baseline approach to execute this query is to scan all objects in R, to calculate their distances to p, to order them by distance, and finally take the top-6 objects. Similar to spatial selection queries, the filtering phase for relevant partitions can be also applied to avoid scanning the entire dataset. But, unlike the cone search, the identification of the right HEALPIX indices and the concerned partition are far from being obvious. Therefore, we propose an algorithm, which, first, identifies the cell that includes the point p using the HEALPix library. Then, it locates the partition that intersects this cell using the partition metadata that we created using our partitioner.
Query3 Cross-matching two catalogs gaia and igsl, with a radius of 2 arc-seconds (cross-matching query). SELECT * FROM gaia R JOIN igsl S ON 1=CONTAINS(POINT('ICRS', R. ra, R. dec), CIRCLE('ICRS', S. ra, S. dec, 2/3600)); Cross-match is one of the most imperative operations in processing astronomical data. In practice, it involves a cartesian product, which leads to a highly expensive query execution. For large catalogs, this execution becomes intractable. Our solution consists, first, to limit the distance computation to pairs belonging either to the same cell or to neighboring cells, based on HEALPix indices, thus generating matching candidates. Then, a refinement step computes their exact distance and filters the actual matching pairs. In the first step, to force the matching between the data in the same cell or in neighboring cells, we use a trick: we augment one of the datasets by replicating all objects on the fly.
In order to facilitate their matching with objects from different cells of the second dataset, we substitute the HEALPix value of the replicates by those of the neighboring cells. Thus, a simple equi-join query on HEALPix value (by far more efficient than the original query) suffices to generate all the candidate pairs. Furthermore, the fact that the datasets are partitioned according to HEALPix order contributes to improve the performances of this operation. Notice that for the candidates belonging to neighbor cells, a naive approach would be to perform a θ-join where the criteria is the membership to the neighbors of each object cell. But θ-join does not benefit from the optimization techniques, unlike equi-join. The second step computes the distance and removes the false positives to get the final result.