dataesr/scientific-paragraphs-categorization huggingface dataset:
Astronomical Queries For a sake of consistency, we introduce the notations listed in Table  1. In the International Celestial Reference System (ICRS), the sky is projected on a celestial sphere. The position of a star is recorded as a two dimensional coordinates, namely the right ascension ra and the declination dec. The ranges of ra and dec are [0  ] respectively. ra is the celestial equivalent of terrestrial longitude. It measures the angular distance eastward along the celestial equator. dec is the latitude-like coordinate, it measures the angular distance of a celestial object north or south of the celestial equator. ASTROIDE focuses on three main basic astronomical queries, that require more calculations than ordinary search or join in legacy relational databases. They are formally defined hereafter.
We define the spherical distance between r and s as SD(r, s), it reflects the shortest distance between r and s over the celestial sphere. It is calculated with the Harvesine formula  [15]  as the length of the great circle arc between two points on a sphere: SD(r, s) = 2 arcsin sin 2 (d)/2 + cos(d 1 ) cos(d 2 ) sin 2 (a/2) Where: Cross-Matching query aims at identifying and comparing astronomical objects belonging to different observations of the same sky region. Cross-matching takes two datasets R, S and a radius as inputs and returns all pairs of points (r,s) such as their spherical distance is lower than. d 1 = XMATCH(R,S, ) ={(r, s) | (r, s) ∈ R × S, SD(r, s) ≤ } Cross-Matching is equivalent to a spatial distance join on two datasets R and S (R S) in database terms. kNN search: Given a query point p, a dataset S and an integer k > 0, the k nearest neighbors from S denoted kNN(p, S) is a set of k objects such that: ∀o ∈ kN N (p, S), ∀s ∈ S -kN N (p, S), SD(o, p) ≤ SD(s, p) Moreover, we can extend these queries to encompass other attributes.
For instance, a query example can be: Selection of sources within a certain angular distance from the specified center position (cone search) fitting as well limitations on a certain magnitude range (or a particular spectral type) ordered by magnitude. Astronomical queries are commonly expressed using the Astronomical Data Query Language (ADQL)  [9]. ADQL is a well-known language adapted to query astronomical data, and is promoted by the International Virtual Observatory Alliance (IVOA). It is an SQL-Like language improved with geometrical functions which allows users to express astronomical queries with alphanumeric properties. ADQL provides a set of geometrical functions: AREA, BOX, ENTROID, CIRCLE, CONTAINS, etc. For example, CIRCLE expresses a circular region on the sky that corresponds to a cone in space.
The ADQL expression of the above query example is formulated as follows: SELECT * FROM gaia WHERE 1=CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 266,-29,0. 0833)) AND magnitude > 18 ORDER BY magnitude ASC A full specification of ADQL is available in  [9]. Here, we present and implement the astronomical queries cone search, cross-match, and kNN search, that are expressed using ADQL. These queries use ADQL geometrical functions including DISTANCE, CIRCLE, POINT, CONTAINS. For the time being, our prototype does not provide a complete compliance with all the features of ADQL such as geometrical data types. However, the full coverage of ADQL does not raise any research issue.
Spark Apache Spark  [4]  is rising in popularity as an alternative to disk oriented processing, due to its ability for executing most computations in memory and its expressive API for complex data processing. Apache Spark is a fast and general purpose engine for large scale data processing. Spark mainly offers an immutable distributed collection of objects for in-memory cluster computing called Resilient Distributed Dataset (RDDs). RDDs are splitted into multiple partitions and operated on in parallel. Spark SQL  [17] , a component on top of Spark core, introduces a new data abstractions called Dataset (a distributed collection of data) and DataFrame. A DataFrame represents a dataset with a defined schema unlike RDDs. Internally, each DataFrame is represented as logical plan in Spark. This logical plan is transformed to a physical plan by applying a set of strategies.
Spark SQL provides also an extensible query optimizer called Catalyst  [17] , which makes adding new optimization techniques easy. It proposes two ways: by defining custom rules and strategies. Catalyst is a general library for representing logical plans as trees and sequentially applying a number of optimization rules to manipulate them. Recently, a cost based optimization module (CBO) has been introduced in Catalyst. This module analyzes all execution plans, assigns a cost to each plan and chooses the lowest cost for the physical execution. However, Spark SQL does not provide natively efficient astronomical query processing capabilities. For that, we need to design and implement a new framework suitable for astronomical data that leverages the power of this framework and profits from the extensibility of its optimizer.
Related Work Early astronomical servers. The VizieR service  [18]  developed by the Centre de Données de Strasbourg (CDS) is an on-line database for accessing astronomical data listed in published catalogs and executing cross-matching queries. Linking database and astronomy has been also introduced by Jim Gray in the SkyServer project  [19]. An early cross-match algorithm  [20]  uses a zoning algorithm to implement of points-near-point, spatial cross-match, and self-match queries. The basic idea is to map the sphere into zones, each zone is a declination stripe of the sphere. The implementation of the zoneMatch algorithm is based on Microsoft SQL server where each object is associated to a zoneID and the cross-match is done through SQL statements using predicates on zoneID. Besides, some recent works  [21]  [22]  [23]  [24]  [20]  proposed customized solutions to execute the main astronomical queries.
In  [21]  and  [24] , authors introduce a cross-matching function including a partitioning approach with HEALPix. Q3C  [22] , standing for Quad Tree Cube, provides a sky partitioning schema for PostreSQL and offers an SQL interface for the main astronomical queries. OPEN SKYQUERY  [23]  allows querying and cross-matching distributed astronomical datasets. The authors propose zoning and partitioning algorithms for parallel query execution using RDBMS technologies. In  [25]  [26], authors extend MonetDB, an open source database system based on a column store approach to manage astronomical data. The goal of the authors in  [26]  is to optimize a column-oriented database to enable the support of large-scale surveys. They implemented a new cross-matching algorithm using hash indexes on a single sorted column of declination values.
AscotDB  [27]  builds on the combination of several pieces of technology: the SciDB, a scalable database that stores data in distributed and multidimensional arrays, AStronomical COllaborative Toolkit (Ascot) for graphical data exploration, and Python for easy programmatic access. It also indexes the spherical coordinates by using HEALPix. This have led the authors to propose a specific mapping schema between the spherical grid induced by HEALPix and the multidimensional array data structure on which SciDB is based. However, none of these efforts provide a scalable astronomical server with a unified programming interface like ADQL, as we target. Their performance are still limited because they are mainly based on centralized or rigid server architecture style. Approaches based on distributed frameworks.
In  [28] , authors report the ability of existing MapReduce management systems (Hive, HadoopDB) to support large scale declarative queries in the area of cosmology. But the query cases used in this work differs from our context because they don not cover the spatial queries, which are typical in astronomical applications. Recent works have addressed the support of spatial data and queries using a distributed data server. Their architecture have followed the development of the Hadoop ecosystem. We devise these works in four representative proposals. SpatialHadoop  [5]  is an extension of Hadoop that supports spatial data types and operations. It improves each Hadoop layer by adding spatial primitives. SpatialHadoop adopts a layered design composed of four layers: language, storage, MapReduce, and operations layers.
For the language layer, it adds an expressive high level SQL-like language for spatial data types and spatial operations. In the storage layer, SpatialHadoop adapts traditional spatial index structures, Grid, R-tree  [29]  and R + -tree  [30] , to form a two-level index structure, called global and local indexing. SpatialHadoop enriches the MapReduce layer by adding two new components, SpatialFileSplitter and SpatialRecordReader. In the operations layer, SpatialHadoop focuses on three basic operations: range query, spatial join, and k nearest neighbor (kNN). [7]  is a scalable multi-dimensional data store for Location Based Services (LBSs), built as an extension of HBase. MD-HBase supports a multi-dimensional index structure over a range partitioned Key-value store, builds standard index structures like kd-trees and Quad-trees to support range and kNN queries.
MD-HBase GeoSpark  [8]  extends the core of Apache Spark to support spatial data types, indexes and operations. In other words, the system extends the resilient distributed datasets (RDDs) concept to support spatial data (SRDDs). GeoSpark provides native support for spatial data indexing (R-Tree and Quad-Tree) and query processing algorithms (range queries, kNN queries, and spatial joins over SRDDs) to analyze spatial data. More recently, SIMBA  [6]  has been proposed as an extension of Spark SQL (not only at the core level of Spark) to support spatial queries and analytics over big spatial data. SIMBA builds spatial indexes over RDDs. It offers a programming interface to execute spatial queries (range queries, circle range queries, kNN, Distance join, kNN join), and uses cost based optimization.
In  [31] , authors provide a comprehensive tutorial that reviews all existing research efforts in the era of big spatial data and classifies existing works according to their implementation approach, underlying architecture, and system components. The aforementioned systems are designed for the geo-spatial context that differs from the astronomical context in its data types and operations. Our work considers celestial objects as points in a spherical coordinate system, whereas these systems process other spatial data types (polygons, rectangles. ) in quadrant plane. They do not provide a high level query language adapted to the astronomical context like ADQL. They do not offer astronomical functions tailored for the spherical coordinates system. They use conventional spatial indexing techniques while we adopt specialized indexing method for astronomical data by using HEALPix.
ASTROIDE Architecture ASTROIDE  [32]  is a distributed data server for big data in astronomy. It is based on the distributed in-memory computing framework Spark, to query huge volume of astronomical data. Figure  2  shows the overall architecture of ASTROIDE. ASTROIDE adapts data partitioning to efficiently process astronomical queries. Queries are expressed using ADQL, providing astronomical functions. To this end, the query parser is in charge of translating the ADQL query into an internal algebraic representation. ASTROIDE extends the Spark SQL optimizer called Catalyst by integrating specific logical and physical optimization techniques. The query processing of ASTROIDE relies on the code generation engine of Spark to operate on in-memory datasets.
Data Partitioning Partitioning is a fundamental component for parallel data processing. It reduces computing resources when only a sub-part of relevant data are involved in a query, and distributes tasks evenly when the query concerns a large number of partitions. Hence, this globally improves the query performances. Spark provides two predefined partitioning mechanisms using DataFrames: hash partitioning and range partitioning. Hash partitioner is the default partitioner. It calculates a partition index based on an element's Java hash code, and puts the keys that have the same hash index in the same partition. The partition index is determined quasi-randomly; consequently, objects having close key values are likely to end up in different partitions. Whereas the range partitioner divides data into roughly equal ranges.
To achieve the first requirement, a spatial grouping of the data is necessary. Nevertheless, a basic spatial partitioning may lead to imbalanced partitions due to the typical skewness of astronomical data. Therefore, the partitioning should be also adaptive to the data distribution. ASTROIDE partitions Spark DataFrames in a way the partitions are balanced while favoring data locality. To this end, we employ HEALPix as an indexing scheme to map the two dimensional spherical coordinates into a single dimensional ID, this ensures the data locality requirement. Close points on the sky have their HEALPix IDs close to each other. So, we sort the data based on their IDs. To achieve the load balancing, we leverage the Spark range partitioner, which yields data partitions with roughly equal sizes. Then, the partitions are stored on HDFS  [33].
This causes a shuffling and distributes efficiently the data among partitions. Since Spark is an in-memory analytical engine, this efficient distribution is only accessible in-memory. To amortize the construction cost for future queries, we store partitioned data on HDFS. So, our solution is to associate a partition number nump to each partition in memory and pack each partition with this number on HDFS. The DataFrame is partitioned by nump using a Spark existing function partitionBy(nump). So, ASTROIDE structures the data on HDFS in such a way that each partition is saved in a separate subdirectory containing records with the same partition number nump. ASTROIDE divides also each partition into buckets using the spark function bucketBy using HEALPix index as the bucketing column ipix. This technique optimizes query execution in a way that makes it efficient to retrieve the contents of a bucket and obviate scanning irrelevant partitions.
This extraction helps ASTROIDE to fetch the query types described in Section 2. 2. After that, we convert the initial query into a parse tree by using a tool provided by the CDS  [34]. Our query rewriting module receives this processed tree and applies a number of transformations and optimizations to produce a new equivalent, but optimized query compliant with the syntax of Spark SQL. The transformations performed depend on the type of the query and substitute parts of the query matching an astronomical pattern with replacement SQL text. We incorporate filters into the optimized query in order to avoid scanning irrelevant partitions. Query parsing performed in ASTROIDE is completely transparent to the user. Internally, there is no difference between using the DataFrame API (case 1) or ADQL (case 2) as the same execution engine will be used for both.
Query Optimization The query optimizer is an important component since it is the responsible for generating a query execution plan that computes the query result efficiently. ASTROIDE extends the Spark Catalyst optimizer by adding custom strategies for astronomical queries. We integrate new strategies for converting Spark non optimized logical plan to an optimized physical plan. The output physical plan is the actual plan which ASTROIDE executes for the final data processing. The query optimizer performs the implemented strategies on the input query to transform the query plan tree into equivalent, but with optimized form by rewriting costly operators or by adding filter operator on our data structure. For instance, Figure  4  shows the transformation of a cross-matching query planning. The naive approach of cross-matching uses a cartesian product to execute a join on the two inputs datasets, and then filters the output according to the join condition.
The cartesian product is inefficient for big datasets. Instead, ASTROIDE's optimizer adds new optimizations strategies that transform all logical plans that implements a cartesian product with filter on a spherical distance formula to an optimized physical plan that we will explain in Section 5. 2. 5 Astronomical queries in ASTROIDE In contrast, ASTROIDE introduces an efficient plan that handles the query using partitioning. ASTROIDE starts by finding all pixels within the query cone using the HEALPix library, group these cells by range and uses the partition boundaries to obtain IDs of partitions which overlap these ranges and thus avoid reading irrelevant partitions. Finally, ASTROIDE filters each selected partition with the spherical distance formula condition. Using ADQL or DataFrame API to execute the cone search gives the same execution time, because both generates the same physical plan (as shown in Figure  5 ).
Cross-matching Query To avoid a cartesian Product, our idea is to first limit the distance computation to pairs belonging either to the same cell or to neighboring cells by using the HEALPix indices, thus generating matching candidates. Then a refinement step computes their exact distance and filters the actual matching pairs In the first step, the objective is to match the data in the same cell or in neighboring cells based on their HEALPix indices. We basically leverage a traditional equi-join of the datasets R and S on HEALPix indices to generate the for each n ∈ Neighbors(t. ipix) do Neighbors is a user defined function which returns the current cell and all the neighboring cells. end for 6: end for 7: candidates = Join(R, S + , R. ipix= S +. ipix) 8: matched = Filter(candidates, sphericalDistance < ) 9: return matched sphericalDistance is a user defined function which computes the harvesine distance between two data points candidates sharing the same cell.
As for the candidates belonging to neighbor cells of an object, a naive approach would be to do a θ-join of the two datasets where the criteria is the membership of the HEALPix indice in the one to the set of HEALPix indices of the neighbors in the other dataset. But θ-join does not benefit from the optimization techniques, which exist for equi-join. To cope with this problem, we augment one of the datasets by replicating all objects on the fly as the cross matching query is issued. In order to facilitate their matching with objects from different cells of the second dataset, we substitute the HEALPix value of the replicates by those of the neighboring cells. Thus, a simple equi-join query on HEALPix value suffices to generate all the candidate pairs. Furthermore, the fact that the datasets are partitioned according to HEALPix order contributes to improve the performances of this operation.
Parquet Scan Filter PartitionFilters PushedFilters SphericalDistance S + = S + ∪ { t | t. ipix = n ∧ t. a = t The pseudo-code of cross-matching (HX-MATCH) is explained in Algorithm 1. It takes as input two partitioned files R and S and a search radius and returns all matching points that satisfy a distance criteria. The algorithm runs as follows: 1. We chose the smallest dataset as a reference, let say S. We augment S (into S + ), by creating for each object as many replicates as the number of neighboring cells, where we substitute the HEALPix value (here ipix) by the current cell and the one of each neighbor cell (Line 4). 2. We apply an equi-join query between R and this augmented dataset S + to get the candidate pairs (Line 7). At last, the refinement step checks the exact objects' harvesine distance and returns the cross-match results (Line 8).
k Nearest Neighbor Query A kNN query takes a dataset R, a point p and an integer k as inputs and returns the k closest objects to p in R. Using Spark SQL, the baseline approach is to scan all objects in R, to calculate their distances to p and selects the top-k objects. An example of a relational query expressing a Such a query requires at least a sort with a complexity in O(n * log n ). In ASTROIDE, we avoid this and design a more efficient algorithm for kNN queries. First, we identify the cell that includes the point p using the HEALPix library, we use the function getPix which returns the HEALPix index of an object represented by spherical coordinates at a defined resolution. Then, we locate the partition that intersects this cell using partitions metadata that we created using our partitioner.
Experiments This section provides an evaluation study of ASTROIDE performance. We compare the effectiveness and the efficiency with the closest prototype in the state-of-the-art, i. e. SIMBA  [6]  and Spark SQL. Experiments were performed over a distributed system composed of 6 nodes having 80 cores with one Gigabit network. The main memory reserved for Spark is 156 GB in total. We used three datasets in our experiments: Each record contains a sourceId, a two-dimensional coordinate (ra and dec) representing the star position on the sky and other attributes including magnitude, metalicity. • GAIA. The public GAIA DR1 dataset  [1] [35]  describes the positions of more one billion sources and represents the most detailed all-sky map in the optical to date. • IGSL. Short of the Initial GAIA Source List  [36] , it is a compilation catalog produced for the GAIA mission.
• Tycho-2. An astrometric reference catalog related to prior surveys containing positions and proper motions of more than two million brightest stars in the sky. In all experiments, since we focus on the query processing cost, and not the result materialization cost, we use COUNT(*) to return the total number of rows in each output DataFrame. Figure  9  measures the partitioning time in ASTROIDE using both hash partitioner and range partitioner on GAIA DR1 dataset. The difference between the two partitioners is explained in Section 4. 1. The two partitioners show a linear growth when the data size increases, the hash partitioner is slightly faster as it partitions data quasi-randomly without keeping data locality. However, the range partitioner is more efficient in querying data. It divides the dataset into approximately equal-sized partitions, each of which contains records with HEALPix indices within a specified range.
Cone Search Query Figure  10  presents how the data size affects the performance of cone search queries on GAIA DR1 datasets. The query execution time increases when the data size increases, this is due to the processing of more partitions when increasing the data size. ASTROIDE and SIMBA scan only few partitions to execute the query. Indeed, ASTROIDE requires less access to partitions than SIMBA but Spark SQL has to scan all objects in the dataset. Figure  11  studies the impact of query radius on execution time, we increased the radius from 2 arc-seconds to 50 arc-seconds. The performance of ASTROIDE, SIMBA and Spark SQL remains constant. ASTROIDE is 2x faster than SIMBA and 5x faster than Spark SQL.
In this paper, we improved the execution time of our algorithm using a different HEALPix resolution based on new experiments. The HEALPix order value 12 is the best choice in our case, it ensures the fastest query as well as the correctness of the result. Next, we evaluated the performance of ASTROIDE, SIMBA and Spark SQL for executing cross-matching queries. As shown in Figure  13 , ASTROIDE is scalable and efficient. The performance shows a linear trend. Our approach shows the best performance compared to SIMBA because it requires less access to partitions and fewer objects along the borders. Since ASTROIDE is using the HEALPix library for its indexing module, a sky partitioning technique adapted to astronomical data, experiments show that ASTROIDE outperforms SIMBA for astronomical queries.