dataesr/scientific-paragraphs-categorization huggingface dataset:
g. JDBC drivers) and multidatabase techniques to integrate diverse data. Odyssey  [11]  and MISO  [14]  go one step further by addressing materialization of data across NoSQL and relational data stores and physical tuning, aiming at optimal data materialization. With MISO for instance, an application queries the execution layer with an SQL-like API, then the relational store retrieves data from both its data warehouse and the Hadoop store via materialized views. JEN  [23]  is another multistore system that allows joining data from two data stores, HDFS and RDBMS, with parallel join algorithms, in particular, an efficient zigzag join algorithm, and techniques to minimize data movement. As the data size grows, executing the join on the HDFS side appears to be more efficient. These systems are characterized by the absence of data store autonomy and the limited capability of integrating diverse set of data models, which distinguishes them from CloudMdsQL.
Data Model CloudMdsQL sticks to the relational data model, because of its intuitive data representation, wide acceptance and ability to integrate datasets by applying joins, unions and other relational algebra operations. To be robust against schema evolution and driven by the fact that NoSQL data stores can be schema-less, CloudMdsQL keeps its common data model schema-less, while at the same time it is designed to ensure that all the datasets retrieved from the data stores match the common data model. Operators. The common data model supports basic relational operators (projection, selection, joins, aggregation, sorting, set operations). In addition, in order to satisfy the common language requirements, the data model includes another two operators as follows. First, to support data and data-metadata transformations, CloudMdsQL introduces a Python operator that can perform user-defined operations over intermediate relations and/or generate synthetic data by executing embedded code of the programming language Python.
Language Concepts The CloudMdsQL language itself is SQL-based with the extended capabilities for embedding native queries to data stores and embedding procedural language constructs. The involvement of the latter is necessitated by the requirement for performing data and data-metadata transformations and conversions of arbitrary datasets to relations in order to comply with the common data model. To support such procedural programmability, CloudMdsQL queries can contain embedded constructs of the programming language Python, the choice of which is justified by its richness of data types, native support for iteration with generator functions, ease of use, richness in standard libraries and wide usage. An important concept in CloudMdsQL is the notion of "table expression", inspired from XML table views  [6, 15] , which is generally an expression that returns a table (i.
This approach fills the gap produced by the lack of a global schema and allows the query compiler to perform semantic analysis of the query. A named table expression is usually defined as a query against a particular data store and contains references to the data store's data structures. However, the expression can also instantiate other named table expressions, defined against other data stores, thus chaining data as per the requirement for nesting queries. For example, the following simple CloudMdsQL query contains two subqueries, defined by the named table expressions T1 and T2, and addressed respectively against the data stores DB1 (an SQL database) and DB2 (a MongoDB database): T1(x int, y int)@DB1 = ( SELECT x, y FROM A ) T2(x int, z string)@DB2 = {* db. B. find( {$lt: {x, 10}}, {x:1, z:1, _id:0} ) *} SELECT T1.
x, T2. z FROM T1, T2 WHERE T1. x = T2. x AND T1. y <= 3 The purpose of this query is to perform relational algebra operations (expressed in the main SELECT statement) on two datasets retrieved from a relational and a document database. The two subqueries are sent independently for execution against their data stores in order the retrieved relations to be joined by the common query engine. The SQL table expression T1 is defined by an SQL subquery, while T2 is a native expression using a MongoDB query that retrieves from the document collection B the attributes x and z of those documents for which x < 10. The subquery of expression T1 is subject to rewriting by pushing into it the filter condition y <= 3, specified in the main SELECT statement, thus reducing the amount of the retrieved data by increasing the subquery selectivity.
Worker Workers collaborate to execute a query plan, produced by a master, against the underlying data stores involved in the query. As illustrated in Section 4. 2, there is a particular worker, selected by the query planner, which becomes in charge of controlling the execution of the query plan. This worker can subcontract parts of the query plan to other workers and integrate the intermediate results to produce the final result. Each worker node acts as a lightweight runtime database processor atop a data store and is composed of three generic modules (i. e. same code library) -query execution controller, operator engine, and table storage -and one wrapper module that is specific to a data store. These modules provide the following capabilities: • Query execution controller: initiates and controls the execution of a query plan (received from a master or worker) by interacting with the operator engine for local execution or with one or more workers (through communication processors) in case part of the query plan needs to be subcontracted.
Second, a special Python object named CloudMdsQL that represents the context, in which the Python expression is executed, can be invoked from the Python code, thus providing access to data retrieved from data stores by other table expressions. This approach allows for a Python expression to use as input the result of other subqueries and thus provides ability for nesting subqueries as explained in more detail in the following subsection. From query engine's point of view, the difference between Python and native expressions is that Python expressions are processed by a Python operator, which is part of the operator engine, while native expressions are delivered to the corresponding wrappers to process them. From CloudMdsQL programmer's point of view, the common point between Python and native expressions is that if a data store provides a Python API, the native expressions against it can also be written in Python and can also make use of the interoperability tools mentioned above, so that while writing a native expression, the programmer can benefit from the same high expressivity and data integration ability that is available in Python expressions.
To provide this programmability, the wrapper implementation can reuse the CloudMdsQL framework for embedding Python expressions. However, if the data store does not provide Python query interface, it is the wrapper implementer's responsibility to provide mechanism to yield tuples and expose the CloudMdsQL object using native language's concepts. Enhanced features of CloudMdsQL include parameterizing and storing of named expressions. In a parameterized named expression the names and types of the parame-ters are declared in the signature following the WITHPARAMS keyword. Each parameter is then referenced inside the expression by a named placeholder (e. g. the parameter name prefixed by a dollar sign). Parameterized expressions need to be instantiated from other expressions or in a FROM clause by passing actual parameter values.
Nested Queries As stated in the language requirements, CloudMdsQL must provide a mechanism for nesting queries -i. e. a named table expression must be able to instantiate other named table expressions, available in the execution context of the same query, and use their result sets as input. This is achievable in all types of expressions: in native/Python expressions by invoking the CloudMdsQL object, and in SQL expressions by simply referencing named table instantiations directly in the FROM clause, often in combination with references to the data store's tables.
Within SQL Expressions An SQL expression against a data store contains references to data store tables, but may also refer to named table expressions in its FROM clause. If the SQL expression contains such mixed references, its corresponding subquery plan is split by the query compiler into two sub-plans. The first one contains only references to data store tables and is identified as a subquery plan that will be passed to the wrapper. The other one references only the root node of the first sub-tree and instantiations of other named table expressions from the context of the CloudMdsQL query and will be executed by the query engine as part of the common execution plan. This is illustrated with the following example: Original query Rewritten equivalent query Here the query planner, upon parsing the original query and building the subquery plan for T2, detects the usage of the named table T1, plans the join with T1 as the outermost operation within the sub-plan, and pulls it into the common plan, thus transforming the whole query plan to correspond to the rewritten equivalent query above.
Within Native Expressions This subsection focuses on the capability of nesting subqueries within native/Python expressions. CloudMdsQL introduces two approaches that allow the programmer to write expression definitions that iterate through data retrieved by other subqueriestable iteration and join iteration. With table iteration, the Python code of a table expression can iterate through the result set of another table expression by requesting a forward iterator through the CloudMdsQL object, instantiating the iterated table by its name. Because of the forward iteration pattern and due to the pipelining fashion of the query execution, the Python expression will start consuming tuples once a few tuples of the iterated table are available, without having to wait for the entire table to be retrieved.
To build a relevant and adequate query execution plan, the query compiler needs to identify all dependencies between table expression, i. e. for each named expression, the engine needs to know which other named table expressions it iterates through. For native/Python expressions, since a black-box approach is used, the query engine does not perform any processing of the code; therefore the referenced inside the expression tables must be explicitly specified in the expression's signature. CloudMdsQL provides an additional REFERENCING clause, by which the programmer specifies that the expression definition performs iterations through a named table instantiation. For example, let us consider the following query, assuming that DB1 is a relational database with a table person, containing names and addresses of persons, and DB2 is a graph database with Python API providing the function GetShortestDistance, which finds the shortest distance between two cities.
Now we want to query both databases to retrieve persons who work in department Herault, the cities where they live and work and what is the distance between their home and work cities. The execution flow of the above query is quite straightforward. It contains specialized subqueries which are chained in a strict way -first the table person_herault is retrieved for persons who work in Herault; then its dataset is used as input to the other subquery, the result of which is the table person_distance that contains one more column retrieved from the graph database by calling its function GetShortestDistance; and finally a projection in the main SELECT statement defines the format of the result table. This approach provides good functionality because it allows arbitrary chaining of data across subqueries.
The query is processed as follows. First, the subquery against DB1 is rewritten by adding the condition work_dept = 'Herault' and removing work_dept from the projection (it is not needed for the execution of the common query plan). Then, the subquery is executed and the query engine starts retrieving from DB1 tuples that form the result set of the outer relation. Then, the wrapper of DB2 starts the execution of the Python code that queries the graph database. It consumes a projection on the attributes h_city and w_city of the outer relation, iterating through it via the special iterator object CloudMdsQL. Outer, and generates the tuples of its own result set. To handle join iteration, the operator engine pipelines the join attribute values of the outer relation to the iterator object, which allows for the native/Python expression to start immediately iterating through them as soon as a few tuples are available, without having to wait for the entire outer relation to be retrieved.
CloudMdsQL SELECT Statement SELECT queries in CloudMdsQL retrieve data from several data stores using embedded subqueries (for each data store) and integrate the data to build the result dataset. The CARDINALITY clause specifies a user-defined cost function that can be used by the optimizer to estimate the expected cardinality of the named expression's result set. The function is expressed as an arithmetic expression that may refer to the cardinalities of the referenced named tables, e. g. card(T1), and/or any of the named table expression's parameters. Similarly, a SELECTIVITY function may also be defined, which can give an estimate of the expected selectivity of an equality condition on the specified columns. <from_clause> is a regular SQL FROM clause containing references to named table expressions -global or ad-hoc, parameterized or not.
Query Decomposition During query decomposition, the query planner compiles the query and builds a preliminary query execution plan (QEP). A query plan in its simplest form is a tree structure, representing relational operations, where the leaf nodes are references to tables, results from the execution of the subqueries against data stores. At this step, the planner also prepares a set of native queries which will be passed to the corresponding wrappers and hence to the underlying data stores (this process will be explained later). Each node of the query plan represents a relational operation and an intermediate relation, resulting from the operation. For more complex queries, since the language allows a single named table expression to be used as operand to several operations (e. g. referenced in other named table expressions and also in the main SELECT statement), it is possible for an intermediate relation to be the input of more than one operator, therefore the query plan appears to be a directed acyclic graph rather than a tree structure.
At query decomposition step, the boundary between the two subsets is preliminary and may be modified during the query optimization step by pushing operations from the common plan to sub-plans to improve the overall execution efficiency or by pulling operations from sub-plans to the common plan in case a data store is not capable of handling them. The next step of the decomposition is the semantic analysis of the query. Within only the common query plan, all table and column names are verified against the query's ad-hoc schema. On the other hand, since the query engine is agnostic to the underlying data stores' schemas, it does not perform semantic analysis of sub-plans, presuming that this will be done by the data stores upon handling each subquery's native equivalent. In fact, all the sub-plans are kept as abstract syntax trees and are never transformed into execution plans.
• If the data store is not capable of estimating the cost of a subquery, but keeps database statistics (such as cardinalities, number of distinct values per column, etc. ), the wrapper implementation should make use of all available database statistics to provide implementations of the desired cost functions. • If none of the above methods are applicable, but the data store supports aggregate queries like COUNT(*), MIN and MAX, the wrapper should contribute to the catalog information by periodically running in background probing queries  [25] , thus synthesizing and keeping statistics such as the number of tuples in a table, the number of distinct values of an attribute, and the min/max values of an attribute. • However, because of the lack of cost models in some NoSQL data stores and the limited (or lack of) capability to build database statistics, the CloudMdsQL query engine gives its database administrator the possibility to define cost functions that give default cost information in case it cannot be retrieved using the above methods.
• Finally, the user may also provide user-defined cost functions, which is particularly useful in the case of native queries. For example, the native named table expression below defines a simple cardinality function, which gives information that the estimated cardinality of the returned table will be equal to the cardinality of the Outer relation, over which the native expression performs iteration. distance(city_1 string, city_2 string, distance int JOINED ON city_1, city_2 CARDINALITY = card(Outer) )@DB2 = {* for (c1, c2) in CloudMdsQL. Outer: yield ( c1, c2, GetShortestDistance(c1, c2) ) *} With this cost information, the query optimizer executes its search strategy to transform the preliminary execution plan into an optimized one. Notice that, when building its search space, the optimizer considers all sub-plans that are assigned to data stores just as atomic leaf nodes, meaning that the operations within the sub-plans are not subject to reordering.
The approach to perform a bind join is the following: the left-hand side relation is retrieved, during which the tuples are stored in an intermediate storage and the distinct values of the join attribute(s) are kept in a list of values, which will be passed as a filter to the righthand side subquery. For example, let us consider the following CloudMdsQL query: A(id int, x int)@DB1 = (SELECT a. id, a. x FROM a) B(id int, y int)@DB2 = (SELECT b. id, b. y FROM b) SELECT a. x, b. y FROM b JOIN a ON b. id = a. id Let us assume that the query planner has decided to use the bind join method and that the join condition will be bound to the right-hand side of the join operation. First, the relation B is retrieved from the corresponding data store using its query mechanism. Then, the distinct values of B.
Query Execution The QEP is passed to the first worker node for execution. The query execution controller is responsible for interpreting it and controlling its execution by passing native queries to the corresponding wrappers and instructing them to deliver the retrieved datasets to the operator engine and providing the operator engine the sequence of operators it must apply to the retrieved datasets. The execution plan is received by the query execution controller in the form of a JSON document that contains sufficient information to configure and run efficiently each of the CloudMdsQL operations. The first step of the query execution controller is to identify the sub-plans within the plan that are associated to the collocated with the worker data store. Each sub-plan is sent to its corresponding data store wrapper to be translated into a native query.
When the operator requires an existing named table, it is retrieved from the table storage and pipelined into the query execution flow. Data is never directly provided from operators to the wrapper: when necessary, the operator informs which named tables are required to solve the operation inside the data store. There is also a specific operator for CloudMdsQL that executes a Python program and pipelines the result in the form of tuples. This iterator approach obtains tuples as soon as they are generated unless there exists a blocking operation. The resulting tuples are stored as a temporary named table into the table storage. This final table can be retrieved by the application with a forward sequential tuple iterator that supports rewinding and repositioning into marked rows. When the named table is no longer required it is automatically removed from the table storage.
Interfacing Data Stores Wrappers are implemented as plugins to the query engine. In order for a data store to be accessed through the query engine, the wrapper developer must implement the corresponding wrapper following a common interface that is used by the query processor to interact with all wrappers. Whenever a CloudMdsQL query is processed, the query engine prepares a set of native subqueries (or subquery plans) that need to be executed against the data stores. The engine then passes each subquery to the corresponding wrapper, which is responsible for the following: • The execution of native subqueries against the data store, for which there are two possibilities: (1) Server-side execution: The wrapper passes the query to the data store for remote execution (e. g. SQL); (2) Client-side execution: The wrapper executes the query locally, accessing the data store through a client library and API (e.
g. Sparksee and its Python API); • To guarantee that the retrieved data matches the number and types of columns, specified in the signature of the expected dataset in the CloudMdsQL query; • To deliver the tuples of the retrieved datasets to the operator engine; • To be able to instantiate other named table expressions, hence to access intermediate relations from the operator engine (table storage) during execution. To add support for a new data store to the query engine, the database administrator must implement a new wrapper. Whether the new data store will be subqueried through CloudMdsQL with SQL or native expressions depends on the data store's native query mechanism. If the new data store is an RDBMS or a mapping between the data store's query interface and SQL statements exist, the data store can be queried with SQL expressions and its wrapper should be implemented to handle subquery plans by translating them to native queries (see Sections 6.
Querying SQL Compatible NoSQL Data Stores Since the data model of some NoSQL data stores (e. g. key-value or document databases) can be considered as a subset of the relational model, in most cases it is possible to map simple SQL commands to native queries, without compromising the functionality. In fact, SQL-like languages are already commonly used with data stores based on the BigTable data model, e. g. CQL for Cassandra. For such data stores, the recommended approach for subquerying within CloudMdsQL is to use SQL table expressions against the data store, even though the data store does not natively support SQL. Whenever an SQL table expression is used as a nested query against a data store, it is considered as a sub-select statement and hence is transformed into a sub-tree in the query execution plan.
The query involves all the three databases and aims to discover 'conflicts of interest in publications from Inria reviewed in 2013' (a conflict of interest about a publication is assumed to exist if the author and reviewer are friends or friends-offriends in the social network). The subquery against DB3 uses the Sparksee Python API and user-defined functions and in particular, a function FindShortestPathByName defined over a graph object, which seeks the shortest path between two nodes by performing breadth-first search, referring the nodes by their 'name' attributes and limited to a maximal length of the sought path. This query contains two SQL subqueries -one against a relational database and the other against a document database. The parameterized native named table expression friendships against the graph database defines a relation that represents the level of friendship between two persons (expressed by the length of the shortest path between them).