# Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora
## 3 A Framework for Blind Querying
### Concepts, Definitions and Problem  
**Definition 1**. : _Data Set. A Data Set ds is a set of homogeneous objects (i.e., having the same schema). It is described by a tuple (data set descriptor)_  
\[ds:<ds\_id,\,dataset\_name,\,schema,\,metadata>,\]  
_where \(ds\_id\) is a unique identifier, \(dataset\_name\) is the name of the data set (not unique); schema is a set of field names contained in objects belonging to the data set; metadata is a set of pairs (label, value), which are additional meta-data associated with the data set._  
_The instance of the data set is the actual set of objects in the data set._  
In our model, a data set instance contains objects, specifically, JSON (JavaScript Object Notation) objects. In fact, the traditional format for open data sets is as CSV (comma-separated values) files, where each data item is a row; the first row of the file reports the schema (list of field names) of the data set; consequently, the other rows have the same structure. A CSV file can be easily represented as an array of flat JSON objects, all having the same flat structure. Nevertheless, JSON has become a very popular format for data exchange; many data sets on Open Data portals are provided as JSON data sets. In this case, a data set is an array of JSON objects. They usually are the straightforward representation of flat CSV files; however, even if they are not, objects in the data sets all have the same structure. This is why we decided to rely on the concept of (JSON) object to deal with data items. **Definition 2**. : _Corpus and Catalog. With \(\mathrm{C}=\left\{ds_{1},\,ds_{2},\,\dots\right\}\), we refer the corpus of data sets. The catalog of the corpus is the list of descriptors, i.e., meta-data, of data sets in \(\mathrm{C}\) (see Definition 1)._  
We can now specify the concept of _Query_, in order to illustrate our approach for blind querying a corpus of data sets. **Definition 3**. : _Query. Given a Data Set Name dn, a set \(P\) of field names (properties) of interest \(P=\{pn_{1},pn_{2},\dots\}\) and a selection condition sc on field values, a query \(q\) is a triple \(q:<dn,P,sc>\) (This definition appeared in [2])._  
In order to introduce the general idea behind blind querying and to clarify the query technique in the remainder of the paper, we need to introduce the concept of _query term_. **Definition 4**. : _Query Term. Consider a query \(q:<dn,P,sc>\). With Query Term (term for simplicity), we denote a data set name \(q.dn\), a field name appearing in \(q.P\) or in \(q.sc\), a constant appearing in q.sc (This definition appeared in [2])._  
A term could be composed of many words, separated by blank spaces, underscores or lines. In fact, in JSON, field names can contain blanks, underscores and lines. **Example 1**. : Figure 1 depicts the application scenario. The user wishing to find interesting data sets on an Open Data portal can perform a blind query without knowing anything about the data sets published by the portal. It is the responsibility of the query engine to address the query to the portal, possibly rewriting the query in order to capture lexically or semantically similar terms actually present in the catalog of the corpus. The result is a pool of data sets possibly meeting the wishes of the user. As a concrete example, suppose a labour-market analyst wants to get information about job offers for \"Data Scientist\" located in a given city named \"My City\". The query could be  
\(q:<dn=\) [Job-Offers],  
\(P=\{\)[Company], [Skills], [Contract]\(\}\),  
\(sc=(\)[Location]=\"My City\" AND  
[Profession]=\"Data Scientist\"\()>\). However, in the Open Data portal, there could not exist a data set with name Job-Offers: desired job vacancies could be in a different data set named Jobs. Thus, objects of interest could be retrieved by a slightly different query, i.e.,  
\(nq_{1}:<dn=\)[Jobs],  
\(P=\{\)[Company], [Skills], [Contract]\(\}\),  
\(sc=(\)[Location]=\"My City\" AND  
[Profession]=\"Data Scientist\"\()>\).