# Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora
## 5 The _Hammer_ Prototype
### Enhancing Step 4: Exploiting High-Level Abstractions Provided by Spark  
To continue the study about the impact of the Map-Reduce approach on our query engine, we developed an enhanced version of _Step4/5 Executor_, in which Map and Reduce tasks do not interact with _MongoDB_. This version exploits a high-level abstraction provided by _Spark_, called _Dataset_, that has been introduced in version 2 of _Spark_; _Spark_ datasets are developed on top of RDD. The advantage of this abstraction is simplicity: as we will show, the program is written in a very simple way. In the rest of the paper, with the term _sdataset_, we will refer to _Spark_ datasets, while, with _data set_, we will refer to a data set in the corpus \(C\). Algorithm 5 reports the enhanced _Spark_ version of procedure **Step4/5_Executor**. The procedure has two extra parameters, i.e., \(I\) and \(D\): \(I\) is a _sdataset_ containing the inverted index; \(D\) is an _sdataset_ containing the catalog of the corpus; now, \(Q\) is a _sdataset_, still containing queries. Table 1 reports the structure of crucial _sdatasets_ in the procedure. As the reader can see, they are non-flat tables (non 1-NF, non first normal form). In particular, a query in \(Q\) contains vector \(\mathtt{KW}\), that describes keywords in the query (field \(\mathtt{kw}\)) with the corresponding weight (field \(\mathtt{w}\)). Similarly, each element in \(I\) is a tuple that associates a term with the vector \(\mathtt{DS}\) of data set identifiers. \(D\) associates a data set identifier \(\mathtt{ds\_id}\) with a vector \(\mathtt{schema}\), where each tuple describes a term in the schema, its role (e.g., data set name, field, etc.) and its weight (that depends on the role). Let us consider now the procedure **Step4/5_Executor**. Algorithm 5 reports the pseudo-code of the Java-hosted _Spark_ procedure that we implemented (by means of the pseudo-code, we avoid useless technicalities). Lines from 1 to 3 correspond to Step 4, while lines 4 and 5 correspond to Step 5. * On line 1, the temporary _sdataset_ named _q_k_ds_ is computed: it has to contain tuples associating a query (by its identifier q_id) with a keyword kw and a data set (by its identifier ds_id) that contains that keyword. See the structure in Table 1. Line 1a flattens vector KW in tuples describing queries: this is necessary to associate the query with data set identifiers through the inverted index. Notice the **explode** operation in the **select** operation, which transforms the nested tuples into flat tuples (one tuple for each element in the vector). Line 1b. performs a join with the _sdataset_\(I\), that contains the inverted index (notice the join condition after the **on**). The resulting table contains vector DS, with data set identifiers: for this reason, the **select** operation on line 1c exploits the **explode** operation, in order to flatten the tuples by unnesting fields ds_id. * Line 2 prepares _sdataset q_k_ds_ in order to aggregate common terms between a query and a data set. Specifically, a **groupBy** operation is performed, in order to group tuples having the same q_id and ds_id. The subsequent **aggregate** operation aggregates single values of field kw into one single vector, which is given the name K. The resulting temporary _sdataset_ is named _q_ds_; its structure is shown in Table 1.
* The actual computation of the _krm_ value (keyword relevance measure) for each data set with respect to a query is computed on line 3. Specifically, line 3a joins the previous temporary _sdataset q_ds_ with _sdataset_\(Q\), in order to get again the KW vector of keywords with weights. This vector is necessary to compute the cosine similarity. Line 3b actually computes the cosine similarity, which is the value of the new field krm. This is performed by explicitly performing a **nap** operation: for each tuple handled by means of the iterator t, the code within braces after the -> is executed. This code calls function **CosineSimilarity** to compute the relevance measure and extend the tuple with field krm.