# Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora
## 5 The _Hammer_ Prototype
### Step 4/5 Executor: Basic Algorithm
On line 12, the **If*
* instruction compares \(ds\_krm\) with the threshold \(th\_krm\): if the former is greater than or equal to the latter, the descriptor \(ds\) is added to the set \(SelectedDS\); otherwise, it is discarded. * Finally, line 14 checks if the set \(SelectedDS\) is not empty: if so, the set of descriptors contained in \(SelectedDS\) is appended to the _MongoDB_celetion **FoundDS*
* by procedure **MongoDB_Add_FoundDS**. This terminates procedure **Process_Query**. * Let us come back to procedure **Step4/5_Executor**. After Step 4 is concluded, Step 5 is performed on line 4, by calling procedure **Evaluate_RM**, reported in Algorithm 3. It is described in detail hereafter:
* The goal of this procedure is computing the _Schema Fitting Degree_ (**r**_sfd_) for all data sets selected by Step 4, in order to compute the overall relevance measure \(rm\). Similarly to Step 4, descriptors of data sets to work on are stored in the _MongoDB_ collection named **FoundDS**, while descriptors of data sets to produce as output of the step are stored in the _MongoDB_ collection called **ResultDS**. Line 1 calls procedure **MongoDB_Clear_ResultDS*
* to clear this collection. * Before starting the iterative process, line 2 initializes the set of data set descriptors to produce as output (\(ResultDS\)); line 3 reads the set of data set descriptors produced by Step 4 from the _MongoDB_ collection **FoundDS**. * The **For Each*
* loop on line 4 operates on each data set descriptor, in order to evaluate the best relevance measure \(rm\); before starting the inner loop, line 8 reads the schema of the data set from the catalog stored in _MongoDB_. The inner loop on line 7 evaluates the data set against each query \(q\in Q\). Specifically, line 8 computes the relevance measure by calling function **Compute_SFD_and_RM**, that computes the _sfd_ and, based on it, the relevance measure \(rm\), which is assigned to variable \(new\_rm\). Line 9 compares this value with the former best value for the data set and, if the new value is greater than the old one, it becomes (line 10) the new best value and (line 11) the data set is associated with the corresponding query. - The last action of the outer loop is the comparison of the relevance measure with respect to the threshold \(th\_rm\): if the measure is greater than or equal to the threshold (line 12), the data set descriptor is added to the output set \(ResultDS\) (Line 13). - Finally, after the outer loop, if the set \(ResultDS\) is not empty (Line 14), it is stored to the **ResultDS** collection in _MongoDB_. ```
Procedure Process_Query(\(q\), \(th\_rm\)) Begin \(DS\): set-of \(ds\) : (ids_d,W : [(t,w)],krm) \(DS\):= \(\emptyset\) \(FoundDS=\) MongoDB_Get_FoundDS() For Each \(t\in q\_terms\)do if\(DS=\emptyset\) Then \(DS=\) MongoDB_DS_for_term_Except(\(t\), \(FoundDS\)) Else \(DS=DS\cap\) MongoDB_DS_for_term_Except(\(t\), \(FoundDS\)) End If \(DS=\emptyset\) Then End Procedure End For Each \(SelectedDS=\emptyset\) For Each \(ds\in DS\)do \(ds\).krm = **CosineSimilarity(\(q\).W, \(ds\).W) If \(ds\).krm \(\geq th\_rm\) Then \(SelectedDS=SelectedDS\cup\{ds\}\) End If End For Each \(SelectedDS\
eq\emptyset\) Then \(\textbf{MongoDB_Add_FoundDS(}SelectedDS)\) End If End Procedure
```  
**Algorithm 2**Procedure for extracting relevant data sets for a query. ```
Procedure Evaluate_RM(\(Q\), \(th\_rm\)) Begin
1. MongOB_Clear_ResultDS()
2.\(ResultS:=\emptyset\)
3.\(FoundDS:=\) MongOB_Get_FoundDS()
4.For Each \(ds\in FoundDS\)do
5.\(ds\_schema=\) MongOB_Get_Ds_Schema(\(ds\_\)id)
6.\(ds\_rm:=0\)
7.For Each \(q\in Q\)do
8.\(new\_rm:=\) Compute_SFD_and_RM(\(ds\), \(ds\_schema\), \(q\))
9.If\(ds\_rm<new\_rm\) Then
10.\(ds\_rm:=new\_rm\) Then
11.\(ds\_\rm:=\)\(q\)End If End For Each
12.If\(ds\_rm\geq th\_rm\) do
13.\(ResultDS:=ResultDS\cup\{ds\}\)
14.End If End For Each
14.If\(ResultDS\
eq\emptyset\) Then
15.\(\textbf{MongoDB_Save_ResultDS}(ResultDS)\) End If End Procedure
```  
**Algorithm 3**Pseudo-code of procedure **Evaluate_RM**, performing Step 5.