# Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora
## 6 Experimental Campaign
### Results
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \(Env_{1}\) & **Step 1** & **Step 2** & **Step 3** & **Step 4** & **Step 5** & **Step 6** & **Total** \\ \hline \(Conf_{1}\) & 79,009 & 29 & 4408 & 4,775,258 & 606,463 & 423,296 & 5,888,463 \\ \(Conf_{2}\) & 94,323 & 224 & 689 & 10,249,925 & 659,099 & 427,799 & 11,432,059 \\ \(Conf_{3}\) & 162,491 & 81 & 722 & 12,518,246 & 826,261 & 75,810 & 13,583,611 \\ \(Conf_{4}\) & 212,678 & 89 & 768 & 38,377,416 & 2,147,812 & 324,245 & 41,063,008 \\ \(Conf_{5}\) & 123,362 & 112 & 1232 & 54,605,547 & 15,343,763 & 416,889 & 70,490,905 \\ \(Conf_{6}\) & 74,573 & 42 & 465 & 96,387,450 & 519,479 & 132,383 & 97,114,392 \\ \(Conf_{7}\) & 51,649 & 54 & 1210 & 50,381,822 & 2,348,037 & 283,393 & 53,066,165 \\ \(Conf_{8}\) & 137,705 & 81 & 1750 & 68,159,465 & 17,755,300 & 477,857 & 86,532,158 \\ \(Conf_{9}\) & 116,601 & 115 & 1258 & 96,466,081 & 875,712 & 150,340 & 97,610,107 \\ \hline \(Env_{2}\) & **Step 1** & **Step 2** & **Step 3** & **Step 4** & **Step 5** & **Step 6** & **Total** \\ \hline \(Conf_{1}\) & 18,639 & 25 & 809 & 942,283 & 130,175 & 357,763 & 1,449,694 \\ \(Conf_{2}\) & 15,782 & 154 & 168 & 2,072,424 & 145,164 & 347,329 & 2,581,021 \\ \(Conf_{3}\) & 37,310 & 49 & 176 & 2,703,332 & 195,522 & 63,648 & 3,000,037 \\ \(Conf_{4}\) & 41,833 & 80 & 188 & 9,397,267 & 390,018 & 299,062 & 10,128,448 \\ \(Conf_{5}\) & 21,482 & 68 & 264 & 12,910,417 & 2,916,485 & 363,245 & 16,211,961 \\ \(Conf_{6}\) & 17,548 & 31 & 108 & 20,408,651 & 125,260 & 109,565 & 20,661,163 \\ \(Conf_{7}\) & 12,658 & 46 & 204 & 92,838,15 & 393,117 & 224,765 & 9,914,605 \\ \(Conf_{8}\) & 27,944 & 68 & 321 & 14,424,530 & 3,400,795 & 398,221 & 18,251,879 \\ \(Conf_{9}\) & 26,909 & 86 & 217 & 22,155,578 & 171,715 & 115,647 & 22,470,152 \\ \hline \(Env_{3}\) & **Step 1** & **Step 2** & **Step 3** & **Step 4** & **Step 5** & **Step 6** & **Total** \\ \hline \(Conf_{1}\) & 9762 & 26 & 879 & 444,505 & 50,496 & 148,585 & 654,253 \\ \(Conf_{2}\) & 7341 & 182 & 175 & 1,058,477 & 72,370 & 203,713 & 1,342,258 \\ \(Conf_{3}\) & 16,910 & 58 & 193 & 1,492,059 & 86,309 & 37,498 & 1,633,027 \\ \(Conf_{4}\) & 19,457 & 95 & 188 & 4,627,830 & 254,045 & 146,739 & 5,048,354 \\ \(Conf_{5}\) & 9366 & 80 & 274 & 5,935,994 & 919,258 & 166,406 & 7,031,378 \\ \(Conf_{6}\) & 8061 & 36 & 111 & 8,743,991 & 56,670 & 50,373 & 8,859,242 \\ \(Conf_{7}\) & 6615 & 54 & 221 & 4,272,484 & 246,609 & 105,990 & 4,631,973 \\ \(Conf_{8}\) & 11,850 & 80 & 348 & 7,256,341 & 1,255,804 & 135,068 & 8,659,491 \\ \(Conf_{9}\) & 11,609 & 93 & 219 & 9,869,213 & 73,973 & 45,676 & 10,000,783 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Execution times for the corpus of 15,000 data sets (first Map-Reduce version). Step 4 appears to be the most time-consuming step: its execution times are always at least 10 times larger than execution times exhibited by other steps. This is not surprising because this step performs the VSM evaluation of each query to process with respect to the inverted index, in order to retrieve data sets that possibly match the query. Nevertheless, Step 6 contributes to the total execution time significantly, even though with execution times which are, often, a tenth of Step 4. This is not surprising because Step 6 is responsible for downloading instances of data sets from the Open Data portal and filtering objects that satisfy the selection condition. Nevertheless, with execution environment \(Env_{3}\) (first Map-Reduce version), it happens that Step 4 is faster than Step 6, with configurations \(Conf_{1}\), \(Conf_{2}\) and \(Conf_{3}\); this happens only with 1000 data sets (Table 5).