# Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora
## 3 A Framework for Blind Querying
### Retrieval Technique  
The technique we developed for blind querying is fully explained in [2]. Here, we report a synthetic description (from [2]), with the goal to let the reader understand the rest of the paper. Figure 1: Blind querying scenario. The proposed technique is built around a query mechanism based on the _Vector Space Model (VSM)_[25], encompassed in a multi-step process devised to deal with the blind querying approach based on a query rewriting technique. * Step 1: Term Extraction and Retrieval of Alternative Terms. The set \(T(q)\) of terms is extracted from within the query \(q\). Then, for each term \(t\in T(q)\), the set \(Alt(t)\) of similar terms is built. A term \(t^{\prime}\in Alt(t)\) either if it is lexicographically similar (based on the Jaro-Winkler similarity measure [26]) or semantically similar (synonym) based on _WordNet_ dictionary, or a combination of both. Terms in \(Alt(t)\) are actually present in the catalog of the corpus. * Step 2: Generation of Neighbour Queries. For each term \(t\in T(q)\), alternative terms \(Alt(t)\) are used to derive, from the original query \(q\), the _Neighbour Queries_\(nq\), i.e., queries which are similar to \(q\). Both \(q\) and the derived neighbour queries are in the set \(Q\) of queries to process. * Step 3: Keyword Selection. For each query to process \(nq\in Q\), keywords \(K(nq)\) are selected from terms in \(nq\), in order to find the most representative/informative terms for finding potentially relevant data sets (introduced and extensively described in [1]). For each query \(nq\in Q\), its keyword vector \(K(nq)\) is computed: it is represented as a vector \(K(nq)=[k_{1},\,\ldots,k_{n}]\); the accompanying vector \(\overline{w}(nq)=[\overline{w}_{1},\,\ldots,\,\overline{w}_{n}]\) represents the weight \(\overline{w}_{i}\) of each term \(k_{i}\) in vector \(K(nq)\); each weight \(\overline{w}_{i}\in[0,1]\) is the result of the previous steps, on the basis of (lexical or semantic) similarity with respect to original terms (see [2]). * Step 4: VSM Data Set Retrieval. For each query \(nq\in Q\), the selected keywords \(K(nq)\) are used to retrieve data sets based on the Vector Space Model [25] approach: in this way, the set of possibly relevant data sets is obtained. Notice that we select only data sets with a similarity measure \(krm\in[0,1]\) (cosine similarity with respect to a query \(nq\in Q\)) greater than or equal to a minimum threshold \(th\_krm\). Specifically, given a data set \(ds_{i}\), vector \[W(ds_{i}) =[w_{1},\ldots,w_{n}]\] is the vector of weights of each keyword \(k_{h}\) for data set \(ds_{i}\), obtained through the inverted index (refer to [2] for details). The _Keyword-based Relevance Measure_\(krm(ds_{i},nq_{f})\in[0,1]\) for data set \(ds_{i}\) with respect to query \(nq\) is the cosine of the angle between vectors \(\overline{w}(nq)\) and \(W(ds_{i})\). When \(krm(ds_{i},nq_{f})=1\), the data set is associated with all the keywords of the original query. * Step 5: Schema Fitting. The full set of field names in each query \(nq\in Q\) is compared with the schema of each selected data set, in order to compute the _Schema Fitting Degree_\(sfd(ds_{i},nq)\in[0,1]\). Data sets whose schema better fits the query will be more relevant than other data sets. For this reason (refer to [2]), the relevance measure of a data set \(rm(ds_{i})\) is defined as a weighted composition of keyword relevance measure \(krm\) and schema fitting degree \(sfd\). A minimum threshold \(th\_rm\) is set, to select relevant data sets. * Step 6: Instance Filtering. Instances of relevant data sets are processed (i.e., downloaded from the portal and filtered) in order to filter out and keep only the objects that satisfy the selection condition. The result set is so far built: it is a collection of JSON objects.