# A Query Understanding Framework for Earth Data Discovery
## 5 System Design and Experiment
### Experiment Setup  
Before training models are called in the query understanding framework, metadata and logs were collected as the train data. Publicly available, collection-level PO.DAAC metadata are harvested from the PO.DAAC Web portal through its Web service API. Private portal user logs were provided by the data center as another source of knowledge. The collected metadata are in Directory Interchange Format (DIF) that was recommended by NASA Earth Science Data systems and contain descriptive information, such as organization, category, topic, term, sensor, platform, and description. User logs record users' behavior in the portal, such as searching a query, downloading a data during a time period.  
Figure 6: Workflow of query expansion.  
In the concept recognition module, the values of \"min_count\" and \"phrase_threshold\" in Gensim toolkit are set to 5 and 10, respectively, to build bigram and trigrams models while using the content of metadata description as input. In the name entity recognition step, some metadata attributes were selected as the entity attribute sources and some were regarded as query attributes. Six PO.DAAC metadata attributes (i.e., \"DatasetParameter-Variable\", \"DatasetParameter-Term\", \"Dataset-ProcessingLevel\", \"DatasetVersion-Version\", \"Collection-ShortName\", and \"DatasetSource-Sensor-ShortName\") serve as entity attributes summarizing the unique phrases for the six user-defined name entities. Expressions and sentences in query attributes (i.e., \"Dataset-LongName\", \"DatasetCitation-Title\", and \"Dataset-Description\") are regarded as independent queries. Phrases in these independent queries are labeled with the six types of name entities. Expressions and sentences with labeled information are transformed to a certain format and then passed to the Stanford NLP software for training a NER model. For the query expansion module, a vocabulary similarity database was computed from metadata and usage logs and a RESTful Web service is published to derive highly-related terms for a given phrases. After these models were pre-trained, the query-understanding framework can aid query understanding in the PO.DAAC portal.  
A set of synthetic queries are populated from PO.DAAC logs, in which user input query and corresponding filters that narrow the search are combined as a synthetic query, to evaluate the accuracy of the query understanding framework. For these synthetic queries generated from PO.DAAC query logs, NASA and NOAA scientists were invited to evaluate the parsing result of each step in the query understanding tool while using a four-point scale consisting of \"Excellent\", \"Good\", \"Bad\", and \"Null\":  
* Null means the original query does not contain the corresponding fields (e.g., location, time terms).
* Bad indicates the tool fails to detect the corresponding phrase in a step at all (e.g., failing to recognize \"sea surface temperature\" as a phrase in the concept recognition step).
* Good is used to mark the understanding results that locate somewhere between \"Excellent\" and \"Bad\", and it represents the query understanding tool partially captures the search intent in one step (e.g., detecting both \"sea surface temperature\" and \"group high resolution\" as phrases or detecting \"sea surface temperature\" as a phrase, but failing to identify \"ocean circulation\" as phrases in the concept recognition step).  
Excellent demonstrates that the tool successfully achieves its objective in one step (e.g., labeling \"sea surface temperature\" as variable in the name entity recognition step).  
Qualitatively analyzing the results of a sample query and quantitatively calculating the overall accuracy of different modules in the framework for a collection of synthetic queries that were populated from logs are utilized to estimate accuracy evaluation.  
Figure 7: Workflow of query understanding.