# Understanding Query Combination Behavior in Exploratory Searches
## 3 A Hierarchy of Query Combination Behavior
### Manual QCB Classification  
To better understand QCB, in this section we develop a system to manually classify QCB sessions. Once this was achieved, we were able to characterize features to build models which can automatically classify QCB types. In this paper, we extracted QCB sessions from the AOL and the Sogou search log data sets. We chose these two data sets for the following four main considerations: firstly, search log data are extremely private, so collecting search logs may raise privacy issues, thus leading to few publicly available query log data sets. Secondly, the query logs we can collect is limited; however, both AOL and Sogou data sets have sufficient search logs available for analyses and for training machine learning algorithms (the AOL search log data set is released by America Online with search logs of 650,000 real users [32], and the Sogou search log data set--founded by Chinese commercial search engine Sogou--consists of real query logs of a given month [33]). Third, search logs in these two data sets are collected from real information needs, which make them more able to reflect the user's real exploratory process [32]. Fourth, both AOL and Sogou data are publicly available, so that our research findings could be reproduced. These factors also make AOL and Sogou data sets the most used data sets in information retrieval. For each search log, we used 30 min for identifying time gap sessions of our query log [34]. To find sessions in which users may show exploratory behavior, we performed the following:  
* Filter short sessions: We first filtered sessions with less than three queries since these cannot be exploratory search sessions [35; 36]. Similar criteria have been used in previous work, e.g., [34]. * Filter no click sessions: We also removed search sessions with no click, since these cannot reflect user behavioral characteristics. * Normalize plural nouns in sessions: We replaced the plural nouns in QCB sessions with their corresponding singular nouns. We applied these criteria to find sessions in which users are likely to show exploring [34]. We found that more than 5% of the search sessions show QCB (145,783 QCB sessions in 2,667,714 AOL search sessions and 78,927 QCB sessions in 1,511,916 Sogou search sessions), which illustrates QCB as a common exploratory search behavior. In addition, 98% of these QCB sessions (219,227 QCB sessions) have two CPQs. Due to the large scale of the data set, we sampled 6000 QCB sessions (4000 sessions are from the AOL data set and the others are from the Sogou data set). We believe that in many cases, QCB types can be deduced from looking at user behaviors which are available to the search engine [37]. Therefore, judges classified these QCB sessions based on behaviors, such as: (1) queries; (2) search results; (3) clicked-on web pages; (4) dwell times; (5) further searches after CBQ. Previous work on behavioral analysis has shown that dwell times are associated with the queries, which can yield satisfying information for searchers [38; 39]. For the issued query in a search session, the time stamps of this query and the following query can represent the dwell time of the query. For instance, at time t0, the user issues a query, and at time t1, the user issues the next query. Given the two timestamps, we calculate the dwell time of the query as t1-t0. Similar criteria have been used in previous work to demarcate search sessions, e.g., [7]. For the experiments, we recruited 12 participants affiliated with the Northeastern University campus. We divided these participants into three groups, and the judges in each group were asked to work together on the manual classification of 2000 QCB sessions.