University of North Carolina's HARD Track Experiments at TREC 2004

In the experiment described in this paper, we investigate the effectiveness of a document-independent technique for eliciting additional information from searchers about their information problems. We propose that such a technique can be used to elicit terms for use in query expansion and as a follow-up when ambiguous queries are initially posed by searchers. We use a clarification form to obtain additional information from searchers and create a series of experimental runs based on the information that we obtained from this form. Although we were successful at eliciting more information from searchers, we were unable to demonstrate that this additional information increased performance because of an indexing error that resulted in very poor performance for our baseline and experimental runs. Additionally, we use our clarification form to investigate an alternative measure of topic familiarity and demonstrate how it relates to the length of searchers’ topic descriptions and responses to our clarification form. In this year’s HARD track, we took advantage of the one-shot interaction provided by the clarification form to investigate the effectiveness of various techniques for eliciting additional information from searchers about their information problems. Our experiments were motivated by several interests. First, we were interested in creating a feedback technique for use in situations where a searcher’s initial query is unclear or ambiguous. Although previous research has successfully developed and evaluated the clarity measure for predicting query ambiguity [3], it remains unclear what steps should be taken to clarify ambiguous queries once they are identified. Thus, we sought to extend the work on the clarity measure by investigating techniques that could potentially be used to follow-up ambiguous queries. Second, we were interested in developing and evaluating a generic, document-independent feedback technique that could be used in multiple information-seeking situations. This interest was motivated by the supposition that traditional relevance feedback techniques, which typically present topranked documents or keywords to searchers for feedback, are unlikely to work well in situations where ambiguous queries are posed because there is a large chance that documents retrieved in response to such queries will be irrelevant. Further, a generic, document-independent feedback form could potentially be used to assist digital reference librarians elicit more information from patrons about their information problems and manage voluminous service requests by providing support for question triage. For instance, when initial requests are received by a digital reference service the clarity of the request could be determined. If the request is determined to be inadequate, then the patron could be asked to complete a generic clarification form with the goal of eliciting more information from the patron. Although we are proposing a generic clarification form, it is possible to imagine digital reference clarification forms tailored to specific topic areas and collection types. Finally, we took advantage of the experimental setup of the HARD track to investigate our interest in developing techniques for measuring searchers’ topic familiarity. The familiarity data collected as part of the topic metadata provided a nice opportunity in which to compare data collected using an alternative familiarity measure. Thus, we included a familiarity measure as part of our clarification form. One motivation for including this question as part of our clarification form was that during the planning stages of this year’s track, there was quite a lot of discussion about appropriate techniques for measuring familiarity. Although some consensus was reached about how this would occur in this year’s track, there is still much work to be done on developing valid and reliable techniques for assessing familiarity. Given the length and complexity of the discussion surrounding familiarity in the planning stages of this year’s track,