TREC-8 Interactive Track

William Hersh
hersh@ohsu.edu
Division of Medical Informatics & Outcomes Research
Oregon Health Sciences University
Portland, OR 97201, USAPaul Over
over@nist.gov
Natural Language Processing and Information Retrieval Group
Information Access and User Interfaces Division
National Institute of Standards and Technology
Gaithersburg, MD 20899, USAApril 5, 2000
For TREC-8 the high-level goal of the Interactive
Track remained the investigation of searching as aninteractive task by examining the process as well as
the outcome. To this end a common experimental
framework was designed with the following features:

The experimental design allowed groups to estimate the e ect of their experimental manipulation
free and clear of the main (additive) e ects of searcher
and topic and it was designed to reduce the e ect ofinteractions, e.g., searcher with topic, topic with system, etc.
In TREC-8 the emphasis was on each group's exploration of di erent approaches to supporting thecommon searcher task and understanding the reasons for the results they get. No formal coordinationof hypotheses or comparison of systems across sites
was planned, but groups were encouraged to seek out
and exploit synergies. Some groups designed/tailoredtheir systems to optimize performance on the task;others simply used the task to exercise their system(s). Groups from the following institutions took
part: New Mexico State University at Las Cruces,
Oregon Health Sciences University, Royal Melbourne
Institute of Technology/CSIRO, Rutgers University,Sheeld University, the University of California at
Berkeley, and the University of North Carolina at
Chapel Hill. A total of 936 searches were performed
as part of the experiments.

 an interactive search task - a form of question

answering 6 topics - brief statements of information need 12 searchers - a minimum, more if possible a document collection to be searched: Financial

Times of London 1991-4 with 210,158 articles totaling 564 megabytes.

 a required set of searcher questionnaires
 5 classes of data to be collected at each site andsubmitted to NIST

 3 summary measures to be calculated by NIST

for use by participating groups1

In addition to running its experimental system(s), Each searcher performed six searches on the docueach participating site chose a control system appro- ment collection using the six interactive track topics
priate to the local research goals. The control and the in a pseudo-random order. Each searcher performedexperimental systems were to be provided with equal 3 searches on one of the site's systems and then 3 oncomputing resources within a site but not necessarily the other to avoid the extra cognitive load of switchthe same as those provided at other sites.
ing systems with each search. Instructions on theSix of the 50 topics created by NIST for the task preceded all searching and a system tutorial preTREC-8 adhoc task were selected and modi ed for ceded the rst use of each system. In addition, eachuse in the interactive track by adding a section searcher was asked to complete a questionnaire, priorcalled \Instances" and removing the \Narrative." to all searching, after each search, after the last search
The six topics were entitled \tropical storms" (408i); on a given system, and after all searching was com\Cuba, sugar, imports" (414i); \declining birth plete. The detailed experimental design determinedrates" (428i); \robotic technology" (431i); \tourism, the pseudo-random order in which each searcher used
increase" (438i); and \tourists, violence" (446i).
the systems (experimental and control) and topics.
Each of the six topics described a need for informa- Six sorts of data were collected for evalution of a particular type. Contained within the doc- ation/analysis (for all searches unless othuments of the collection to be searched were multiple erwise speci ed) and are available from thedistinct examples or instances of the needed informa- TREC-8 Interactive Track web page (wwwtion. Here is an example interactive topic.
nlpir.nist.gov/projects/t8i/t8i.html). The typesof data are sparse-format data - list of documents
Number: 408i
saved and the elapsed clock time for each search;
Title: tropical storms
rich-format data - searcher input and signi cantevents in the course of the interaction and their
Description: What tropical storms (hurritiming; searcher questionnaires on background, user
canes and typhoons) have caused propsatifaction, etc.; a full narrative description of one
erty damage and/or loss of life?
interactive session for topic 408i; and any further
Instances: In the time alloted, please nd as
guidance or re nement of the task speci cation given
many DIFFERENT storms of the sort
to the searchers.
described above as you can. Please save
Only the sparse-format data were evaluated at
at least one document for EACH suchNIST
to produce a triple for each search: instance
DIFFERENT storm. If one document
precisionand recall (these as de ned in the next secdiscusses several such storms, then yoution)
andelapsed clock time.
need not save other documents that reEvaluation
by NIST of the sparse-format data propeat those, since your goal is to identify
ceededasfollows.
For each topic, a pool was formed
as many DIFFERENT storms of the sortcontaining
theuniquedocuments saved by at least
described above as possible.
one searcher for that topic regardless of site. For
The task of the interactive searcher was to save each topic, the NIST assessor, normally the topic audocuments, which, taken together, contained as many thor, was asked to read all of the documents from thedi erent instances as possible of the type of infor- pool, determine the set of possible instances, record
mation the topic expressed a need for - within a 20 which instance(s) occurred in which document(s).
minute time limit.
Then for each search (by a given searcher for a given
Searchers were encouraged to avoid saving docu- topic at a given site), NIST used the submitted list
ments which contribute no instances beyond those of selected documents and the assessor's instancein documents already saved, but there was no scor- document mapping for the topic to calculate:
ing penalty for saving such documents and searchers
were to be told that.
 the fraction of total instances (as determined by2the assessor) for the topic that are covered by
the submitted documents (i.e., instance recall) the fraction of the submitted documents whichcontain one or more instances (i.e., instance precision)
The third measure, elapsed clock time, was taken directly from the submitted results for each search.
Comparison of systems across sites is not supported by the experimental design, so comparisonspresented here are between systems within a givensite. Still, a general theme running through the results was that there was little di erence between eachgroup's experimental and control systems even when
they di ered by the presence of techniques shown
to be e ective in non-interactive experiments in thepast. This suggests these techniques, such as relevance feedback, Okapi weighting, document summarization, and greater control over search terms andBoolean operators, do not show bene t in the instance recall task. Only further research can showwhether this is true or due to a lack of experimental
power. The actual results obtained by each group aresummarized in the following paragraphs.
 New Mexico State University looked at whether
users could nd relevant information with a userinterface for viewing retrieval results that showed
query term occurrence and distribution alongwith extracted names of people and locations
shown in document surrogate lists and summaries. Their results showed no di erence ininstance recall between the two systems. However, 11 of 12 users reported that they liked thesummary display better than the full text control. However, this preference did not match performance. Of note was that users viewed more
documents in the full text condition.
 Oregon Health Sciences University used the interactive track to assess whether batch and user
evaluations give the same results. Batch experiments with TREC-6/7 data showed substantial
di erences for various weighting schemes, andparticular bene t for Okapi. User experiments
showed no such comparable bene t. For more information see the site report (Hersh et al., 2000).

 RoyalMelbourne Institute of TechnologyCSIRO tested the hypothesis that by allowing
the user control over the organization of the information, and the selection of documents using
the organization, the user would nd a betterset of documents to view, and hence achieve abetter coverage of aspects. Their control system
featured three windows with a list of document
titles, one document displayed, and a saved instances window. The experimental system replaced the list of document titles with a window containing a list of categories and documents clustered therein. The categories were derived from WordNet. Results showed that using
the categorized interface, users read more documents, saved the same number of documents,and saved more aspects, but with less accuracy.
User satisfaction did favor the categorized system. For more information see the site report(Fuller et al., 2000).
 Rutgers University compared two di erent techniques for supporting query reformulation byterm suggestion in interactive IR: user-controlled
relevance feedback (RF) and system-controlled
Local Context Analysis (LCA). Their results
showed that LCA did not perform better, but
was easier for the user. E ectiveness and usability were the same for each system. In LCA mode,
more terms were suggested than in RF and moresuggested terms were used in the queries, whichwere equally long in both systems. Thus users
had to do less (cognitive) work in LCA. The authors speculated that if LCA terms were "better," then maybe the approach would be more
e ective, usable, and preferred. For more information see the site report (Belkin et al., 2000).
 Sheeld University focused on searching behavior and user perception of an experimental retrieval task assessing the impact of document
ranking, best-passage retrieval, and a query expansion facility. The experimental setting used
two versions of Okapi, one with relevance feedback and one without. Their ndings showedthat while user outcomes were the same, searchcon dence was positively associated with the3References

number of instances retrieved. For more information see the site report (Beaulieu, Fowkes,Beaulieu, M., Fowkes, H., Alemayehu, N., & SanderAlemayehu, & Sanderson, 2000).
son, M. (2000). Interactive Okapi at Sheeld TREC-8. In E. M. Voorhees & D. K. Harman(Eds.), The Eighth Text REtrieval Conference University of California, Berkeley assessed new(TREC-8). Gaithersburg, MD, USA.
features added to its Cheshire II experimentalsystem, in particular the Boolean NOT capability and new ways for navigating results and se- Belkin, N. J., Head, J., Jegn, J., Kelly, D., Lin, S.,
Park, S. Y., Cool, C., Savage-Knepshield, P., &lecting relevant items. Users achieved the same
Sikora, C. (2000). Relevance Feedback versus
instance recall as they did with the previous sysLocal Context Analysis as Term Suggestion Detem. For more information see the site report
vices: Rutgers' TREC-8 Interactive Track Ex(Larson, 2000).
perience. In E. M. Voorhees & D. K. Harman(Eds.), The Eighth Text REtrieval Conference(TREC-8).
Gaithersburg, MD, USA.
 University of North Carolina found no di erence
among various levels of relevance feedback. For Fuller, M., Kaszkiel, M., Kimberley, S., Zobel, C.,
more information see the site report (Yang &Justinand Ng, Wilkinson, R., & Wu, M. (2000).
Maglaughlin, 2000).
The RMIT/CSIRO Ad Hoc, Q&A, Web, Interactive, and Speech Experiments at TREC 8. In
E. M. Voorhees & D. K. Harman (Eds.), The
These results place an imperative on continued
Eighth Text REtrieval Conference (TREC-8).
user-oriented evaluation. While non-interactive evalGaithersburg, MD, USA.
uation will continue to have its role, such as in assessing the feasibility of new algorithms and approaches Hersh, W., Turpin, A., Price, S., Kraemer, D., Chan,and the parameterization of system features, interB., Sacherek, L., & Olson, D. (2000). Do
active experiments must verify that new system adBatch and User Evaluations Give the Same Revances can be used with their intended bene ciaries,
sults?: An Analysis from the TREC-8 Interacreal users. Just because users and user studies are untive Track. In E. M. Voorhees & D. K. Harmanpredictable as well as resource-consuming, this does(Eds.), The Eighth Text REtrieval Conference
not mean we should avoid them.
(TREC-8). Gaithersburg, MD, USA.
Since the interactive track has focused on the in- Larson, R. R. (2000). Berkeley's TREC-8 Interstance recall task for three years running, a growactive Track Entry: Cheshire and Zprise. In
ing consensus of participating groups prefer to asE. M. Voorhees & D. K. Harman (Eds.), The
sess di erent retrieval tasks and documents. Next
Eighth Text REtrieval Conference (TREC-8).
year's track will likely move to more of a questionGaithersburg, MD, USA.
answering approach using data from the Web track.
The participants also hope to explore speci c aspects Yang, K., & Maglaughlin, K. L. (2000). IRIS atof the interactive retrieval task. For example, fuTREC-8. In E. M. Voorhees & D. K. Harman
ture experiments might decompose the overall task(Eds.), The Eighth Text REtrieval Conference
into pieces, such as query composition or document(TREC-8). Gaithersburg, MD, USA.
selection. Likewise, there is a desire to base experiments on sound underlying models, such as those of The Proceedings of the Eighth Text REtrieval Conthe user, the task, and the typology of information ference will appear in print in mid-2000. Softcopies ofneeds. Future discussion will ensue on the track list- most papers are available now from the TREC website (trec.nist.gov).
serv (trec-int@ohsu.edu).
4