Evaluation of Web Document Retrieval:

A SIGIR'99 Workshop

Maristella Agosti and Massimo Melucci
Universita` di Padova
Dipartimento di Elettronica e Informatica
Via Gradenigo 6/A -- 35131 -- Padova -- Italy{agosti,melo}@dei.unipd.it
A Framework for the Workshop
In this decade theinformation retrieval research community
has
startednew relevant efforts toimprove previously available
evaluation techniques and methods and to address new ones. Evaluation
techniques are critical to rese archin all areas of science andengineering, so they
are for the
information retrieval area.
Theevaluationof information retrieval systems
can take different
approaches,
face different issues, and propose
or use different
methods, as it has been addressed by Harter and Hert in [1]. The twoorthogonal themes
they propose to consider
are the ``Different IR
Theoretical Perspectives'' and the ``Diversification and Hybridisationof IR Systems''. These two themes are compreh ensive of many aspects
and efforts that most of IR researchers believe are important for
conducting future IR evaluation efforts. This framework was proposedto the thirty-three people who attended the workshop. Fourteen people
were from United States, thirteen from Europe, three, two, and one
people respectively were from Korea, Japan and Australia.
Different IR Theoretical Perspectives.
The different
and alternative approaches need tobe consideredseparately,
because different c onceptual modelsof IR requiredifferent and alternativemodels of evaluation. Someof these
theoretical perspectives that have been put forward over timeandpresented in [1] are: the so called black -box model, where inputs areused to produce different outputs, and the evaluation approach isbasedon a Cranfield -like evaluation methodology; the interaction
model, where IR is considered as an interactive communication process;the approach that sees IR as creativity, explora tion, and discovery;
the approach that considers IR as navigation which is based both on
the network/hypertext
paradigm
together with thenetworked
environments; and the approach
where IR is considered as logicalinference.
Diversification and Hybridisation of IR Systems.
The second theme to be considered is the proliferation of systems that
often implement in an integrated way some of the different views andfunctions of IR, but which are not only IR systems. Some of those are:the World Wide Web, and a digital library. Both of them are systems
which include IR functions, but they are systems that have many other
different functions that are not traditionally the target or scope ofan IR system.

The Focus of the Workshop
Web Functions and User's Information Needs.
Some of the possible types of information need are: Retrieving aims tolocate relevant Web pages where emphasis may be placed either onrecall or precision. Finding aims to locate a unknown attribute usin ga known one. Mining aims to locate novel information, associations,patterns, orrules. Explorationis retrieving
information being
relevant
to an information need that may change at search time.
Extracting is transforming an unstructur ed set of unstructured Web
pages into a structured database of structured record.
By ``function'' we mean theservice being provided by a softwaresystem, such as a search engine, to meet a final user's informationneed. The main functions be ing performed within the Web context arebrowsing hyper-links, querying search engine indexes, and navigating
search tool directories. One of the aspects that characterizes theWeb isthatsearch engines are used
not only for retrieval
purposes. Indeed one may use search engines to look for a specificresource, e.g. an address or a business. In general, there is amany-to-many correspondence between function and information need.
Thus, an information need can be met throu gh one or more function, ora given function can meet more than one information need. This is oneof the issues of diversification and hybridisation of IR systems.
Web Data.
Another facet of diversification and hybridisation is that about thedata. The issues
aboutdata should be considered wheneveranevaluation framework is set up are listed in the following.
1 No collections -- we can hardly speak about collection for Web
document retrieval since the Web evolves rapidly and the evolution
cannot be controlled.
2 Data and structure -- in the Web context it is possible to findall different types of data, i.e. un -structured, semi-structured,or structured data.
3 Multi-lingual data -- though English is thewrite Web pages and to formulate queries,
are becoming more and more used.

mai n language used tonon -English languages

4 Multi-media data -- the Web contains different types of document
media, other that programs, such as Java applets.
5 Hyper-links -- the presence of links, though untyped 1makes Web
document retrieval different since relevant information are often
in linked documents.
6 Metadata -- metadata implement thesemantic interoperability
between heterogeneous data sources. They may be used to improve
retrieval effectiveness, but they are rarely used.

Web Technology.
Web technology and standards change. Among the others, we think thatthe technologies which are a characterization of the Web context are
the following ones.
* The Web browser
with its predefined browsing functionalities isultimately theunique interface between the final user and theWeb. Additional
functionalities are provided by
the searchengines and implemented using Web languages.
* Slow connection
can make
impossible an effective
browsing.
Indeed, to be effective, browsing or navigation should allow thefinal user to follow links rapidly in order to retrieve thedesired information.
* Web pages are mostly implemented asHTML documents, butnotonly. For example, XML or Java are often used to implement Webpages and HTML is going to be legacy.
Workshop Presentations
Nick Craswell, Peter Bailey and David Hawking asked themselves if ``Is
it fair to evaluate Web systems using TREC ad hoc methods''. In otherwords, the question they posed is if the Cranfield evaluation model isstill adequate for Web document retrieval. A preliminary ans wer tothis question was given by the authors at the beginning of their paper
by stressing that ``systems of TREC VLC Track participants were more
effective than live Web systems''. The authors provide a possiblereason for that -- Web and TREC
ad hoc systems solve
differentproblems. Then, they analyse into detailfour hypotheses which may
explain the difference in performance:* The importanceof hyper -links.
Web search engines
retrievehyper-documents,
while
TREC systems
retrieve``flat''documents. TREC ad hoc evaluation scheme do not consider links
between documents and a document is judged as non -relevant though
it is linked to relevant documents.
* Different topics. As stressed in Section 1, the authors highlightthat there are many information needs which reflect in different
topics being expressed as very
short sentences, perhaps without
any structure.
* Document quality. Within TREC a document
is judged about its
relevance to a topic without considering
its quality. On thecontrary, search engines can retrieve high quality non -relevant
documents,and miss low quality relevant documents if they use
quality as retrieval criterio n.
* Duplicates. The
presence of many copies of the same document
induce abias into retrieval
effectiveness measures. Indeed,systems that retrieve many copies of anon -relevant document
appear to be less effective.
Roberto Zamparelli
addressed one of theworkshop
topics whichconstitutes one of the issues mentioned in Section 1. The main idea issummarized in the title: ``Metadata Do It Yourself''.
Zamparelli
argued that automatic techniques to describe t he semantic content ofWeb pages have some limits.
Automatic extraction of keywords looses
many important information such as non -text data, document structure
andlayout,
and``details'' improving
thedocumentquality. Information propagation is sometimes used by the systems toincrease the evidence about the relevance of a document or a site if
the latter has many in -going links from another document or site. The
use of links to propagate evidence may lead to circularity -- the morea site is judged as relevant, the higher the number of in -going links,
the more thesite is judged as relevant. The author proposed a
semi-automatic tools supporting authors to add metadata to their own
documents which is based on the ite ration of manual editing supported
by automatic processes,
such as automatic
keyword extraction andrelevance feedback.
Annelise M. Pejtersen, Mark Dunlop and Raya Fidel presented ``A use
centred framework for evaluation of the Web'' for ide ntifying the``experimental boundary conditions''.
They identified some issues
making Web evaluation rather difficult -- lack of understanding on how
search engines work, user disorientation and cognitive overload in acomplex of links, the evoluti on of the Web may make user understanding
and learning more difficult,absence of functionalities to support
decision tasks which is then left to the user.
Mildrid Ljosland presented an ``evaluation of Web search engines andthe search for better ranking algorithms''. The paper was in two
parts: The firstpart reported on a case study consisting in twocomparisons. A comparison was among three search engines in order totest their effectiveness, while another comparison aimed to study thesize of the set of documents indexed by a search engine. The second
part was a review of the possible ways forimproving ranking
algorithms.
The last presented paper was by Monica Landoni and Steven Bell. Theypresented a``critical o verview'' on the ``information
retrievaltechniquesforevaluating search engines''.
They envisageacollaboration between the IR and the Web communities in order to takeadvantage of the synergy withthe experience in evaluation ofth eformer. The paper proposes a framework for evaluating search engines.
The starting point of the proposed guidelines is the assessment of thescenario, i.e.
of the context
within which evaluation has to becarried out. Then, it is necessary to se lect the measures being mostadequate to fulfil research objectives. Collecting information about
the searchtools is the next
step and aims to have data about
functionalities and databases. The experiment is then defined in order
to select queries, relevance model, judges and documents.
Some Reflections
The final part of the workshop was devoted to the discussion on thetopics arisen
from thepresentations. In
particular, comments
concentrated on the following issues expressing di verse positions:* The Cranfield model may be the most adequate one if user studies
are hard and expensive to do, and links are investigated in order
to employ theinformation they provided
about content andtopology. Indeed, hyp er-links have to be considered within any Web
document retrieval evaluation
model since they may
express
relevance relationships between documents.
* Morework should be done
to investigate what ``Web user's
information need'' means. The difficulty of understanding what a
Web user's information need is depends on the fact that there is
no average users. Logs can not always be used to infer what theuser wants.
* Web queries are rather different from th e TREC topics. The latterare well-formulated query expressing a well defined informationneed. The former are on the contrary ill defined expression ofill defined information needs.
Acknowledgments
The workshop organizers thank F red Gey for the help and the support,
and all the attendees for their active participation.
References[1] S.P Harter and C.A. Hert. Evaluation of information retrieval
systems: Approaches,
issues, and methods.
In Annual Review ofInformation Science and Technology (ARIST), volume 32, chapter 1,
pages 3--94. Information Today, Inc., Medford, NJ, 1997.