Challenges in Information Retrieval and Language Modeling
Report of a Workshop held at the Center for Intelligent Information Retrieval,
University of Massachusetts Amherst, September 2002
James Allan (editor), Jay Aslam, Nicholas Belkin, Chris Buckley, Jamie Callan, Bruce Croft (editor), Sue Dumais,
Norbert Fuhr, Donna Harman, David J. Harper, Djoerd Hiemstra, Thomas Hofmann, Eduard Hovy, Wessel Kraaij,
John Lafferty, Victor Lavrenko, David Lewis, Liz Liddy, R. Manmatha, Andrew McCallum, Jay Ponte, John Prager,
Dragomir Radev, Philip Resnik, Stephen Robertson, Roni Rosenfeld, Salim Roukos, Mark Sanderson, Rich Schwartz,
Amit Singhal, Alan Smeaton, Howard Turtle, Ellen Voorhees, Ralph Weischedel, Jinxi Xu, ChengXiang Zhai

Summary

1. Introduction

Information retrieval (IR) research has reached apoint where it is appropriate to assess progress andto define a research agenda for the next five to ten
years. This report summarizes a discussion of IR
research challenges that took place at a recent
workshop.

What is information retrieval? Early definitions,
dating from the 1960’s, emphasize the very generalnature of the task. For example, in Salton’s classic
textbook*:
Information retrieval is a field concerned with thestructure, analysis, organization, storage,
searching, and retrieval of information.

The attendees of the workshop consideredinformation retrieval research in a range of areaschosen to give broad coverage of topic areas thatengage information retrieval researchers. Those
areas are retrieval models, cross-lingual retrieval,
Web search, user modeling, filtering, topic detection
and tracking, classification, summarization, question
answering, metasearch, distributed retrieval,multimedia retrieval, information extraction, as wellas testbed requirements for future work. The
potential use of language modeling techniques inthese areas was also discussed.

In that textbook, information retrieval is assumed toalso include database systems and question
answering systems, and information is construed tomean documents, references, text passages, or facts.
Over the 1970’s and 1980’s, much of the research in
IR was focused on document retrieval, and theemphasis on this task in the Text Retrieval
Conference (TREC) evaluations of the 1990’s has
further reinforced the view that IR is synonymouswith document retrieval. Web search engines are, ofcourse, the most common example of this type of IR
system.

The workshop identified major challenges withineach of those areas. The following are recurring
themes that ran throughout:
•
•
•
•
•
•
•

The enormous increase in the amount of online text
available and the demand for access to different
types of information have, however, led to arenewed interest in a broad range of IR-related areasthat go beyond simple document retrieval, such asquestion answering, topic detection and tracking,summarization, multimedia retrieval (e.g., image,
video and music), software engineering, chemicaland biological informatics, text structuring, text
mining, and genomics. Salton’s general definition iseven more applicable now than it has been in the
past.

User and context sensitive retrieval
Multi-lingual and multi-media issues
Better target tasks
Improved objective evaluations
Substantially more labeled data
Greater variety of data sources
Improved formal models

Contextual retrieval and global information accesswere identified as particularly important long-term
challenges.

*

G. Salton, Automatic Information Organization and Retrieval,
McGraw-Hill, New York, 1968.

1

These factors, plus the resurgence of interest informal, statistical methods for language-related tasks
such as IR, make this an appropriate time to reassess
and more clearly define the IR research agenda. To
respond to this need, a group of IR and language
technology researchers met at the University of
Massachusetts Amherst to discuss IR researchchallenges. This report summarizes those
discussions. The workshop was the second in aseries funded by ARDA* that is focused on thelanguage modeling approach to IR. For thisworkshop, the first priority was to identify the
research challenges overall and the second was todiscuss how language modeling could help toaddress these challenges.

One common theme that has been used to
distinguish IR-related research from research indatabase systems is that the information that is
retrieved is derived from “unstructured” data
sources. In the past, this distinction has been veryclear, but if marked-up text is regarded as “semistructured” and in the domain of database systems,
then the boundary between the two areas becomesless obvious. Given the number of papers in recent
database conferences on nearest-neighbor andsimilarity search, distributed search, Web search,and information extraction, it seems apparent that IRand database systems now have many common
interests.

The discussion of IR challenges is broken down by
topic. In each topic area, there is a short overview of
the area followed by a list of “near-term” challenges(3-5 year timeframe). In some cases, there is also a
discussion of how language modeling could be used
and what resources would be required for progress.
The topic areas are: retrieval models; cross-lingual
retrieval; Web search; user modeling; filtering, TDT,
andclassification;summarization;questionanswering; metasearch and distributed retrieval;multimedia retrieval; information extraction; andtestbeds.

The huge success of Web search engines such as
Google might lead some to question the need forextensive IR research. There are a number ofpossible answers to this question, but here are somemajor points:-

Web search and IR are not equivalent. As
mentioned previously, IR encompasses many
types of information access. Web search is only
part (although an important part) of thisspectrum of information systems.

-

Web queries do not represent all informationneeds. A broad range of information accesstechnologies are being created to address thediversity of information needs of people indifferent contexts. If we focus only on thecurrent mix of queries in Web search engine
logs, many of those information needs will notbeaddressed.

-

Before discussing this detailed list of research issues,
we first consider some possible longer-term
challenges for a ten-year timeframe and beyond.
This type of challenge, although necessarily quite
abstract, is valuable for illustrating general research
directions and emphasizing the importance ofcollaborations.

Web search engines are effective for some typesof queries in some contexts. Retrieval
experiments in the TREC environment, andcommercial success, demonstrate that, for a very
popular type of query (find the right homepage), retrieving the pages containing all thequery words and then ranking them according toother features based on links, anchor text, URLs,
and HTML tags is very effective. For othertypes of queries, and in other environments (e.g.
corporate), this approach to ranking is lesssuccessful.
*

Advanced Research Development Agency.
http://www.ic-arda.org/

22. Long-Term Challenges

A definition of the second challenge is:In the discussions of longer-term challenges at theworkshop, two main themes emerged. These wereglobal information access and contextual retrieval.
A definition of the first challenge is:Contextual retrieval: Combine search technologies
and knowledge about query and user context into asingle framework in order to provide the most“appropriate” answer for a user’s informationneeds.

Global information access: Satisfy human
information needs through natural, efficient interaction with an automated system that leverages
world-wide structured and unstructured data in any
language.

In general, interactions with Web search engines
could be characterized as “one size fits all”. Thismeans that all queries are treated as simple Web
queries where the aim is to locate useful home pages
and the burden is placed on the user to scan andnavigate the retrieved material to find the answers.
There is no representation of user preferences,
search context, or the task context.

On the World Wide Web alone, digital information
is being accumulated at a phenomenal rate and in avast number of languages. By some estimates, thenumber of web pages written in Mandarin Chinese
will overtake those written in English in the not too
distant future. A “grand” challenge for IR will be todevelop massively distributed, multi-lingual retrieval
systems. Such systems would take as input an information need, encoded in any language, and returnrelevant results, encoded in any language. The
design of such a system would probably borrowfrom techniques in distributed retrieval, data fusion,and cross-lingual IR, but it is unlikely that simply
combining known techniques from these fields
would yield such a system. What would thearchitecture of such a system be? What data sets
could be used to develop and test such a system?

Despite some recent attention to this problem, little
progress has been made due to the difficulty ofcapturing and representing knowledge about users,
context, and tasks in a general Web search
environment. Future search engines should be able
to use context and query features to infercharacteristics of the information need such as query
type, answer type, and answer level, and use thesecharacteristics in retrieval models to rank potential
answers such as sentences, passages, documents, or
combinations of documents.
An example of contextual retrieval would be acontext aware, transmedia IR system. With such asystem, if a user enters a query such as “Taj Mahal”,
and if the user is at a desktop computer and has
spent time earlier planning a conference trip to India(reading emails on the trip; examining travel web
pages; placing entries in a diary), then the system
will be aware of this context and will be more
inclined to retrieve pictures and videos of the Indianmausoleum, while music from the jazz band with thesame name would be less likely to be retrieved. The
ranked output would contain images indexed by thequery text, and images similar in visual content to
the retrieved text-tagged images. The highbandwidth of the machine's connection would also
be taken into account when choosing images orvideos to be delivered. Indeed, the bandwidth may
even be a factor in the retrieval algorithm. On theother hand, if a user were starting out on a long car
trip (the system being aware of this context fromdiaries and location sensors) and the user has often

Wireless devices are being rapidly developed which
can spontaneously form ad hoc peer-to-peer networks. A world in which such devices are ubiquitous
gives rise to a number of challenges for IR. For
example, one might fancifully imagine flying to an
unknown city for a business trip, renting a car,opening one’s wireless PDA, and connecting to an
ad hoc peer-to-peer network in order to submit
requests for information such as: (1) background on
local points of interest, (2) reviews of local
restaurants, and (3) “dossiers” on people who arelikely to be encountered.. How would suchinformation needs be described? (Free text,
structured queries, etc.) To whom would such
requests be sent? How would the results obtained be
combined and presented to the user?

3

requested audio in such a situation, then the musicfrom the jazz band will be more likely to be
retrieved. It is this context awareness, making use ofknowledge about the user, the current task, thehistory and future plans, location and device, and thefact that retrieval can be of any kind of digital
information, regardless of query type or mode,
which makes this a long-term challenge.

their information needs and the context in which theretrieval task is embedded. The reliance on formal
retrieval models is one of the great strengths of IR
research.
Information retrieval encompasses a broad range ofcomplex information seeking tasks and current
retrieval models capture only a small part of thatcomplexity. Current models of text, for example,
deal with relatively simple aspects of language(words, phrases, names) and do not capture aspectsof linguistic structure that may be important tospecific retrieval tasks. Similarly, current models of
the user are very weak or, often, missing fromcurrent retrieval models. Current retrieval models
generally focus on a narrow range of retrieval tasks(e.g., ad hoc retrieval and filtering) while ignoringother important tasks (e.g., browsing, known-item
retrieval, questions answering, summarization).

Other types of challenges can involve contextual
information. Examples are:
1. Developing intelligent classification algorithms
that will be able to unobtrusively elicit user
feedback, combine it with contextual andhistorical evidence, and produce effective
structured annotation of new data.
2. Summarizing content from a number of textual
and semi structured sources, including databases
and web pages, in the right way (language,
format, size, time) for a specific user, given atask and the user profile.

Research aimed at developing more comprehensive
retrieval models is critical. The goal of a single
comprehensive model of retrieval is beyond ourcurrent level of understanding, but models that better
describe individual tasks or a range of related tasks
are possible and necessary to progress in the field.

In addition to these two major themes, it is clear thatmany of the major research opportunities of the
future will involve collaborations between fields.
For example, IR and database collaborations will be
crucial for the development of probabilistic models
for integrating unstructured, semi-structured andstructured data, for the design of effective
distributed, heterogeneous information systems, andpossibly for systems that detect new and interesting
information in sensor-based streams. Collaborations
with natural language processing and machinelearning researchers will be needed for advanced
question answering systems, and, of course, there aremany exciting opportunities in collaborations withthe sciences (e.g. bio-, geo-, astro-, ...).

Near-Term Challenges
1. Models that incorporate the evolving
information needs of users performing realistic
tasks and the utility of information as well as
topicality (relevance is more than topicality).

3.1 Retrieval Models

2. More advanced models that, for example,
estimate translation models or compute subtopic models through clustering, currently
require significantly more computation than ispractical for deployment in many applications.
Significant advances in the underlying
estimation and inference algorithms will makeprobabilistic models much more attractive foruse in large scale systems.

Formal retrieval models have formed the basis of IR
research since the early 1960’s. A number ofdifferent models have been developed to describe
aspects of the retrieval task: document content andstructure, inter-document linkage, queries, users;

3. Generalization of current techniques to information sources that may be significantly different
from today’s document collections
4. Models and tools for incorporating multiple
sources of evidence (text, queries, relevance

3. Topic Discussions

4

judgments, user context, annotations etc.).

6. Full elucidation of the relationships between thevarious modeling approaches.

CLIR has very recently achieved one major
milestone:cross-lingual document retrieval
performs essentially as accurately as monolingual
retrieval. This has been demonstrated in formal
evaluations in the TREC in 2000, where English
queries were run against a Chinese corpus, and2001, where English queries were against an Arabic
corpus and CLEF* 2001, with French queries against
an English corpus.

Resource Requirements

Near-Term Challenges

1. More refined taxonomy of search tasks and task
models.

1. Effective user functionality. The technology has
been evaluated formally for accuracy inreturning lists of documents ranked by estimatedrelevance to the user’s need. The next steps for
effective user functionality are to incorporate
effective user feedback about their informationneed and to provide readable translations of(parts of) the retrieved documents to support
document selection. Systems should also
provide better support for query formulation andreformulation based on some set of intermediatesearchresults.

5. Intrinsic and extrinsic evaluation techniques thatmeasure the performance of retrieval and related
technologies in the context of larger tasks, andbased principled error analysis to go beyond‘what’ and address ‘why.’

2. Data, lots of data (documents, queries, relevancejudgments, user behavior, task-related data).
Language Modeling
Retrieval models that incorporate language modelingtechniques have produced promising results over thelast four years. Simple language models have beenshown to incorporate document and collection
statistics in a more systematic way than earlier tf.idf
based techniques. Language models work as well as
the classical models using tf.idf, but further
improvements are likely to require a broad range oftechniques in addition to language modeling. The
essence of the language modeling approach, which isshared with more classical probabilistic approaches
to IR, is that probabilistic modeling is taken to be theprimary scientific tool. At present, this appears to be
the most promising framework for advancing
information retrieval to meet future challenges
presented by more diverse data sources andadvanced retrieval tasks

2. New, more complex applications. We can apply
the technology to the next level of cross-lingual
challenges. Cross-lingual (factoid) question
answering would allow posing factoid questions(e.g., asking who, what organization, when, or
where) in English and receive answers in
English, based on documents (in English or
another language) containing the answer. Crosslingual gisting/summarization would provide
short English summaries of documents inanother language.
3. Languages with sparse data. The technology
has been developed and proved in the context oflanguages with many speakers, e.g., English,
Spanish, French, German, Chinese, and Arabic.
One challenge now is developing ways to very
quickly (a few weeks) and inexpensively (a few
person-weeks) find/create data for languages
where resources are minimal today (for example,
Farsi, Pashto, Hindi, or any of a large number of

3.2 Cross-Lingual Information Retrieval (CLIR)
Though initially the Web was dominated by English
speakers, now less than half of existing web pages
are in English. Accessing information in a host oflanguages is clearly important for many uses.
In monolingual retrieval, the queries are in the samelanguage as the collection being accessed. The
purpose of CLIR is to support queries in one
language against a collection in other languages.

*

Cross-Language Evaluation Forum held in Europe
http://www.clef-campaign.org/

5possibilities).

For the new challenges above, the following seem
crucial:

4. Massive improvements in monolingual retrieval
based on learning semantic relationships fromparallel and comparable corpora. The lessons
learned from CLIR suggest new ways toapproach monolingual retrieval. Given the
successes to date in CLIR, any improvements inmonolingual
retrievalshouldgeneratecomparable improvements across languages andvice versa

1. An effective, inexpensive way to collect far
more (100 times more) queries and results(answers or relevant documents)
2. Appropriate resources for evaluations of CLIR
with low density languages
3. Testbeds (see Testbeds section) for delivering
effective functionality to users

5. Merging retrieval result lists from databases in
multiple languages. This problem is related tothe meta-search problem, but has the additional
challenge that the statistics underlying the
ranking functions are built on different
vocabularies and thus not directly comparable.

Language Modeling
In addition to the language techniques used formonolingual retrieval, CLIR has capitalized on aprobabilistic translation model—e.g., a mixture
model of words from general language usage, andthe product of probabilities of a term coming from adocument and of that term being translated to aquery term. The techniques use a mix of manual
bilingual dictionaries and statistical bilingual
dictionaries automatically derived from corpora oftranslated documents. It is interesting that machinetranslation systems for the document language are
not a prerequisite.

6. More tightly integrated models for CLIR. Most
current approaches to CLIR are based on ashallow integration between translation models
and retrieval models, where independence isassumed for both components. A promising
direction seems to be to explore models withless strict independence assumptions, enabling
the use of context information for translation
directly.

3.3 Web Search

7. More training data.
Language modelingdepends on training data. Today there areseveral hundreds of pairs of query and relevant
documents. For factoid questions, there areroughly 1000 queries with correct answers. It is
generally agreed that developing those materials
over the last 10 years has spurred muchproductive research and led to major
improvements in IR. A new challenge is todevelop ways to collect a hundred times moretraining data for language modeling—e.g.,
100,000 pairs of query and relevant
documents/answers rather than a few hundred.

Search engines for the Web are one of the mostpublicly visible realizations of information retrieval
technology. For some search tasks (e.g., home page
finding), systems such as Google provide highly
accurate results. However, the Web contains morethan just home pages and searchers are interested in
more than just finding single pages. Significant challenges remain for improving general Web search.
Near-Term Challenges
1. Web Structure. The challenges are to define theactual document collection that we call the Web
and to understand how the uniqueness of the
Web collection structure affects retrieval
methodology.

Resource Requirements
Major contributors to progress in CLIR have beenformal evaluations and data that have been madeavailable through TREC, CLEF, and the NIINACSIS Test Collection for IR Systems (NCCTR).

- What is a "document" that can be retrieved
and deemed as relevant to a user's information
need? It could be a single Web page, a group6

of related Web pages (a Web site), a paththrough the Web, or a portal to a hidden
database (e.g., Amazon). A different view ordifferent retrievable “documents” might imply
different retrieval algorithms. For example, if
we take a whole Web site as a document, thenwe would need to measure the relevance of thewhole Web site, rather than a single page.

3. Searching. The challenge is to develop methods
for exploiting all evidence, including the web
structure, meta-data, user context information, tofind high quality information for users.
- How can we capture and then exploit user
information and query context to better
understand a user's information need? What
aspects of a user's context and history can we
capture? Can we exploit language models torepresent information need more precisely?
How can we formally model the interactive
retrieval process? How do we deal withvariations of queries? How can we identify
different types of queries and utilize the factthat specific methods work well for specificquery types?

- What is the boundary of the Web collection?
Should the collection include dynamicallygenerated web pages? How is the global,
public web different from corporate intranetsand extranets? The differences may imply
different approaches to IR applications on
each.
- How does the hypertext structure, which
distinguishes the Web from a traditional
document collection, affect the evaluation
methodology and retrieval models? For
example,theindependent
relevance
assumption is clearly violated due to the links;a page pointing to a relevant document can be
regarded partially relevant, even if it is not
relevant by itself.

- How do we represent a web document? How
do we annotate web pages (“generating new
evidence”)
automaticallyorsemiautomatically?
- The Web is an ideal environment fordeveloping formal retrieval models that
support multiple forms of evidence—e.g.,multiple types of document representation, useof document structure, etc. The community
has talked about this, but never really done it(except possibly for the inference networkmodel). How can statistical language models
be combined with other structural evidence?
How can domain/linguistic/context knowledge
be integrated with statistical methods?

2. Crawling and Indexing. The challenge is todevelop an architecture for information accessthat can ensure freshness and coverage of
information in a rapidly growing web.
- It is especially challenging to maintainfreshness and coverage in a centralized searchengine. The current approach is to havedifferent re-visit frequencies for different
types of pages/websites. There is somethinginherently wrong with waiting for a crawler tocome around and pick up your new content
before it can be “found” by people and as the
web grows the issues of freshness will get
worse.

- There is a growing “semantic Web” effort
based on widespread use of controlled
vocabulary metadata to improve the accuracy
and consistency of Web information services.
The IR community has a long history of
studying controlled vocabulary metadata,
especially how to assign it automatically, how
to evaluate its quality, how to cope withmissing or inconsistent metadata, and how to
map among different vocabularies orontologies. The IR community needs to takesemantic Web efforts more seriously andcontribute its expertise to controlling the hypeand solving the problems surrounding this- Would other alternative search architecture
help maintain freshness and coverage? For
example, would a distributed searcharchitecture have an advantage here? Would
topic-specific dynamic crawling be useful?

7issue.

Additionally, some information retrieval tasks are
ill-defined without taking the user into account:summarization is meaningful only in the context of aparticular task and/or class of users. Other tasks,
such as cross-lingual document retrieval, appear byclassic (not user oriented) measures to be quite
effective, but no one knows how such systems
would be used by a person.

- How can we measure “trust” of information
sources and perhaps incorporate this trust
factor into retrieval? How can we trust the
search engine vendors not to take advantage of
their powerful position in providing editorial
influence on the material we access from theweb?

We claim that greater focus on the user will enablemajor
advancesininformation
retrievaltechnologies, perhaps dwarfing those made possible
by better core algorithms. The focus may involve
better modeling of the user's background and type,
personalization, the context in which information
access occurs, the interactive information seeking
process, explanation of what happened, resultspresentation, or combinations of all of those.

4. Data Collection and Evaluation. The challenge
is to develop a shared platform for gathering
appropriate data and to design evaluation
methods that reflect realistic characteristics of
web environment.
- What can, and should, we log from Web
searching in order to capture user context, how
can we provide this log data as a public utility
for researchers and what are the architectural
implications for logging such data? What does
this mean in terms of privacy issues?

Near-Term Challenges
1. Shared data-gathering environment. It is very
difficult to gather information about users
completely enough that it can be used inexperiments. The process is expensive, labor
intensive, and requires expertise that is notuniversal in the information retrieval
community. It requires constructing a user's
working environment that tightly integrates
information retrieval.

- The collection is changing constantly, we
cannot freeze a copy, and the linkage
characteristics are changing over time, so how
can we evaluate on this “moving target”?
3.4 User Modeling
Much active research in information retrieval is
carried out by abstracting the user away from the
problem: judgments are captured and held constant,non-binary relevance is ignored as too complex,
evaluation is on a single round of query-results
without opportunity to adjust the query, and so on.
This abstraction has been incredibly successful inenabling research to advance rapidly, creating more
effective and efficient systems for retrieving andorganizing information.

The community should support one or moregroups in creating such a data-gatheringlaboratory, defining what types of informationneed to be acquired (interaction histories,
annotations, explicit preferences, etc), andsharing all results. This process would becooperative among a large number of sites. As aresult, it will not be necessary for every group togather these types of data in order to do
experiments. The cost would be amortized overmany experiments.

However, those improvements in retrieval accuracy
appear to have dwindled in the past half dozen years.
It may be that one reason researchers are unable toadvance beyond the current plateau is that theevaluation model forces systems toward user-genericapproaches that are “good enough” for everyone,
and therefore “never great” for anyone.

2. Testbed of IR interactions. One or more
standard experimental data sets need to becreated that are heavily annotated with information about users, context, etc.
Such aresearch collection should allow evaluations that
focus on the user in more detail, but in a waythat does not require expensive user studies.

8Such a dataset would allow for some degree ofcomparability and comparison between resultsfrom different groups.

retrieval system by automatic disambiguation oruser-focused query expansion.
3.5 Filtering, TDT, and classification

3. Evaluation successfully incorporating the user.
The community needs to develop an evaluation
methodology
that
explicitly
leverages
information about the user and the context of theinformation retrieval process.
Such anevaluation would have the goal of dramatically
improving effectiveness for a particular user
rather than providing merely satisfactory results
for an unknown user.

Semi-structured text and media objects, created andmanaged by users, form an increasingly large
proportion of online data. Information accesstechniques that require schemata at data-entry time
are not appropriate for this data. Semantics must be
imposed after the fact, in a dynamic task context,
and often under the guidance of a nonprofessionaluser. The imposed semantics may be as simple as“show me/don't show me,” as in topic tracking,message routing, and content filtering systems. Or it
may be as complex as sorting documents into ahierarchical taxonomy, or producing richly linked
hypertext. Systems for topic detection, terminology
discovery,
namedentityextraction,andunsupervised learning find novel semantic
regularities across media types and time-varying
data, aiding users in imposing semantics.

4. Privacy. Support for rich modeling of the userthat also preserves privacy.
5. Extended user models. Many, if not all of theother areas discussed in this report explicitly
demand or implicitly require much moresophisticated models of users than are presently
constructed for any IR purposes. There is
therefore a significant need for research in how
to include characteristics of such elements as
users' goals, tasks, and contexts in user models,
as well as how to make use of this knowledge inaffecting IR system design and response.

Some of these technologies have been adopted
widely, for automated metadata assignment inknowledge management systems, routing of email
and phone requests for help in customer relationshipmanagement, categorization of trouble tickets to
support data mining for process improvement, andmany others. Despite this commercial penetration,
only a tiny fraction of text data is ever classified, andmost applications require setup and maintenance bycomputing professionals.

6. Long term user models. As it becomes possible
to gather and store more historical information
about a user’s interactions, we need to develop
models that span tasks and applications. Such amodel would be in contrast to the short-term,
single-task models that are more common to
date.

Near-Term Challenges

Language Modeling

1. User Models. Supervised machine learning has
allowed the production of classifiers frommanually labeled example documents, reducing
the need for rule writing by specialists.
Unfortunately, the emphasis in machine learning
research on a tabula rasa approach means that aburdensome number of documents often must belabeled. Better ways to elicit and incorporate
user knowledge in machine learning algorithms,
to leverage domain-specific databases, and togather implicit feedback from user actions, are
all needed to reduce the demands for labeling.
Promising new approaches to leveraging
unlabeled data, including pseudo-feedback, co-Very little has been done that explicitly uses
language models to capture the user. Arguably, pastwork modeling judged documents for relevance
feedback is a form of language modeling, but theconnection is tenuous.
One hope for language models is to represent theuser by a probability distribution of interests (words
and phrases), actions (information seeking and use
behavior), and annotations (judgments).
This
information, if appropriately captured, mightplausibly be used to improve the accuracy of a9training, active learning, and transduction, need
to be made more predictable in their effect.

necessitates development of conceptually new
approaches to the problem.

In addition to algorithmic progress, we need abetter understanding of inherent properties ofclassification. How do, and how can, users andcommunities of users classify documents to
support their tasks? What properties do sets ofclasses (whether informal shared interests or
structured hierarchical taxonomies) have, andhow are they connected with latent or link
structures inherent in the text data itself.

3.6 Summarization
Text Summarization is an active field of research inboth the IR and NLP communities. Summarization isimportant for IR since it is a means to provide accessto large repositories of data in an efficient way. It
shares some basic techniques with indexing, sinceboth indexing and summarization are concerned withidentifying the essence of a document. High quality
summarization
requiressophisticated
NLP
techniques in addition, which are normally notstudied in IR. In particular, for domains in which the
aspects of interest can be pre-specified,summarization looks very much like Information
Extraction. Summarization is therefore a goodchallenge problem to bring together techniques fromdifferent areas.

2. Semi-structured data. The fastest growing area
in text classification is in classifying real world
entities by associated text: customers by theircomments, broken devices by partially textual
repair reports, students by their essay exams,
and so on. Most such applications to date rely
on a known connection between text and realworld entity. Broadening the range of suchapplications requires systems that can detect
those connections as well (“hardening” the text
database), and in particular detecting textual
records from multiple sources are associatedwith the same entity. Techniques such as link
analysis and information extraction, long ofinterest in the intelligence community, need to
be improved and extended to a wide range oftext mining applications.

In comparison with IR, the field of summarization
suffers from the difficulty of defining a wellspecified and manageable task. Since truly reusable
resources like the TREC test collections did not exist
for summarization, it was hard to measure progress.
Importantly, the high amount of variance acrosshumansummaries
complicatesevaluations.
Improved, more tightly specified tasks are currently
being developed within the DUC program.

3. Novelty Detection. Detection of novel data isquickly gaining popularity as a necessary
complement to real-world filtering systems. The
problem has received relatively little attention,
but the exploding growth of available
information makes redundancy a very real
obstacle to a satisfactory user experience.

Near-Term Challenges
1. Define clearly specified summarization task(s)in an IR setting. Examples: headline-lengthsummarization, topic-based summarization.
2. Move to a new genre, since producing text
summaries is almost trivial for newswire andnewspaper documents.

Existing novelty detection systems are based
primarily on pairwise comparison of potentiallynovel items to some form of “history,”
representing classes of items already known tothe user. Error rates of such systems can beexactly predicted from the effectiveness of thecomparison function, and this error rate rapidly
increases with the size and richness of history.
This makes novelty detection very challenging
in large-scale filtering environments and3. Move beyond extractive summarization.
Extractive summaries are clearly sub-optimal
with respect to obtainable compression rate andoverall coherence.
4. Integrate the user’s prior knowledge intomodels. Users do not want in summaries
material they know already.

105. Specification of a clear task. Suggested tasks
have included query biased summaries, news
clustering summaries, browsing tools for finding
relevant passages, and summarizing updates tothe news.

3.7 Question Answering
A Question Answering (QA) system takes as input anatural language question and a source collection,and produces a targeted, contextualized natural
language answer. To build the answer it gathers
relevant data, summary statistics, and relations fromthe sources (which might be structured or
unstructured), fuses or summarizes them into asingle answer as appropriate, and also gathers
information about the epistemic status of the answer(including
its
reliability,
recency,
factuality/hypotheticality, etc.). Building answers
might also involve a dialog with the questioner toobtain clarification or otherwise refine the question
and answer. Specifically, QA:

6. Development of an evaluation measure. The
BLEU measure developed for machinetranslation may provide inspiration. It has beensuccessfully applied to headline-style summaries
of documents.
Resource Requirements
1. Improved models that capture a user’s background knowledge
2. Large data sets with example summaries(preferably outside news domain)–e.g., thecreation of a SummBank, similar to the one
created at the JHU summer workshop in 2001,
but significantly larger. Ideally the SummBank
would contain summaries of the same texts
created from different perspectives, at different
compressions (from 30% down to headline
only), and in different styles (fluent, telegraphic,bullet points, etc.).

-

includes questions with short, factual answers(“factoids”)-

includes questions with longer natural language
answers(passages
andmulti-passage
summaries)-

includes answers that involve some data miningor gathering of summary statistics, possibly
fused from multiple sources; summary statistics
are operations such as maximum, mean,clustering, conjunctions and joins (thusincluding some aspects of what databases
provide)-excludes answers that require complex reasoning
with models that require no data or informationfrom external sources (thus not including
mathematical story problems or solving college
physics exam questions)-excludes taking actions (not “make a flightreservation for me” or “what is the translation ofthis document”, but does include “what is theearliest flight from LAX to LGA?”)

Language Modeling
Language Modeling has successfully been applied
for content selection, compression of sentences anddocuments, generation of headlines and reverseengineering the cut-and-paste process applied byhuman summarizers. Since it is highly unlikely thatsummarization can do without modeling higher
order structure, integrating linguistic intuitions into
probabilistic models poses a particular challenge.
Also the scalability of increasingly complex models
is an important issue for working systems providing
on-line summarization.

Near-Term Challenges
1. Improve performance of “factoid” QA to thepoint that the general public would find it
reliable and useful.

112. Create systems that provide richer answers(beyond factoids):- Answers that require summary statistics. For
example, use the web pages of a million
companies to answer the question “Whichstate in the country had the largest increase inhigh-tech job openings last month?”
Necessarycapabilities
might
includeextracting from formatted and tabular data,
mining structured data sources, creating
structured data from unstructured sources,ability to select the appropriate summary
statistical operations, and generating natural
language answers from structured data.

- Provide epistemic status of answer (including
is it a fact/opinion/hypothesis; at which times
was it a valid answer; what is thetrustworthiness of the answer and thereliability of the source; etc.) The systemshould provide its confidence in its own
analysisandanswer.
- Support interactive QA (dialogue) to clarify
and explore the topic (here there may not be
only one answer but a series of answers, eachleading to the next). Systems should be able
to detect when no answer is possible or when
the answer is not to be found in the resourcesat the system's disposal; thus the systemshould know when to reply "no answer
available," "no answer possible," or "questionis inconsistent". This relates to other dialoguesystems.

- Longer answers that have structure to them.
This includes answers ranging from those withfairly standard structure (biographies, eventstories, etc.) through those with some structure(causal accounts “What are the causes of the
Korean war?”), pro/con advice (“should I takehormone replacement therapy?”) to ones thathave very little structure as in general-purpose
descriptions of events and resources (“What is
this movie about?”) and objects (“Describe atelephone”). Solutions might involve using alanguage model and discourse model that
depends on the question.

- Take into account user models and session
models, and keep the models updated. Don't
tell the questioners what they already know.
Remember what you have told them and don't
repeat. Give them answers in their preferredstyle.
Handle sequences of contextuallyinterrelated questions and responsive answersof varying complexity that need to beunderstood within a larger model of the user's
information needs/requirements.

3. Create systems that leverage richer data sources:- Give answers that require integrating multiple
passages, and multiple data sources.

5. Develop better evaluation methodologies thatare scientific and repeatable.
Is there an"optimal" answer? What does "optimal" mean?

- Give answers that require integrating
structured and unstructured data sources,
including databases, semi-structured text andnumerical content. This might include theability to extract the desired capital city from atable of countries with capitals. This work
could be used to build (semi)structured
knowledge bases.

Language Modeling
Current statistical/language models are provenuseful for at least the first part of QA - locating
documents/passages with candidate answers. For thesecond part, the accuracy required in pinpointing
exact answers (for both factoids and for longer
answers that weave together factoids) demands morethan current language models can support. One
suggestion is to extend language models to include
more structured patterns (like information extraction
template patterns). (Most current work use handtailored special-purpose pattern matching for dates,
amounts, abbreviations, etc.)- When using dynamic sources, track how ananswer changes over time. This relates directly
to novelty detection.
4. Create systems that provide richer interaction
with the human questioner, and also provide
increased transparency.

12We envision a unified statistical approach in whichto accurately integrate evidence, find the rightanswer, and emit it in the best natural language.
3.8 Metasearch and Distributed Retrieval

1. Can a standard resource descriptor be devised
such that if a resource published its descriptor(e.g., on the Web), it could participate in a
generic distributed retrieval system? What data
must be present in such a descriptor? A language
model of the underlying data set? A semantic
description of the content? A model of therequired query syntax which permitsinteroperability? Could it be useful both forresource selection and metasearch data fusion?

Metasearch is the process of retrieving andcombining information from multiple sources, and it
is typically studied in one of two forms: (1) data
fusion, the combination of information frommultiple sources that index an effectively common
data set and (2) collection fusion or distributed
retrieval, the combination of information frommultiple sources that index effectively disjoint data
sets. As more and more retrieval mechanisms
become available over common data sets (e.g., theWeb) and specialized data sets (e.g., medical andlaw libraries), the process of identifying likely
sources of relevant information, retrieving
information from those sources, and effectively
combining the information thus gathered will only
grow in importance. A future wherein ubiquitous
mobile wireless devices exist, capable of forming ad
hoc peer-to-peer networks and submitting andfielding requests for information, gives rise to a newhost of challenges and potential rewards.

2. The performance of language modelingtechniques is, at present, on par with that of ad
hoc techniques. Can a theoretically groundedmodel of distributed IR be developed which
consistently outperforms ad hoc techniques?
3. The performance of distributed IR techniques isapproaching that of a “single database,” at least
within research environments and whenevaluated for high precision. Can this beachieved in practice? Furthermore, through thejudicious use of resource selection, distributed
IR should, in theory, outperform a “single
database.” Can this be achieved?
4. The performance of search engines varies fromquery to query. The goal of metasearch is often
to outperform the (a priori unknown) best
underlying search engine on a per query basis,and this can typically be achieved whencombining systems of similar performance.
However, this goal is often unachieved whencombining search engines of widely varying
levels of performance. Can a metasearchtechnique be developed which consistently
outperforms the best underlying search engine?
Or can a technique be developed which iscapable of distinguishing the “good” underlying
systems from “bad” on a per query basis?

The issues typically addressed by a distributed
retrieval system include resource description,
resource ranking, resource selection, searching, andmerging of results.
Many of the techniquesdeveloped to address these issues are ad hoc innature, though language modeling techniques havebeen successfully employed to address resource
description, ranking and selection.
Classic techniques for metasearch data fusion
typically address the following issues: (1) relevance
score normalization to a comparable space and (2)normalized score combination to obtain a final scorefrom which a final ranking may be obtained. Other
data fusion techniques that have been developed
include (1) modeling the problem as a multicandidate election and employing rank-aggregation
algorithms from Social Choice Theory (e.g., theBorda Count and Condorcet methods) and (2)various supervised learning techniques (e.g.,boosting, cranking, naive Bayes, etc.).

5. Techniques for data fusion typically assume(and often implicitly require) that the underlying
search engines index an effectively common
data set; techniques for distributed IR typicallyassume that the underlying search engines index
effectively disjoint data sets. Can techniques bedeveloped which effectively combine the resultsof underlying search engines that index data sets
of varying and unknown overlap? (Search

Near-Term Challenges

13

engines on the Web fall within this category.)
Can the metasearch problem be modeled in a
unified way such that data fusion and collection
fusion are merely two extremes of a single
underlying problem?

media objects. Automatic annotation may be one
way of associating text with such media objects.
This may involve learning from examples how
text and media features are associated and thenusing that information for annotation.
2. The context of user activity when a media object
is created may provide a good approach toindexing: for example, emails, diary entries or
location-based information associated with thetime that a photo was taken could be examinedfor pertinent text. Context at retrieval time will
also be important: the location of the searcher (aparticular country; a particular part of an office)or the type of device a user is using will informthe type or content of objects to be retrieved.

3.9 Multimedia Retrieval
Devices for creating, storing, and transmitting
multimedia information are increasing in prevalence
and capacity, and are decreasing in price. With little
prospect of such changes slowing in the foreseeable
future, it is not hard to predict with some confidence
that content-based access to multimedia information(indexing, retrieval, browsing, summarization, etc.),
is set to become a significant problem for web andenterprise search as well as for personal retrieval.

3. One of the major reasons for the successes of(text) IR has been the application of formal
models for indexing and retrieval. Applying
such models to the multimedia domain has beenchallenging partly because the features which
are often most useful do not easily lendthemselves to indexing. The extraction of
appropriate features which can be used forindexing is a challenge. So is the application offormal IR models to existing features.
Multimedia (image, video and audio) IR tasksneed to be formulated which cannot be solved
using text alone but will require advances infinding such features and advances in applying
formal IR models to such tasks.

The problem space of this topic is large because thetypes of objects to be retrieved are varied ranging
from collections of audio (e.g., speech, music),
images (e.g., photographs, clip art, scanned
documents), video (e.g., TV, camcorder, or security
camera output), as well as less common objects (e.g.,vector drawings, recorded pen strokes, and VRML).
The methods available for indexing and retrieving
such objects vary significantly depending on theirtype, which has a strong impact on the forms of
retrieval possible; however, it is also clear that theforms of retrieval users will wish to conduct willvary for each media type.
Near-Term Challenges
The current challenge in multimedia retrieval centerson indexing: given a non-text media object, thefollowing options are available. Text may be
associated with the object (e.g. captions, juxtaposedtext); part of the object might be convertable to text
(e.g. through speech recognition or OCR); metadatamight be assigned manually or media specificfeatures might be extractable.

4. We also need to think of different kinds of tasks
which involve data mining and retrieval of time
sequences of images, video and audio from otherdomains. Example: A few years ago there was apaper on collecting time sequenced images ofstorms from radars and then retrieving similarstorms from a database of time sequenced
images of storms to predict the future track of
the storm. This is basically a natural applicationof video retrieval applied to this task.

1. Extracting good indexing features from mostforms of multimedia is hard (except within
restricted domains). As an alternative, fragmentsof text or pieces of metadata may be located andused for indexing. For certain media types, text
may be easy to come by, for others, however,
little or no text will be directly associated with

5. To deal effectively with multimedia retrieval,
one must be able to handle multiple query anddocument modalities. In video, for example,
moving images, speech, music, audio, and text
(closed captions) can all contribute to effective

14

retrieval. Integrating the different modalities inprincipled ways is a challenge.

Information extraction is at the heart of much of the
anticipated progress in many fields. Question
answering, novelty detection, cross-lingual retrieval,and summarization all hope to leverage IR to
improve their effectiveness. IE is also viewed as a
database-filling technique that serves as a first step
toward data mining and other decision support
systems.

Resource Requirements
In order to assess research effectively in multimedia
retrieval, task-related standardized databases on
which different groups can apply their algorithms
are needed. In text retrieval, it has been relatively
straightforward to obtain large collections of old
newspaper texts because the copyright owners do
not see the raw text being of much value, however
image, video, and speech libraries do see great value
in their collections and consequently are much morecautious in releasing their content. While it is not a
research challenge, obtaining large multimedia
collections for widespread evaluation exercises is apractical and important step that needs to beaddressed. We suggest that task related image andvideo databases with appropriate relevancejudgments be included and made available to groups
for research purposes as is done with TREC. Usefulvideo collections could include news video (inmultiple languages), collections of personal videos
and possibly movie collections. Image collections
would include image databases (maybe on specifictopics) along with annotated text - the use of library
image collections should also be explored.

Near-Term Challenges:
1. Sufficiently high accuracy of traditional entity
extraction that it can be straightforwardly
incorporated into systems that consume it.
Current accuracy in the low 90s percent may
possibly be sufficient for use in isolation, but is
not sufficient when these results are combined
into n-ary relations, or incorporated into
question answering, novelty detection, or data
mining—all of which cause errors to compound.
2. Ability to extract literal meaning from text. This
could also be called automated proposition
extraction. The University of Pennsylvania,
BBN and New York University are creating a“PropBank”, analogous to the “TreeBank” forparsing. We should have as a goal to enable
automated “PropBank”ing of new text. This
would be a major step toward automatedcomputer “understanding” of language.

Language modeling
The application of information retrieval and otherstatistical machine learning techniques, analogous tolanguage modeling, may be useful in multimedia
retrieval. Language modeling has been successful intext related areas like speech, optical character
recognition and information retrieval. There is someevidence that some of these models may be useful inautomatic annotation, combining image and text
retrieval and image segmentation.

3. Large-scale reference matching.
In many
important applications there are many thousands
or even millions of entity strings to be deduplicated and matched. Performing this task
efficiently and accurately requires new
approaches and techniques.
Also, manyreferences (such as “police”) are generic andambiguous. New representations are needed thatwill facilitate top-down and knowledge-based
disambiguation, or useful consumption ofrepresentations that preserve the ambiguity.

3.10 Information Extraction

4. Cross-lingual information extraction. This includes the ability to combine evidence frompassages in multiple languages when filling a
database slot. (Both a British and a Jordanian
newspaper mention an event; use them both tomore accurately build a complete databaserecord of the event.) Cross-lingual IE also

Information extraction (IE) fills slots in an ontology
or database by selecting and normalizing subsegments of human-readable text. Examples include
find names of entities and relationships between
them.

15

includes the ability to normalize entities acrossmultiple languages.
(The entity "President
Bush" is written differently in Chinese than inEnglish, and furthermore, has multiple Chinesetransliterations in common use.)

document retrieval,answering, etc.

video

retrieval,question-

In addition to the evaluation exercises, these
conferences created a number of significant data setsthat fueled further research in IR such as the use oflanguage models for IR. The TREC events havecreated a set of document collections (a few million
documents) with queries (a few thousand) withcorresponding relevance judgments (a few million).
These data sets have played a key role to promote
progress in the field. However, given the significant
increase of online content over the past few years(the current Web is estimated to be about ten billion
pages) and of the increasing rate of using search(tens of millions of queries per day), the current
TREC data sets are too small (perhaps a thousandfold too small) to be representative of the “realworld”.

5. Relation extraction. What makes a database or
knowledge base most interesting is not lists ofisolated entities, but the relations among thoseentities. To make further progress in accurately
extracting relations, new models are needed.
Relation extraction also includes event
extraction.
6. Confidence. Information extraction systems
should provide accurate estimates of its
confidence in its own analysis and answer.
7. Robust ability to be trained from limited labeled
data. Make efficient use of unlabeled data andhuman interaction.

Hence a community-based effort is needed to createa more realistic (in scale and function) common data
set to fuel further research and increase the relevanceof the research activities to the commercial andgovernment activities in IR. We outline below someof the elements needed for creating a set of common
resources including data sets, possibly annotated,
and testbed platforms for IR research.

Language Modeling
Language modeling has been at the heart of many ofthe most successful information extraction systems.
Language modeling has been applied to extraction
tasks, particularly to name extraction, including
hidden Markov models, maximum entropy models,and conditional random fields. Such trained systems
achieve performance comparable to the best
handcrafted, rule-based systems.
Additionally,language modeling is being applied to relation
extraction, co-reference resolution, and extraction of
the literal meaning of text.

The first element is a data collection effort to capture
real users performing real search tasks encompassing
a sufficiently large set of queries and corresponding
set of retrieved documents; the acquisition of thesedata would allow the exploration and developmentof algorithms for learning retrieval models. To
facilitate the data collection effort, access to a stateof-the-art (SOA) search capability is required toconvince users to use this testbed for their searchtasks. One approach is to use a proxy or a metasearch engine to a SOA search service and provide
users with its search results. This approach would
rely on instrumenting the proxy to log users’interactions over a session including all results andancillary documents to create a complete data set ofuser queries and corresponding results andunderlying documents; the details of what needs tobe logged would be determined through the IR
community’s input. The scale of this data collection
could easily be on the order of tens of thousands ofThere is much evidence that language modeling alsolies at the heart of future needed progress. The
models will need to be more sophisticated, makemore targeted approximations, and have improved
parameter estimation procedures.
3.11 Testbeds
Over the previous decade, the IR researchcommunity has benefited from a set of annual US
government sponsored TREC conferences thatprovided a level field for evaluating algorithms andsystems for IR. The conferences have included
exploration of several new areas such as spoken

16users each performing hundreds of queries over a
period of several months.

The above-shared common resources can be
developed using a parallel approach in developing
the various elements with an appropriate mechanism
for integration at a set of planned timelines to benefit
from the shared work. To enhance the chance of
creating the shared resources in a timely fashion, thecreation and funding of specific projects to satisfy
the various elements would be needed. NSF via its
ITR program may be a source of funding to help
create the Common Testbed for IR; in addition,other agencies such as ARDA or DARPA may
contribute to the creation of this testbed. It is
anticipated that such an approach is required in the
IR field to enable the next generation of research to
be relevant to the new challenges and opportunities
created by the explosion of the Web.

The management of user privacy and document IP
rights would require an experienced organizationsuch as the Linguistic Data Consortium (and/or theInternet Archive) to coordinate the data collection
process, in addition to the actual system building
activity to create the instrumented platform toperform the data collection experiment. In addition,
the LDC could play the role of a data distribution
center similar to the role it currently plays in similar
data collection efforts for current human language
technologies research. The scope of the datacollection can easily reach millions of queries withcorresponding tens of millions of retrieved
documents with associated logs of user activity.
A second element for a common resource would be
the creation of a snapshot of a significant fraction of
the web (larger than say 10%, i.e. from 200 million
to 2 billion pages) to be made available for researchin conjunction with data logs based on real users
doing real tasks. The instrumentation of the interface
will be based on the community’s input.

4. Conclusion
During the last decade, IR has been changed from afield for information specialists to a field foreveryone. The transition to electronic publishingand dissemination (the Web) when combined withlarge-scale IR has truly put the world at our
fingertips. However, information specialists andordinary citizens alike are beginning to drown in
information. The next generation of IR tools must
provide dramatically better capabilities to learn,
anticipate, and assist with the daily information
gathering, analysis, and management needs ofindividuals and groups over time-spans measured in
decades. These requirements require that the IR
field rethink its basic assumptions and evaluation
methodologies, because the approaches andexperimental resources that brought the field to its
current level of success will not be sufficient toreach the next level.

A third element of the common resource, would be
the creation of a complete testbed framework which
would enable researchers to incorporate newmodules into the testbed using the specified API setof the testbed and to conduct user-based researchand data collection by introducing new functionality
beyond the SOA search technology. The creationand maintenance of a testbed for conducting plugand play experimentation would require significant
effort but could be leveraged by several researchgroups to advance the SOA in IR.
A fourth element would be the annotation of certain
data sets for specific research projects such as thelabeling of “aspect” information in a set of
documents (i.e. identifying the various subtopics in adocument relative to the query) retrieved by a query.
The identification of an efficient approach to theannotation task would be a challenge for the specificproject that undertakes the task. The development ofannotation standards would increase the sharing ofeffort across many teams.

17