-5CHAPTER 1.
INTRODUCTION
1.1

Project Origins

Improvements in the performance of information retrieval (IR) systems
as presently designed seem to be limited to only marginal gains in termsof complete recall and precision or complete user satisfaction (see e.g.
Robertson and Sparck Jones, 1976). We*report here on a design study foran experimental IR system based on radically different hypotheses andassumptions than those underlying present systems, which we think may
allow the design of IR systems which produce significantly better
performance than that offered by present IR systems.

We anticipate that,

if successful, such a system could be used in both interactive on-line
and batch off-line searching of any machine readable data bases which
include abstracts as part of the document record.
This design study is the beginning of what could be a four-stage project.
Because of the nature of the problem, it is convenient to break down theentire project into these four stages, each successive stage being
predicated on the success of the preceding stage, each stage contributing
knowledge necessary for implementing the next.

The design study results

are discussed in detail in the following chapters; here we can indicate
in rather general terms what we anticipate the successive stages might
involve.
The second stage would be actually building a prototype system based on
the specifications of the design study, developing and processing anappropriate data base, and testing the system with simulated users.

Themajor effort here would be on discovering the implications of the design
proposals for an interactive system, such as searching techniques andcosts, and programming>

The result of this stage would be a prototype

working interactive IR system, with known characteristics.
The third stage would be a small-scale experiment testing the systemdeveloped in stage two with real users.

This stage will include modifi-cation and retesting of the system as results are accrued.

The result

would be a fully operational robust interactive IR system which had
produced good retrieval results for a small number of users in a limited
discipline.
The fourth stage would take the system and its components and test themin a large-scale, multi disciplinary environment, such as that of the-6proposed 'ideal test collection1 (Sparck Jones and Van Rijsbergen, 1975),
and compare the results of the system against those of other, traditionallydesigned IR systems, probably in terms of recall and precision. The
results will tell us whether the IR system we have designed can accountfor scale and discipline differences and whether its good results on asmall scale translate to significantly better (than other systems) results
on a large scale.
The benefit of this long-term project could be great indeed, for it
represents the first (to our knowledge) attempt at designing an IR system
which is not based on traditional hypotheses regarding document representation, request specification, information need and matching.

If successful,it could be the basis of a 'second generation' of IR systems, designed ona specific theory of IR in which the user's role is the key, and producing
IR performance much closer to complete user satisfaction than any presentday systems.
1*2

Background

Recent work by Belkin (Belkin and Robertson, 1976; Belkin, 1977 a,b#, Belkin,in press) and others (see e.g. Harbo & Kajberg, in press) has called intoquestion some traditional assumptions of IR, in particular those concerning
the relationship of the request put to the IR system to the informationneed underlying the request, the basis for text and request representationin IR systems, and the retrieval mechanisms suitable for IR.

This newapproach recognizes that the fundamental element in the IR situation is
the user's information need, from this realization going on to say thatfor IR to be successful, that information need must be represented inappropriate terms, with the remaining elements of the system (i.e. document
representation, retrieval mechanism) being represented or constructed on
the basis of that representation.

The means to an appropriate represent-ation is consideration of information need as an 'anomalous state ofknowledge' (ASK) (see Belkin, 1977 aj in press).

The ASK hypothesis isthat an information need arises from a recognized anomaly in the user's
state of knowledge concerning some topic or situation and that, in general,the user is unable to specify precisely what is needed to resolve thatanomaly.

Thus, for the purposes of IR, it is more suitable to attempt todescribe that ASK, than to ask the user to specify her/his need as a request
to the system.

One major difficulty with the ASK hypothesis is the questionof determining the best system response to a particular ASK structure.
One aim of the design study is to investigate this problem.
Oddy's (1975,1977 a,b) experimental interactive information retrieval system
was designed to allow information retrieval without query formation, celying

-7on the system to construct an 'image1 of the user's need.

The program

operates upon a graph whose points represent documents, subjects andauthors; the lines stand for associations.

A structural image of thesubject area of interest to the user is maintained during the on-linedialogue.

Its main component is a subgraph of the program's document

collection structure.

Strictly speaking, this image is not asserted to

be a representation of the user's need, but rather a formal context within
which documents satisfying the need might be found.

Thus it is thestructural properties of the image, and not merely matching terms, which
determine the program's choice of documents to show to the user.

Theuser may react to the documents, and their descriptions, and his response
is used by the program to determine modifications to the image.

This

work has strong affinity to the ASK hypothesis, but is limited in its
means for representation of document and user to traditional document
descriptors, although in explicitly structural terms.
study, we have investigated means

In this designof combining and extending these twodevelopments to deal with complex IR situations in an entirely new way,
which the ASK hypothesis predicts would produce better results thantraditional IR systems.

The design study has attempted to resolve anumber of problems raised by the new hypotheses, and to provide apreliminary specification for a system which incorporates these newassumptions.

In addition, we report some results which can be interpreted

on their own.
1.3

Project Specification

The system with which the design study was concerned consists of a structural text-analysis program, a suite of retrieval mechanisms, a data-baseof abstracts and a mechanism whereby user and system interact.

The system

would work as follows (summarised in Fig 1 ) :1.

The user discusses rer/his information problem in an unstructured
statement (say, 2-3 paragraphs long).

2.

The problem statement if converted to a structural representation of
the user's ASK by the texc analysis program.

3.

According to the type of problem structure (PS), one of the several
available retrieval mechanisms is chosen to interrogate the database (each member of which is represented by a structural representation of the information associated with the text).

In this context,

a retrieval mechanism is a strategy for resolving the anomalous aspectsof a PS.
4.

The abstract (i.e. the text) is printed out for the user to read.

-8Simultaneously, the user is presented with a brief explanation of
why that particular text was chosen (explanation of the retrieval
mechanism), indicating aspects of the text structure which thesystem finds significant in that choice.
5.

The system then initiates a structured dialogue with the user, basedon the information presented to her/him, inferring from the responses
the user's attitude toward:a)the method of choice;b)

the suitability of the document to the problem; andc) whether her/his information problem has changed.
6.

7.

a) The system changes retrieval mechanism if necessary, and/orb)

the system modifies the problem structure if necessary, orc)

the system stops if the user is satisfied.

The system returns to step 2 or 3.

Physically, this is an interactive on-line system with both printer andVDU.
1-4

Aims of the Design Study

Aspects of this system which must be clarified by the design study
before any experiment is begun are:
1. The text analysis program.

This is based upon an existing algorithm(Belkin, 1977 a) which determines statistically-based structures.
The suitability of the eventual algorithm must be tested on real
problem statements (see 2 below) and on abstracts (see 3 below).
2.

The feasibility of obtaining problem statements, the possible typesof problem statement, and the analysis of problem statements. To
these ends, a number of real problem statements were collected andanalysed, and the analyses verified.

3. The feasibility of using abstracts as the basic documents in the system,and the analysis oi abstracts. To these ends, real abstracts werecollected and analysed, and the analyses verified.
4.

Some specific retrieval mechanisms must be proposed, based in largepart on the types of problem structures encountered, although the ASK
hypothesis implies some strategies.

5.

The interactive system (especially the dialogue with the user) mustbe specified (although not necessarily fixed).

The design study thus required:a)Collecting a suitable number of problem statements under appropriate
conditions;b) building a collection of abstracts (not necessarily related to the-9-

1. USER'S PROBLEM STATEMENT
2. STRUCTURAL ANALYSIS OF PROBLEM STATEMENT
3. CHOICE OF RETRIEVAL STRATEGY ACCORDING TOTYPE OF ASK
4. ABSTRACT PRESENTED TO USER SIMULTANEOUSLY WITHEXPLANATION OF WHY TEXT WAS CHOSEN (STRATEGYAND SIGNIFICANT FEATURES)
5. STRUCTURED DIALOGUE BETWEEN SYSTEM AND USERFOR SYSTEM TO INFER USER'S EVALUATION OF:A)

METHOD OF CHOICEB)SUITABILITY OF DOCUMENT TO PROBLEMAND/ORc)

WHETHER NEED HAS CHANGED

6. MODIFICATIONS ACCORDING TO EVALUATION, OR FINISH
7. RETURN TO 2 OR 3 AS NECESSARYFIGURE l: AN ASK-BASED IR SYSTEM DESIGN

-10problem statements);c)

developing an analysis algorithm;d) analyzing problem statements and abstracts;e) asking users and authors for constructive comments on the suitabilityof the representations of problem and abstract;f)developing an algorithm for classifying PSsg) developing retrieval mechanisms; andh)

specifying the basic system.