2S u b j e c t

sear-czil-iiLng

p r o b L e m s

2.1 Introduction
In most online catalogues an unacceptable proportion ofsubject searches fail to find anything at all. Markey C1,
p833 gives proportions ranging from 35% to 57% taken fromstudies of four catalogues in the United 5tates.
One of the Final Reports arising from the CLR (Council for
Library Resources) study [2, p133 recommended that users'
subject search terms should be automatically truncated if
there were no retrievals. The same report recommends thatspelling correction should be applied in known-item
searching. This applies at least equally to subject
searching COkapi transaction logs suggest that spelling andkeying mistakes BrB more serious in subject searching thanin known-itemD. The same CLR report also urges the provision of "cross-references online" and "related word lists
to lead users to more general termCsD".
The main problem in online access is that of matching theuser's search to the various ways in which the sought
objects may be described in the database. First, many
searches contain misspellings or miskeyings. Since this isoften unnoticed by the user, who assumes that the sought
subject is not covered, the retrieval system should help
the user to correct misspellings. Then, even when all thewords of the search QFB correct they may not correspond to
the language of the catalogue. Searches can be broadenedby automatic stemming of their constituent words and byautomatic cross-referencing.
It is these three devices - spelling correction, stemmingand the use of cross-reference tables - which are thesubjects of this report.
Rdditionally, in keyword-type catalogues, many searches
fail to find anything because not all of the words of thesearch co-occur in any record, even after they have beencorrected and stemmed and cross-references have been drawn
in. This does not mean that there are no relevant records.
Often it is due to the inclusion in a search of "inappropriate" words such as expressions of time CWOMEN'5 WORKBETWEEN THE WRR5D or of scope CCRITICISM5 OF FRRNK PRRKIND.
5ome, but by no means all, of these terminological problems
can be alleviated by using a fairly large stop list.
CRITICISM cannot be stopped, BETWEEN, UNTIL etc probably-7-2 Subject searchingproblems

can be stopped.
(Whether dates
should be stopped is aninteresting question which there is no room to discusshere}. What is clear is that online catalogue subject
search systems must be able to retrieve records whichcontain only some of the unstopped words of a search. We
refer to techniques for achieving this under the generalname "combinatorial searching - .
On evaluation of combinatorial searching was not one of theobjects of the present work - we regard it as an essential
feature of any keyword search system for untrained users.
Nevertheless there are many references to it in this reportbecause our application of some of the other devices isclosely connected with Okapi's method of term combination.
2.2 What users bring to the catalogue
The majority of user's subject search statements as recorded in system logs are straightforward and comprehensible. Most of them are concise noun phrases - many aresimple one-word concepts. Sometimes they contain synonyms
or related terms which are intended to be treated as alternatives. There are quite a Lot of dubious spellings. 5ome
searches are incomplete or incoherent, and there is always
a proportion where the user is "fooling11 or "playing" orsimply not concentrating.
The searches listed in Table 2.1 were selected systematically from logs of Okapi '84 Cevery twentieth searchstarting at a randomly chosen page3. They were all submitted as searches for "books about something", and in
response to the prompt:
The computer will look for book(s) described by as many as possible of theword(s) you type. Please enter word(s) or a short phrase which describe yoursubject:
It needs to be pointed out that only about half of themwere the first search in a session.
"Independent", forexample, was part of the sequence "itn" [independent television n e w s ] , "independent", "entertainment", "radio one".
Readers of this report who have access to an online catalogue accessing an undergraduate level collection coveringsocial studies, communication, economics and businessstudies are invited to repeat these searches, with orwithout the misspellings.

2 Subject

Table 2.1

searchingproblems

R sample of subject searches

1 Popper
2 behavior teddyboys subculture
3 anything by Frank Parkin
4 raddio
5 tetnology influence on structure
6 drug abuse treatment of drug dependents
7 resource mineral depleation
8 central intelligence agency cia media press news propaganda tv radio
9 hollernzolleran [probably for Hohenzollern]
10 female sexuality
11 consumer decision making models
12 Rees case
13 independent
14 underdevelopment
15 machiavelli
16 modernism
17 Capital radio
18 photo;nature—nude
19 photography
20 education welfare
21 traumatalogy
22 nationalised industries
23 communal ism patrimonalism
24 early development statistics movement

2.3 5ubject search facilities in current online catalogues
2.3.1

Phrase searchingSome online catalogues offer subject access to an index of
Library of Congress Subject Headings CLC5H3. The first
result of a search is a display of headings Cnot bibliographic records) in the alphabetical region of the user's
input. The user can browse alphabetically backwards andforwards, and can select records indexed under a chosen
heading. Some of these "phrase access" catalogues return afailed search if the user's key does not find at least a
partial match with a subject heading, but most of themalways display
something.
2.3.2

ProbLems withsubject headings

Research [3, 4] has demonstrated the inadequacy of LC5H;
many headings lack specificity and the language used isoften out of date. Rt least half of all searches fail tolocate either a heading or a reference at the firstattempt. If subsequent attempts are included this figure
can rise to about 70% C53. The proportion of searches-9-2 Subject searchingproblems

which exactly match a subject heading is usually very low
C25% is typical), but not all of these searches fail. Of
the 24 searches in Table 2.1, five are near enough to a
Library of Congress heading in the PCL catalogue to find at
least one book by browsing headings; two more are near to
PRECIS headings.
Mandel and Herschman [63 suggest using feedback from user
searches to incorporate more "see" references into the LC5H
structure. This is certainly desirable, and it is themethod we used for constructing the automatic crossreference table used in the versions of Okapi described inthis report. However, users should not have to repeat
their searches using the "preferred- form of a heading. On
online catalogue can and must do this automatically. If
our users think of the Department of Education and Science
as DE5, then which is the •best1 term? CProvided we canprevent French "des" entering the index}.
"See also" references are a different problem, and one
which has not been seriously tackled yet. Some of the more
recent commercially available systems do at least allow for
their display and selection by line number.
The LC5H problem is only partly one of language. Several
studies have demonstrated that lack of specificity inindexing often causes searches to fail. Mandel andHerschman [63 point out that this is not always the faultof the content of LC5H but rather the consequence of poor
indexing.
2.3.3 Keyword

searching

Most of the more recent catalogue systems use individual
words rather than headings; a few offer both access methods
Cbut how does the user know which method to choose?). The
words may be taken from subject headings or from titles orboth. Many European Libraries do not use subject headings.
5ubject access, if any, has been provided by printed
indexes to classification schedules. When such libraries
automate their catalogues they sometimes provide "keywordin-title" searching for subject access. It is likely that
titles are a slightly richer source of subject-rich keywords than are subject headings, although users findsubject headings useful for judging the likely relevance ofa retrieved record. CHence subject headings should be
included in the bibliographic display}.

.m.

2 Subject

searchingproblems

Thirteen of the searches in Table 2.1 (numbers 1, 2, 4, 6, 7,
10, 11, 14; 15, 16, 20, 22, 23) were repeated on the current
project's LXP system. RLL find some books through (stems of)t i t l e words and eleven find some via subject heading words.
Title was a richer source in eight of the searches and subject
in two, three being judged equal. The two searches which
didn't work at a l l on subject headings are 'female sexuality1
and apatrimonialisma.
I n almost a l l keyword systems, words are combined using an
i m p l i c i t boolean QND.
Up to a h a l f of the searches may
f a i t i n s p i t e of the f a c t that i n d i v i d u a l words are f a r
more L i k e l y to f i n d something ;than phrases, because thewords do not a l l co-occur i n any record.
Of the 24
searches i n Table 2 . 1 , 12 f i n d ,something when t h e i r words
are RNDed.
5even of these are^single word searches.
2.3.4 Recess

pointsMost MORC records are extremely poor in subject content.
Marcia Bates has recently suggested C7] that LC5H would be
adequate if clever linguistic and other preprocessing isapplied to users' searches. We do not believe that this is
the whole answer. In the long term the emphasis in cataloguing must be moved from physical description to subject
description. Until this happens it is essential to use all
MPRC fields which can contain subject information.
Markey gives a list of subject-rich CU53 MRRC fields in [1,p1583.
Titles tend to use language which is more current than thatof subject headings and indexes, but they are also rich inmetaphors and "noise-words- like "introduction-. 5eries
titles and corporate names are of some value, as arecontents notes when used.
Markey and her team tried keyword access to Dewey indexes[83 and they found that, as with LC5H, the Dewey language
is not rich in the sorts of words used by library users.
CThe use of the actual classification codes as a means oflinking related records is one of the subjects of a related
Okapi project. It is outside the scope of this report.D
2.3.5 Recess method

Phrase matching systems depend on at least the first few
characters of the user's input matching the first few
characters of a relevant heading. Some online catalogues
are undoubtedly less effective than card or microform catalogues because rapid browsing is difficult or impossible or
because there is no cross-reference facility. The Geac
catalogue at the Polytechnic of the 5outh Bank offers-11-

2 Subject searchingproblems

subject access to a List of subject word descriptions Csome
descriptions are based on PRECIS, others are specific to
the institution}. RLthough some of the words are tagged
for retrieval. , other subject descriptions are only retrieved if the first characters are matched. On effectiveness study of this catalogue determined that while it wasquite effective in specific item searches, only 3 4 % ofsubject searches were successful C9, p643.
In some keyword systems difficulties are caused by theway in which keyword access is provided. Some catalogues
allow the entry of only one keyword; most allow more than
one keyword to be entered but then only retrieve records
which contain all of the users' terms.
2.4 How might subject access be improved?
2.4.7 Truncation

andstemming

TRUNCRTION
Truncation makes it possible for a user to retrieve morphologically related terms which may also have a semantic
relationship. Many current catalogues offer some sort oftruncation facility. Most phrase-matching catalogues willautomatically retrieve headings which match the user's
input but have additional characters on the right. Some of
the keyword-type catalogues allow explicit truncation ofwords through the use of a special symbol or command.
One keyword catalogue does a kind of automatic truncation
on a keyword search. This is the the OCLC L5/2000 system
when the user chooses to search by "keyword". It displays
a list of words which the user's word partially matches;
the user has to select one of the words, whereupon thesystem responds with a List of up to 20 or so indexes
Ctitle, subject, series, author etcD in which the chosen
word occurs, together with the number of titles pointed toin each index; finally, the user selects one of the indexes
and can see some bibliographic records [103. This may
suggest that single-keyword searching via an index display
is not always satisfactory.
It is unlikely that explicit truncation can be used without
training. Markey [23 reports that CLR survey respondents
found it difficult to use truncation. This is borne out by
experience at the University of Hull, where the Geac system
was modified to allow explicit truncation [113. Surveys
revealed that there was little use of the facility.
If truncation is to be used it must be automatic, as insystems which display headings which the user's search
partially matches. But the example above shows that the
automatic treatment of keywords
as truncations is
unacceptable.
"Cat" must not retrieve records under "cata-2 Subject searchingproblems

mite" and "catastrophe". Automatic truncation could beapplied to words which do not find an exact match, but itwould rarely have any effect. Of the words in the searches
in Table 2.1, the ones which do not occur in the index
Cafter correcting spellings) are teddyboys,
Parkin,Rees,
traumatology,

communaLism.

None of these searches ishelped by treating the words as truncations (.Parkinfinds Parkinson, the others do not match anything}. If thewords are truncated, communaLism will find communal andtraumatology
will find trauma Cor would if it were in thefile); but this is more efficiently done by automatic
stemming.
RUTOMRTIC 5TEWING
It is obvious that some searches of kevword systems would
work better if at least plurals, singulars and possessives
were conflated. This is not always saf3 but there is noneed for research before deciding whether it should be
done. Rt least six of the words in Table 2.1 work more
effectively if this rudimentary stemming is applied (.drug
abuse= abuse

of

drugs

etc}.

We do not know of any commercial system which provides
this, although there is one where it is in the specification but not yet implemented.
One of the objects of this research was to determine
whether a stronger form of stemming should be applied.
There are several examples in the sample? searches where it
might or might not be beneficial: abuse/-ing,
dependants/entsI-ence,

modernism!-ist, nationalised

I-isation/-ising.

R few of the experimental Cnon-commercialD online catalogues apply some degree of automatic stemming to the words
of a search, and look them up in a stem index.
Foremost amongst these is CITE, developed at the National
Library of Medicine. CITE is briefly described below in
2.5.
R system written by Peter Butcher Cone of the inventors of
the original version of the PRECIS subject indexing method)at City University removes terminal "s".
Bell and Jones' system M0RPH5 at the Malaysian Rubber
Producers' Rssociation [12, 13, 14, 15] is an in-housereference retrieval system rather than an online catalogue.
M0RPH5 incorporates stemming which is semi-automatic; users
have a degree of control over its application. Frakes [16]discusses a system called CRTRLOG which uses Porter's
stemming procedure [17] with, Frakes reports, results as
good as those obtained by intermediaries using manual
truncation.

-13-

2 SyfeJ^etg©©p©b£©(g)

pp©M©©i

Od b©©@ (TD© ©y£d©©©© on 4b@ ©ff@(stiv®R)®^g ®f ^4©m©i©g in ©
pybl£© ©©4©t@gy© 0
S4©©©o£©g 4©©h©£qy©^ © P © di§(gy§§>®d £© Cb©p4©ip 3oL£bp©pi©(n]@ [©©racily 4bioi(k @f ©p©g©-©©f©p©©©i©g ©© p©f©pp£©g4© ©©)©© ©^ p©£©4£©g f>©© °©©©-pp©f©p©©d° 4© Dpr©f©(pr©d0b©©d£(n)(g§ C G ©©@° fp@f©©©©©©©3 ©©d fr@© b©©d£©g© 4© tr3©l©4©db©©d£©g© C°©©© ©l©©° P©f©p©o©©©3c
Kfe £©©Lyd© y©d©p 4b£©
b©©d£©g © ©y©fe©EP ©f b©©£©@© ^ i d h ©r© ©©©4£©©©d b©l©©© ©©dd£©©y©©©d ©4 gp©©4©p (L@(©gft(h £© Cb©p4©p 4 Q
UK (MBFC ©It©©© © P © © © p©f©p©o©©© £© 4b© (bMFC O&m 4£©Ldi©
©44b®ygb ©©©© (L£b©©p£©© d© ©©4 ©©© 4b©©©
Tb© io°h§y§§@©(L£©© ©©4©l©gy© ©4 C©©fc©£bg© U©£©©rs£4© £© ©© ©^©©p4£©© 0
I© fMo©4b O M © P £ © © °©©© fp@©° b©©d£©g© d© ©©4 y©y©LL© ©ceyr£© b£bl£©gf©]pb£© p©©©rd^ 0
Ib©y ©r© ©ft©© b©ld £© ©©p©r©4©
©yth©p£t© f£l©© ^hich ©©© b© y©©d i© ©®©jy©©4i®© ©d4b 4b©
£©d©^©^ 4© ib© b£bl£©gr©pb£© r©©@rd©o
B&mm ©f 4b© php§§§°§ee§§§ 0 bp©©©£©g 4©p© ©©4al@gy©n ©l(L©w
4b© diipt©© ®f ©©d ©©l©©4£®© fp©(© °©©©a ©©dl °©©© ©l©®°^§f§p§ne§io0®©©©© 4© 4b© b£b!L£©g©©phi© fill© £© vi© ©©
©y4b®©iiy f£i@ 0 ©©©b b©©ding £© ©bi©b £§ l£©b©d ©£4b©p 4©
b£b(L£@gr®pb£© ^©©©pdl^fl ©r 4® © app©f®pira©da b©©d£©g aW©
b©v© ©© ©©id©©©© ©b©y4 4h© © K 4 © ® 4 t© ©b£©b ©y©b f©©i(Li4i©s©p© ©©ty©LL© y§ide
P f©^ ©©4©t®gy© ©©®4©©© COCLC, UR1CH3 ©11©© lib(r©pi©§) 4©
©@©^4py©4 4b©ir @ M I ©©©dl ©©d/®p pb©©©© ©gy£v©l©©®© 4©bl©© 5©boi©b ©p© y©©d ©y4©©©4i©©llv©
Obv£®y© ®©©did©4©m ©r© ©?®pd =pbp©m© ©gyi©©[l©©©©® Lib© D USR° s a U©i4©d S4©t©© ©f
Pbro©©£©©D o Th© U©i©©r©it© ©f C a l i f o r n i a © b E L W L ©v©4©myiii © 4©bt© f©r 4b© ©^p©®©i@© ©f ©bbp©©i©4i©n© ©<h©np©©©pd^ ©r© £©d©K©ds © r©©©rd ©©©4©£©£©g a 1©t° g©©©r©t©© ©n©dditi@©©L i©d©K ©otp© G F1R5T° E183«(R!yt©©©ti© L@©kyp f©©iLiti©© ©©© b© y^©d f©p ^©©©r©L pur°p©©©© ©tb©r 4b©© 4b© 4(p©dit£©u©©L ©ybj©©t b©©di©g ©od ©©(©©
p©f ©p©o©©© o Tb©©© £©©lyd© ©©©f(L©4£@© @f y©xy§y©(L £©bL©©°££@©©IL ©©©£©©te C©b£(Ld ® ©bildp©©3 5 ©©t©b£©g v©p£©nt ©p©!L(L£©g© C®^g]®©©g(Dt©©©) s ©pg©©£©©t©@©3 © P © © © M © © ©)£©©p©L L£©g©
C4©b s 4b©3 ©©d bp£©gi©g ©©©©©©i©© £©g©bb©p Cgi©©d©i©© f©^©p§ ©©©©©©©t©©^£^3 o lb©© ©©© b© y©©d 4© d©©L ©?£4b ©©©© ©f4h© pcr'©b(L©©© ©r£©£©g fr©© pbr©©©© ©y©b ©© a bp©©©b ©b©Lk° 0©b)i©b b©© ©©4b£og 4© d© ©£4b Fp©©©© ©©d ©o©(Lv; © P © © © 4 ©
©©©©©©4i@© i©£4h ©b©Lbo
P4 (L©©©4 4©yp ©©pd© ©©d p)bp©©©© ©f 4b© ©©©p©b©© £© T©bL©
2d 1 ^©yLd b© © © P © ©ff©©4£©© if p©L©4©d ©?©rds ©©d pbr©©©©
^?©p© ©yf©©©4 i©©L L© £©©L©d©ds d©b©©£@p 0 ©i© 5 ©©t£©©©ii^©d
iody^tpi©^ CLE y©©m g©©©^©©©©4 ©i©©©pgb©p]) ©©d 4v©

14-

2 Subject searching problems
Two specialised systems make extensive use of referencetables. These are CITE C2.53 and M0RPH5 C2.4.1). CITE
uses the MeSH headings whose definitions contain the users'
search words. M0RPH5 has tables of compound words.
2.4.3 Spelling correction
Spelling and keying mistakes are a serious problem incatalogue use, and one which is much more serious online
than it is in hard copy catalogues. Keyword-type catalogues are far more sensitive to these mistakes than browsing, phrase-search catalogues. R phrase containing a misspelling which is not in the first few characters may stillland the user near to the sought phrase, whereas a miskeyed
word which is to be RNDed simply leads to a search with noresults. This is not a serious argument in favour of thephrase-search type of catalogue, if only because scanningscreen displays containing about 6 to 18 headings is a very
slow process compared to scanning a printed list.
Okapi Logs suggest that 8% to 12% of subject searches
contain an orthographical mistake. We believe this is atypical proportion. Mistakes did not always lead to
failure in Okapi '84, because it used a combinatorial-typesearch, and could still find books indexed under the remaining words of the search.
It is tempting to argue that it is not necessary to help
catalogue users with their spelling and keying difficulties, because they will notice mistakes and re-enter thefailed search correctly. Unfortunately, they very often do
not notice mistakes, and leave the catalogue assuming that
there is nothing relevant in the Library. The least which
can be done is to prevent a search from proceeding until it
contains only words which the system recognises. It willappear Later that the treatment in Okapi '86 of words which
are not found in the index is probably more beneficial than"strong" stemming C8.6.5D.
We do not know of any commercially available online catalogue which attempts spelling error detection or correction. 5ome are fairly tolerant of mistakes in specificitem searches because they use acronym-type keys. In one,the 5WRLCRP LIBERTRS system, keyword subject searches containing a misspelling will sometimes succeed because, Like
Okapi, it does not insist that all the user's words cooccur .
R number of other information retrieval systems do try tohelp users correct spelling mistakes. The Bibliographic
Recess and Control 5ystem CBRC5D [19], developed at the
Washington University School of Medicine Library, includes
a facility which will look for approximate spellings in theauthor, title, subject and series fields. R retrieval
system at Massachusetts General Hospital [20] also attempts-15-

2 Subject

searchingproblems

to correct spelling and typing errors, as does the Paperchase system at Beth Israel Hospital in Boston, Mass [213.
One of the Paperchase design objectives was to allow theuser to enter whatever seems natural; this was based on the
preponderance of abbreviations and acronyms in medical
literature. The system tries to match what is typed inwith items in the database by applying the sort of methods
which a person might use. The main technique is to look
for partial matches on words; thus NEW EN JOUR MED would
retrieve NEW ENGLRND JOURNHL OF MEDICINE. If the truncation of words fails then the order of words is madefuzzier and individual words in assumed phrases sre
rotated.
Spelling correction techniques &rediscussed in Chapter 5.

2.5 CITE
5ince CITE provides more "advanced" features than any other
online catalogue, we give a brief description here.
There
iBre descriptions from various points of view by Doszkocs in[22, 23, 2 4 , 25, 2 6 ] . We have not seen any published dataon its actual use or effectiveness, except when it was compared by Siegel and others 127, 28J with another system
CIL53.
This catalogue accesses the monograph collection at the
National Library of Medicine, where it is used by medical
researchers and students. It accepts queries in ordinary
Cmedical) language, uses automatic stemming and synonym
generation via Me6H headings. It uses term "weighting"
Cstems and headings Bre assigned weights which determine
their relative importance}, combines terms combinatoriallyrather than with BTI implied RND, and outputs records inranked order - "most similar" first. It also allows relevance feedback.
If C11E were suitable for general collections and generalusers, the present work would not need to have been done.
But its stemming procedure is designed for medical terminology and it is dependent on the MeSH structure. It expects Cor at least invites')
the user to perform a ranking
operation on the words and Me5H headings which it derives
from the user's search. Published material does not suggest that it presents itself to users in such a way that it
could be efficiently used by casual patrons of a generallibrary.

-16-

2 Subject searchingproblems

References
1 MRRKEY K. Subject searching in Library catalogs : before
and after
the introductionof online catalogs. OCLC

Online Computer Library Center, 1384.
2 MRRKEY K.

Online catalogue use : results of surveys andfocus group interviews in several libraries.
Finalreport to the Council on Library Resources. Vol. II.

OCLC Online Computer Library Center, 1983.
3 MI5CH0 W. Library of Congress 5ubject Headings: a reviewof the problems, and prospects for improved subject
access. Cataloging & Classification
Quarterly
1 C2/3D,
1382, 105-124.
4 BOTES M J. Factors affecting subject catalog search
success. Journal of the (American Society

ScienceforInformation

28 C3D, May 1377, 161-163.

5 HQFTNER R. The performance of card catalogs : a review
of research. Library Research 1, Fall 1373, 133-222.
6 MONDEL C 0 and HERSCHMRN J. Online subject access :enhancing the library catalog. Journal of Qcademic
Librarianship
9 C3D, 1383, 148-155.
7 BOTES M J. Subject access in online catalogs : a designmodel. Journal of the American Society

ScienceforInformation

37 C63, November 1386, 357-376.

8 MORKEY K and DEMEYER 0 N. Dewey DecimalClassification

Online Project: evaluation of a libraryschedule andindex integrated
into the subject searchingcapabilities
of an online catalogue. Final report to the Council on

Library1386.

Resources.

OCLC Online Computer Library Center,

3 JONES R M. The Online Catalogue at the Polytechnic of
the South Bank : a review and effectivenessstudy.

London : Polytechnic of the South Bank, Library, 1385.
10 L5/2000 at Newcastle University Library : a progress
report. VINE 59, July 1385, 20-25.
11 GROHOM T W, LONE R and RICHORD K M. Keyword and Boolean
searching on Geac at Hull University. VINE 48, May 1383,
3-7.
12 BELL C L M and JONES K P. 0 minicomputer retrieval
system with automatic root finding and rolingfacilities. Program 10 C1D, Jan 1376, 14-27.

-17-

^y&Jdet g©©p©b£©g

ppohlm

13 FELL C L (Ml ©©d JONES K P Q l©dk-©f°tb©-b©©b £©d©^£og s ac©©© for tb© ©ppt£c©t£©© ®f ©ptif£(©£©l £(©t©ll(l£g©fi)Oi© Inn
X©f©p©©b£©© So Fb@ ©©©iy§©© ©f ^@©©£©g 0 Prae©(©bi©g^ ©fd)o @ml£btlI3CS t@©f©ir>®©)©©0 Qxf®pdlo HS)7&o London § 0m lib,13730 1SS-B1o
14 BELL C L (Ml @©d JSbE3 K Pa Ibx© d©©©l®pp^©ot of © bigbly£©t©p©ci£©© ©©©pdhirog) 4©dhoi(°p© f©p (MBPPFB b£n£©®©ipyt©p 0p©p©t©d P©tp£©y©l £P©pt£©fL(Ly b©yp£©t£©])Sy©t©©Do I(n)ff©(r>m<M)t£(Q)(n) Pp®©©§©£©g ©©b b©©©g©©)©©t 1?d, 1SSQ,
37-47
Dl JGMES K P ©nd 33LL C L PL Tb© ©yt©©©b£© ©^tp©©t£®n ©f(w?®P(dl@ fp©© t©^i© ©§p©©£©l(ly f®p £©pyt £©b® £©f ©p©i)©t£©np©tp£©v©(L ©©©t©©© b©©©d @© £©©©pi©d b£d©© 0 ins fe©©©p©b©nd d©©©d©p©)©©t a© 3©f@p©)©ti©© P©tp£©©©l g p/r,©©©©(££©gs©f fb© ihi^dlj©£©t PCS ©©d PCb ^ymp@m£um £Ci©g^© C®L(L©g©(
C<§)^&p£bg© ^ 2-B duly
19B40 Edit©d by C J ©©n PijsborgorioE©©bp£dg© Un£©©p©£t© P P © © © ®n b©b©lf ©f tb© 3r£t£©b
C®(©pyt©p 3@©i©iy, 13330 403-413 0
13 FPRKES 585 lb P © P © ©©nfll©t£©n f®p Inf®p©j©bi®n P©tp£©v©lo
Ins P©g©©p©b ©od d©©©(L@p©)©©t £© J©f©p©©ti©© P©tpi©©©L c°p(r>©©©(§(o?£©)gg ©f tb© tbipd J©£©t PCS ©©b PCb ©ywpo^iy©)
biog'© C@ii©g©, 3©/©bp£dg©, I~S Poly 1334* Editod bv C J
v©n Pij©b©pg©o 0 C©i©bp£dg© Uo£©©p©£t© P P © © © ©n b©b©Lf ©ftb© 3p£t£©b C©©pyt©p 3®©£©ty, 1383 ,1? POPTEP bo Fo Rb ©lg@p£ib©] t®p ©yft£^ ©tp£pping D
14 C35, 1330, 130-137 0

Pp^gpmm

1.3 UNIVERSITY 3F CPLTFPPblPo OlblSTON OF LT3PPPY
RUTQMPTIONo bELbYL P©f©p©oo© b©ny©i % b©i©©p©£ty ©f
Calif©p©i© 0©1£©© C©t©t©g 0 3©p(k©l©y g tb© Uni©©r©ity^
33 KELLY 3 ©nd @tb©Pi© Bibti®gp©pbi© Poo©©© & Control
Symton© Xof©^mmt%©n T©©b©©l®gy ©©b L£bp©p£©s 1 C23, Jon©
1382, 125°132 0
IQ FEblCFEL P P ©ob IMPiNETT B 0 o Pb ©ppl£©©t£@©°£©d©p©©d©©t©yb©v^t©^ ^ ® P bp©©°t©^t ©(©©Ly©£©0 3©©)p©t©p^ ©©b!£©©)©b£©©l P©©©©p©b 3, 1373, 1S3°1B7 0!1 COCblPPIME P Ho 0 pp£©©dty D ©©t©L©gy© f@pg£v©^ y©©p © P P @ P ©
§ ©© i£bp©p£©n £©t©pv©©t£@© © © © © © © © P © ©^ dp©©© @©L£o©
©V©t©p^ ©©LL©d D P©p©p©b©©©° o P©)©p£©©© Libp©p£©^ 13 CSD ,b©y 1332, 303^303o
0OSZKOCS T Eo FED § ©o F©©©o£©t£v© Iot©p©ot£v©
Di©ti@©©py f@p @©L£©© ©©©r©b£©g 0 0©ii©© 3©vi©^
1378, 133°173 0„^\{3\„

C2)

2 Subject searchingproblems

23 DD5ZKDC5 T E and RRPP B P. Searching MEDLINE in English: a prototype user interface with natural language
query, ranked output and relevance feedback. Qmerican
Society
Octoberfor Information
Science.
Rnnual1373 : Minneapolis.?. Information

Meeting
C42nd :
Choices andPoiicies
16. Edited by R D Tally and R R Deultgen. 1379,
131-133.
24 DbSZKOCS T E. From research to application : the CITE
natural language information retrieval system. In :Researchand Development inInformation

Retrieval :Proceedings BerLin 1382. Edited by Gerard Sal ton andHans-Jochen 5chneider. Berlin : Springer-VerLag, 1383,
251-262.
25 D05ZK0C5 T E. CITE NLM : natural language searching in
an online catalog. Information

Technology

andLibraries

2 C4D, December 1383, 364-380.
26 ULMSCHNEIDER J E and D05ZK0C5 T E. 0 practical stemmingalgorithm for online search assistance. Online Review 7
C4D, Rugust 1383, 301-315.
27 5IE6EL E R and others. R comparative evaluation of thetechnical performance and user acceptance of two
prototype online catalog systems. Information
Technology
and Libraries
3 C1D, March 1384, 35-46.
28 5IE6EL E R and others. Research strategy and methods
used to conduct a comparative evaluation of two
prototype online catalog systems. National
OnlineMeeting New York, Rpril 12-14 1383. Proceedings. 1383.

-13-