143

8
8.1

Okapi in use
Introduction'' If information systems are to be truly responsive to users' needs, change
itself must be considered in the process of system design. Systems design
can no longer be viewed as a one-time effort resulting in a static designthat is unchanging for the operational life of the system. ... Evaluation isdefined as the process of analyzing the functioning and/or usage of asystem so that decisions can be made concerning the effectiveness of thesystem in satisfying its design objectives." [1].
8.1.1

Brief literature review

The most important OPAC use study is the CLR-funded evaluation
project in the U S ; for a short description of the scope and objectives of thestudy see [2]; for an overview of the findings see [3].
The CLR evaluation gathered three types of data: transaction log
analysis, "focus group interviews" and questionnaires. This study was
carried out on a very large scale (public, university, government andnational libraries, with questionnaires administered to thousands ofusers). The results of the log analysis are largely quantitative andstatistical and are presented in the form of many diagrams, tables andcross-tabulations. The interviews focused on specific groups of people,
such as cataloguing staff or young library users, and the interviewer madethem talk about the problems they had with OPACs. This resulted in amore qualitative type of data. There were two questionnaires, one forusers and one for non-users; these are extremely detailed and they consist
mainly of "closed" multiple-choice questions.
Apart from this very large-scale study, there are also a number of smallscale usage studies, such as the following: a comparison between two
OPACs [4], or more [5], the study of transaction counts of one OPAC
over a long period of time [6], [7], comparisons of card and online
catalogues in the same library [8]. For a good review of OPAC use studies
see Chapter 2 of [9]. A recent paper of Cochrane and Markey [10]recommends which methods to employ when studying the use of OPACs.

8. Okapi in use

144

The next section analyses what is generally meant by the study of the useof OPACs and attempts to classify different approaches.
8.1.2

What can be evaluated and why?

The performance of OPACs needs to be evaluated. It can be seen from alibrary use point of view: for example by comparing the use of an OP AC
and a COM catalogue in the same library, to see which one satisfies users'
needs best. A designer and a library manager might be interested in thecomparison of the use of several OPACs for different reasons: the former
in order to improve OP AC design, the latter in order to buy the bestsystem available. Both might want to know which ones are preferred by
the users, which features are essential, which systems provide moresophisticated features, etc. OPACs can also be seen from an information
science point of view: their major characteristic is that they are IR
systems used directly by end users; evaluating them is the best way atpresent of gathering information on (a) the information searching
behaviour of a large user population and (b) the usefulness andeffectiveness of traditional IR techniques such as Boolean searching, witha non-trained public and a wide range of information needs. Studying
these last two aspects helps to elucidate the issues raised by evaluationfrom the other viewpoints: if we know more about users' searching
behaviour and how current IR systems respond, then better systems canbe designed for more people.
There are three main aspects of OPACs which can be evaluated.
(1)

The technical performance, which has to be taken into account whencomparing two OPACs. One can examine, for example, theresponse time as a function of the number of users and stations, orthe way response time varies with the type of search done.

(2)

The IR performance, for example: the frequency of matches/nomatches by type of search, the number of hits by type of search,phrase-matching success and recall and precision.

(3)

User behaviour, because system performance is linked to system
usage. This includes studies of users and of use:(i)

Identification of user profiles which should include: information on users' background (demographic, academic), on their8. Okapi in use

145

attitudes to and experience of computers and libraries, on
their information needs. User search patterns and interaction
behaviour should be analysed and correlated with these
profiles.

8.2

(ii)

User search patterns are, to a certain extent, defined by thesystem users are interacting with. But there have been someattempts at identifying some general/common user searchpatterns with OPACs [1], [9] or [11]. These search patterns
include, for example, the frequency of single versus multiple
searches per user session, the time spent doing various types ofsearch, the rate and locus of error, the frequency ofunsuccesful searches in which users re-enter or revise theirquery, etc.

(iii)

User interaction success. This can be collected from the users
themselves (their subjective opinions about ease of use,learning, re-learning, etc.) and by using an experimenter who
knows the system and its limitations (for example, to repeat
logged searches). The study of user interaction includes thefollowing two aspects:(a)dialogue success, e.g. which system capabilities are too
difficult to use, the ease of input of authors, the clarity
and understanding of instructions, messages and help,
which display formats are un/acceptable (content andreadability), keyboard problems, etc.

(b)

IR success, which is obviously linked with (2) above, but
is seen here more from the user's point of view. It covers,
for example, how well users' queries are matched by thesystem, whether users' searching behaviour is supported
or hindered by the system, etc. Users can also be asked
their opinion about the relevance of the records
retrieved.

Methodology

It was impossible to carry out a comprehensive and detailed evaluation
study of Okapi given the staff and time constraints. Nevertheless, it was
important to gather some information on the live use of Okapi, especiallysince ease of use was the primary aim. Also, implementing a "naive"
interface (Section 7.4.1) leaves the possibility of introducing more

8. Okapi in use

146

sophisticated features at a later stage, and seeing how users react to them.
8.2.1

Automatic logging

First of all, it seemed relatively simple and worthwhile to have some
automatic monitoring of usage. Since Okapi is implemented locally, it iseasy to run a logging program as and when wished, without any extra
cost. The only effort was in designing and writing logging procedures;
logging is then completely automatic. As can be seen in the sample printout in Appendix 9, Okapi logging procedure makes quite a detailed recordof the searches done (see [11] for examples of other logs). Searches can be
repeated easily by an experimenter, so that, for example, the records seenby the user can be checked for false drops. All keys pressed are recorded
together with the time taken by the user and the system for various tasks(Section 9.4.6).
SESSION BOUNDARIESThe user session is the natural evaluation unit. As was mentioned in
Section 7.5.1, it was hoped that the use of the BLACK key (labelled'Tress when finished") would delimit user sessions automatically in thelog file. Unfortunately, this was not successful: the students used the
BLACK Key for other unintended purposes, such as getting out of an
option (in particular to make their previous input disappear from thescreen) and starting again from the beginning, rather than using the
RED key, which takes the user back one step only.
LOG ANALYSIS

Nevertheless, it seemed worth analysing the logs, even without session
boundaries. Some analysis was done on 3912 searches logged between 28
November 1984 and 22 January 1985.
The first obvious use of the automatic logging method is to giveindications of the user search patterns mentioned above in Section 8.1.2(3) (ii): the logs were analysed by computer to extract some statisticalinformation, such as the proportion of subject searches, the average
number of words in subject input, etc. The logs can also be used for
judging the IR performance (Section 8.1.2 (2)) and some programs were
written to count, for example, the number of successful and unsuccessful
searches and number of matches, according to the number of words insubject search statements. The main results are given in 8.5.

8. Okapi in use

147

Information on the user interaction success (Section 8.1.2 (3) (iii)) can
also be found from the logs. But no program can be written to do this,partly because of the impossibility of recognising user session boundaries
automatically. It has to be done by hand: a human reader can more or lessguess the session limits, and reading through the logs is extremely richand valuable; but the information gained is very difficult to formalise.
8.2.2

Observation and structured interviews

Automatic monitoring provides only a limited amount of information.
One cannot rely solely on a logging procedure to study the use of aninteractive system. The identification of user profiles and the interaction
success mentioned in Section 8.1.2 (3) (i) and (iii) respectively, cannot berecorded by a logging procedure.
Online interviewing has been used in OP AC evaluation (for example with
Paperchase [12]). This technique was considered (making the system ask
questions such as "have you used Okapi before?", "are you a Social
Sciences student?") for collecting information on the users themselves(i.e. (3)(i) of Section 8.1.2); but this is both unreliable and tiresome forthe users. A human intermediary seemed unavoidable. Human intervention is essential, for example in analysing users' conceptual processes ifthey discuss their search strategies.
Several methods could have been used to gather different kinds ofinformation: interviewing (the talk-aloud technique has been used whilestudents search, but this may interfere with the activity being carriedout), observation (without any interference from the observer), survey(of users and non-users), questionnaires, group interviews (withstudents and staff), controlled experiments.
OBSERVATIONThe observation method was chosen for three reasons: it was useful to getsome feeling of students' behaviour in a natural environment; it involved
comparatively little work; some human marking of user sessions wasnecessary anyway, because it was not clear if the logging procedure would
be reliable enough to delimit user session boundaries.
The simplest procedure was to have the experimenter sitting at a desk not
far from the Okapi terminal, recording the beginning and end of user

148

8. Okapi in use

sessions and recording what was going on, onto an observation sheet,
which would later be compared with the printed log (and the answers to
the structured interview, see below). The experimenter was extremely
familiar with Okapi, so was able to follow what the user was doing
without being too near nor seen (in most cases) and without interfering.
The purpose of the observation sheet was to keep track of activities extra
to those recorded by the logging; i.e. problems with coloured keys andkeyboard, hesitations, looking at papers, writing, waiting, external signsof the interaction (sighing, making loud comments, touching the screen,
shouting...), talking to other students, asking for help from other students
or from the staff and sometimes from the experimenter. This corresponds
to the evaluation of the dialogue success mentioned in Section 8.1.2 (3)(iii) (a). No results of the observation as such will be presented in thisreport.
USER SESSIONSThe observation allowed 96 user sessions to be clearly isolated, between 22
November 1984 and 9 January 1985. A manual analysis was performed on
the logs of these user sessions; it gave more detailed results than theautomatic analysis of the 3912 searches, for example: the spelling
mistakes in different types of input (which a program could not extract).
The user session results are given in Section 8.4.
STRUCTURED INTERVIEW

As one member of the team would have to spend some time on site
observing users, a short questionnaire was designed to get some
additional information (on students' background and opinions, see
Section 8.1.2 (3) (i)). The procedure chosen to administer thequestionnaire was similar to a structured interview: the students were notasked to fill it in themselves but the observer asked the questions and took
down the answers. The general approach was that students should nothave to choose among predefined categories, so that they would give
more natural and spontaneous answers and ideas. Section 8.3 contains themain results of 70 structured interviews.
CONTROLLED EXPERIMENTS

It was also intended to set up some controlled experiments. It wasplanned, for example, to present five or six pre-selected searches of8. Okapi in use

149

different levels of difficulty in randomised order to students (withenough students to do all permutations) and see how different types ofusers cope. Several aspects of search difficulties could have been isolated
and tested: problems with the input of authors (personal, corporate),
problems of no match/lots of matches in subject searching (how tonarrow/broaden) and known item searching, problems of phrase/word
matching. Other areas of interest were considered, such as the readability
and clarity of different types of screens and record displays (including thematter of record ordering).
But these experiments would have involved too much work in design,organisation and programming to be carried out within the allocatedtime.
To summarise, the data collected during the evaluation phase of thisproject consist of: 70 structured interviews, 96 user sessions and 3912
logged searches. It would have been very interesting to try to correlate theinterviews (and perhaps also the experimenter's observations) with thecorresponding user sessions, especially since these two types of data arerarely collected together. But this is not a simple task and would have beenvery lengthy.
8.3Structured interview results

The evaluation took place in the reference area of one site library of the
Polytechnic: the Social Sciences, Business Studies and Communication
Library, in Riding House Street. This area of the library is usually quite
crowded and there is not much space. An effort was made not to interfere
too much with the students and with staff activities. One microfichereader was left in its normal place, so that students would still be able tofind and use their usual catalogue. The COM catalogue used at PCL
consists of two microfiche sequences, author-title and classified (Dewey
numbers). There is a printed subject index available for consultation. The
second microfiche reader situated next to the first one was replaced by an
Okapi terminal; a poster above the terminal explained that Okapi was anexperimental computer catalogue, that anybody could use it, and that allcomments were welcome. There was also a suggestion book.
The data gathered are of three types:

150

8. Okapi in use(1)students' academic background and experience in the use ofcomputers, keyboard, catalogues and computer catalogues — see
Appendix 6 (Part 1).

(2)

users' reactions, opinions and comments on Okapi — see Appendix
6 (Part 2) for quantitative data and Appendix 5 for qualitative data(list of actual comments).

(3)questions about their search: what they were looking for and if theyfound it — see Appendix 6 (Part 3).

Some cross-tabulation of the answers was also done and this is in
Appendix 6 (Part 4).
8.3.1

Users' background and experience

The full results are in Appendix 6 (Part 1). Among the thousand users of
the site library, 70 students were interviewed. More male (41) thanfemale (29) students were observed using Okapi and interviewed. On thewhole, it is a young population: 50 students were between 19 and 25 years
old; there were more third year students than first or second years. A
large majority of the students were on social sciences courses. Most
students interviewed were first-time users of Okapi, in fact of any
computer catalogue. Their previous experience of catalogues was chieflyof the COM catalogue. Although 59 students had been in contact with acomputer before, 39 of them did it through a "quantitative methods"
course where they learn how to run statistical packages on a mainframe.
Most of them felt this had not been useful. A few of them had a suspicious
attitude to computers in general ("I ignore them and don't like them","daunting technology").
8.3.2

Users' opinions of Okapi

QUANTITATIVE RESULTS

A large majority (90%) of the students said they liked Okapi and thatthey would use it again; see Appendix 6 (Part 2). Four users said they did
not like it, but that they would use it again. But a much better indicationof ease of use is the following: nearly two-thirds of the students (62%)

8. Okapi in use

151

thought that they would need five minutes or less to feel confident about
using Okapi; see Appendix 6 (Part 2).(1)Correlating students' background and previous experience of computers,
keyboard and COM catalogue with the fact that they liked Okapi or not is
not terribly conclusive; see cross-tabulation in Appendix 6 (Part 4). The
main reason is that our sample does not include enough people who did
not like Okapi. An interesting result is that whereas 6 of the 41 men(15% ) said they did not like Okapi, only 1 of the 29 women ( 3 % ) said so.
QUALITATIVE RESULTS

The most interesting contusions from users' comments are classified andpresented below. Throughout, the numbers between brackets refer to a
specific user's comment (see full comments in Appendix 5).
Attitude to computers
A few students were obviously suspicious about computers when theyfirst started, "it won't bite?" (11), or rather scared, "I didn't dare press
the keys, I didn't know what might happen" (3). But some found it"easier than they had expected" (12), thought that it could "be used bypeople who don't know about computers" (59) or even became quite
excited about it: "when are you going to put the whole world in i t ? " (11).
One student did the same search three times "to make sure the computer
gave the same result" (13).
Use of keyboard, typing problems, VDU
Typing problems were often mentioned and seem a serious handicap. For
example: "If I was good at typing I would use it all the time" (21,43, 57).
This led some people to say that because they are so bad and slow at
typing, they would rather use the microfiche, e.g. (28, 83). The keyboard
gave the impression of being small (1) (although the key spacing of the
Okapi stations is standard). One student pointed out that it was "toomuch trouble to look at the screen and keyboard at the same time" (14).
Comparison between microfiche and computer
On the whole, students liked Okapi better than the microfiche. There
were quite a number of enthusiastic comments in favour of the computer(1)

The question asked ("when will you feel confident about using Okapi?") contains theimplicit assumption that users do not already feel confident. In spite of this several
people replied they did feel confident now.

152

8. Okapi in use

catalogue: "Brilliant, magic. I'll use it rather than the microfiche" (87),"the microfiche seems dead" (75). Nevertheless, many comments cannot
be clearly labelled "for" or "against": students were able to see
advantages and disadvantages of each.
Physical aspects: first of all, several students said that the microfiche madethem "feel dizzy" (24) or "was painful for the eye" (11) and that thecomputer was "easier to read" (65); this also might be because themicrofiche reader was not especially clean; in some cases, this made them
prefer the computer (24, 88). Another reason for preferring the computeris that it is "physically less tiring" (1), "less fiddly" (33), "all the books
are together" (59) and that the "fiche is always in the wrong place" (57).
It seems to be a question of choosing between the lesser of two evils:
typing or "fiddling around" with the fiche (7). One student preferred themicrofiche, because s/he felt it is "ordered" (6) and another one said thatthe microfiche "allows you to scan, unlike the computer" (5).
Speed, ease of use: many of them thought the computer was faster: "saves
time and less laborious. What would take 3 hours with the microfichetakes 20 minutes with the computer" (35). The students who had typing
problems obviously felt the microfiche was quicker (4, 28). Many users
preferred the computer because it was easier: "just pressing a button,
thing comes u p " (84), "more flexible, more pleasant" (81), "more fun"(8), "neat" (19), "more efficient" (30), "more interesting" and "morethorough" (72).
Subject versus specific item: quite a number of users felt that the computerwas "more useful for subjects than title/authors" (7), that "subject
searching is better on the computer, it is like scanning bookshelves" (58),and that if one cannot "remember the title or author of a book" (44) or if
the book is not on the shelf or out (36), it is possible to do a subject search
instead. They liked being able to retrieve titles and subjects (36), andthought that "the computer gives more useful related information,
unlike the microfiche" (48) and (46); one "gets an answer even if rougharea" (64, 69), "less chance of missing something" (72).
Learning Okapi and ease of use
A few students expressed some caution by saying they needed "time tolearn it" (3) and that it took them time to get used to the instructions(54). But most users said that the instructions were "clear", "straightforward" (77), "very simple" (55) and that "it is NOT a computer, you8. Okapi in use

153

don't have to learn it!" (84). It may also be because of the coloured keys,
which were very much appreciated (8, 30, 32). Several studentssuggested they should be labelled as well (14). The only noticeable
problem was the Black key; a number of students thought it was not
obvious (64, 72).
Comments specific to Okapi
These are, on the whole, about retrieval problems: title/author false
drops (48) which the users generally are not aware of (only one student(25) was); subject search mishaps, in particular when the "hyper-OR"(Section 6.5) does not give anything relevant (1, 11) — (75) also
complained about the hyper-OR not "telling when the search is over" —and when searching for names as subjects (88). When these subject search
problems arose, students complained about them and said that Okapi"ought to cope with that" (88). Only one student (5) complained aboutthe "too many matches" and "no match" situations and asked to be"oriented towards the library shelves". There was only one rather
strange comment about the input of author (14), and somebody asked
how to input two authors at once (5). Record displays were mentioned
only once and then only as compared with COM records which contain"too many book details" especially "numbers" (5). No student made any
comment about having to wait for the hyper-OR to finish; indeed there
was no complaint about the response time at all.
Suggestions for improvements
Apart from the criticisms linked with the IR problems mentioned above,
many suggestions are to do with catalogues in general, particularly withthe enhancement of subject access, both by providing more access points
and by including uncatalogued materials. The following suggestions were
made: including the indexes at the back of the books (5), abstracts "to say
under which angle the topic is treated" (11, 24), journal articles (29),essays and magazines (41), more keywords per book (81). A few students
had a more ambitious approach and asked for access to books from otherlibraries (33, 74), and to external databases (11). Several people asked formore machines (53), on each floor of the library and everywhere in the
Polytechnic (72), and one asked if s/he could "take it home" (74).

8. Okapi in use

154
Users' understanding of and attitudes to the system

Most users did not really understand what was going on. The only
exceptions were (46) who understood the number of postings for each
word, if not the principle of ANDing two words, and (5) who did
comprehend Okapi very quickly and was able to make constructive
suggestions ("If there is only one author [with a given surname], there isno point going through author index"). Nevertheless, many students did
not seem to be passive: they tried to interpret what happened, forexample the author input mentioned earlier by (14), or to explain why
the computer did certain things: "I didn't give it enough parameters so
the computer couldn't find anything" (23) or "the more specific you are
the better" (1). Some of them noticed fundamental IR features: "whydoes it do words one by one?" (10) or the title/author key principle (25).
Although they seemed to trust what the machine told them (75) — orassumed that if they did not find something it was because it was not in
the library (14) and put the blame on the library coverage rather than on
the computer (83) — they were rather wise: "I might find out thelimitations of the computer later" (46); when their search failed they did
not blame one element only: "I realised I made a mistake" (12), "I
assumed it was because my reading list is very bad". One student (58)gave a fair description of the situation from several angles at once:"difficult to know if I made a mistake or if it is because there are no booksin the library ... It is a bit like fishing in the dark ... Opinion depends on
success". Two students also felt that using Okapi avoided "hassling
librarians" (70, 79). One student (2) gave her/his vision of subject
searching: s/he said it is trying to find subjects which "fit" with"librarian-oriented" ways of describing subject matter.
8.3.3

Questions about their search

The full results are in Appendix 6 (Part 3). The main points are: only 19(27%) students said they wanted to look for subjects; the majority ofthem said they came with a course-related reading list given by a teacher,and so were looking for specific items. Thirteen people out of 70 said they
didn't find what they wanted. Once their search was over, most students(42 out of 70) said they only wrote down class number(s).
8.4

Analysis of user sessions

The full results are in Appendix 7. There are some discrepancies between
the number of searches input and the number of searches processed. This

8. Okapi in use

155

is because some searches were interrupted by the user.
8.4.1

Session length, types of searches, search states

On average, sessions lasted for 10 minutes and there were four searches
per session. There were more searches for specific items (62%) than for
subjects. This is not surprising because most students came with areading list (Section 8.3.3).
Out of 96 sessions, 62 contained only one ''search state", i.e. the user
stayed in the same ''state" (started with, for example, an author searchand did not change to any other type of search for that session). The mostfrequent sequences of search states are included in Appendix 7. This is
not very enlightening; it would have been more interesting, for example,
to look for the occurrence of pairs of search types in a session: for example
the proportion of sessions which started with a specific item search andcontained one or more subject searches, and whether the topics wererelated. There was not enough time, and it is difficult to extract this kindof information. There are also some problems to solve first, such as thedefinition of a ''search statement": when a search statement is entered
several times in the same session at different intervals, is it the samesearch? What if the user does it again 10 minutes later? Is it necessary to
decide on a maximum possible interval? (in Appendix 7, all searchstatements were counted).
8.4.2

Search failures

Similar problems also occurred in counting and analysing the types ofmatches by type of search processed. It is useful to know that about half
(90 out of 190) the title/author acronym searches failed; but it would havebeen more useful to correlate these failures with, for example, spelling or
input mistakes (Section 8.4.3). It would also have been interesting toknow whether trying title only and author only afterwards was any use,
and which was more useful, i.e. testing the "search trees" (Sections 6.8
and 7.4.3) to see whether the "paths" chosen are the most efficient ones.
This would have been complicated to work out and very lengthy torecord.
It would be interesting to compare the failures found in the log with theuser's answer to "did you find what you wanted". It would be8. Okapi in use

156

interesting, too, to redo the searches done by the 70 students interviewed,
look at the records to try to evaluate their relevance, and compare thiswith the students' answers. This is the advantage of being able to relate apiece of log with a particular interviewee (Section 8.2). The general
conclusion is that there were many search failures, whatever the reasons,and that users did not complain very much about this, in particular about
specific item search failures.
8.4.3

Use of keys, spelling and input mistakes

The analysis of these 96 sessions also confirms that the Black key was
misunderstood. It was used wrongly in 77 cases out of 125. About 10% of
the search statements contained spelling mistakes, but no attempt wasmade to correlate these with the length of search statements. A major
result is that there are more problems with author input (18 % of author
searches) than was signalled by the users themselves (see above"comments specific to Okapi" in Section 8.3.2); this could also be an
important reason for specific item search failures. It is worth emphasising
this point, especially since the team was particularly aware of thisproblem and worked for a long time on the design of the author input
procedures. This is a complex problem. In spite of prolonged efforts by
the team, this has not been solved satisfactorily.
8.5

Logging results

The full results are in Appendix 8.
The automatic analysis of 3912 logged searches (corresponding to 28
working days) gives a proportion of subject searches (43% compared to38%)) only slightly different from that found in the 96 analysed user
sessions (which therefore can be said to be a fairly representative sample).
The number of subject searches is not much lower than the number of
specific item searches, which the library staff were interested to discover.
The percentage of time spent doing each type of search (56% and 44% ) isalmost identical to the percentage of searches done (57% and 43%). It
would be interesting to see whether this changes with time, i.e. whether
students venture more into subject searching, especially since many users
think Okapi is more useful for subject searching (Section 8.3.2). Most
specific item searches are author AND title searches. Note that the Okapi
specific item input screen (Fig. 7.8) probably encourages the entry of8. Okapi in use

157

both author and title. T h i s is one of the ways in which Okapi differs frommost other OPACs.
T H E ANALYSIS OF SUBJECT SEARCHES

T h e average n u m b e r of words (excluding stop words) per subject search
expression is just over two (calculated from 1652 subject search
expressions). Not all multi-term search statements were A N D e d . For
example, on 603 two-word subject search expressions, only 502 wereA N D e d . T h e r e are two reasons for this difference: occasionally the userinterrupted the search, but mostly at least one of the two terms had no
postings (either because it was misspelt, or there was nothing in theindex). This obviously diminishes as the n u m b e r of terms in the search
expression increases: if there are more terms, even if one of t h e m does nothave any postings, the system still A N D s the others. Obviously moreA N D s fail as the n u m b e r of terms increases. T h e average n u m b e r of hits
even for two-word search statements (30) is not very high. But it is only
an average. T h e table in Appendix 8 shows the n u m b e r of h y p e r - O R s foreach type of subject search, but not the n u m b e r of records looked at by theuser.
T h e results presented here are brief and sometimes only of limited
interest, because of their incompleteness. T h e y are only the tip of theiceberg. M u c h more can be extracted from the logs but more work needs
to be done on how to analyse logs in general.
References
1

2

3

4

5

P e n n i m a n W S and D o m i n i c k W D . Monitoring and evaluationof online information system usage. Information Processing andManagement 16 ( 1 ) , 1980, p l 7 - 3 5 .
F e r g u s o n D K and others. T h e C L R public online catalog survey:
an overview. Information Technology and Libraries 1 ( 2 ) , 1982, p8497.
K a s k e N K and S a n d e r s N P . A comprehensive study of online publicaccess catalogs: an overview and application of findings. Final report to
the Council on Library Resources. Vol I I I . O C L C , 1983.
S i e g e l E R et al. A comparative evaluation of the technicalperformance and user acceptance of two prototype online catalog
systems. Information Technology and Libraries 3 ( 1 ) , 1984, p35-46.
M o o r e C W. User reactions to online catalogs: an exploratory study.
College and Research Libraries 42 ( 4 ) , 1981, p295-302.

158
6

8. Okapi in use

Norden D J and Lawrence G H. Public terminal use in an online
catalog: some preliminary results. College and Research Libraries 42(4), 1981, p308-316.
7 Borgman C L. End user behavior on the Ohio State University
Libraries' online catalog: a computer monitoring survey. OCLC, Officeof Research, 1983.
8
Pease S and Gouke M N. Patterns of use in an online catalog and acard catalog. College and Research Libraries 43 (4), 1982, p279-291.
9Markey K. Subject searching in library catalogs: before and after the
introduction of online catalogs. OCLC, 1984.
10 Cochrane P A and Markey K. Catalog use studies - since theintroduction of online interactive catalogs: impact on design for
subject access. LISR, 1983, p337-363.
11 Tolle J E et al. Current utilization of online catalogs: transaction log
analysis. Final report to the Council on Library Resources. Vol I.
OCLC, 1983.
12 Horowitz G L and Bleich H L. Paperchase: a computer program
to search the medical literature. In: Proceedings of the 6th Annual
Symposium on Computer Applications in Medical Care, New York, 30
October - 2 November 1982. Edited by B I Blum. IEEE, 1982,
pl045-1051.