SECTION IV
COST-PERFORMAICE ANALYSIS
Jeffrey Katzer*

This section of the report is an extension of the performance study
conducted with last year's version of SUPARS (Section V, 1971 Final Report).
Oar major goal was to obtain estimates of system performance — specificallyrecall, paired with scroe measure of the cost required to achieve particularlevels of performance. A cost-performance relationship for various subcomponents of the system can be determined and compared, thereby providing
valuable input to design considerations for future versions of SUPARS. Last
year, for example, we found that searches using simple Boolean operators
performed better and at less cost than searches using syntactical or positional
operators. This finding affected our plans for user education, budget
estimates and the design of this study.
1.

DEFINITIONSSUPARS/DPS - The current version of soft-ware support for SUPARS is ahighly modified and extended version of IBM \s Document Processing System(DPS) . (6) The input-output interactive user language was developed by the
SUPARS programming staff for interface with the batch-mode DPS search modules.
Since system performance is a function of input and output, it is misleading
to separate the SUPARS components of the system from those of DPS. Thus,
SUPARS/DPS correctly designates- the total operating retrieval system.
Document - As used here, a document is the total representation of one
item in Psyclio logical Abstracts. Usually a document consists of a bibliographic citation and an abstract. Whenever Psychological Abstracts does not
publish an abstract for an item, the SUPARS files represent the document asa, citation only.
Document Data Base(DDB - The DDB is that data base searched by SUPARS/DPS consisting of Psychological Abstracts documents (as defined above.) Thatportion of the DDB used in this study consisted of most documents publishedbetween February, 1970 and June, 1971.l(The size and growth of the total* The author wishes to acknowledge the contribution of several staff members
to this report — especially Miss Sandra Browning and Dr. Kenneth Cookfor completing the onerous duties of search experts, and Miss Margaret
Mucia for the lengthy task of data tabulation.
1

Documents published in March and May, 1970, were discovered to be missing
from the original maclline-readable tapes. While we ultimately obtained
these docianents, they were received too late to be included in this study.

150

DDB available to users for searches is described in Section II of this report.)
As used hereafter, DDB refers to that subset of the total DDB used in thisstudy.
Vocabulary Data Base (VDB)- The VDB is a pseudo-data base searched bya simplified version of the SUPARS/DPS user language. The components of the
VDB are the words in the system's vocabulary — i.e. all non-oannmon words
found in the DDB derived from the free-text processing documents by DPS.
The VDB is called a pseudo-data base because we did not have to create a newdata base as defined by DPS. The comnonents of VDB are stored and usedduring construction and searching of the DDB. Our problem was to accessthis vocabulary as a separately retrievable data base — a complex, but not
too difficult prograinuing task.
Search Data Base (SDB) - The SDB is that data base searched by SUPARS/DPS consisting of previously made SUPARS search inquiries. Throughout last
year's and this year's operating periods we attempted to store all interactions
with the SUPARS system. Because of complex systems problems only a subsetof the searches made in 1970 could be identified and stored. These, plus thegreat majority of the 1971 searches to the DDB make up the SDB. At thetime this study was begun, the SDB consisted of 4,235 searches.
Information Requirement Statement (IRS) -* In this study an IRS is a
written statement of a person's information needs. We assume that the written
statement actually represents tliat need.
Operators - The user language accepts as input one or more keywords(search words) which represent the IRS of the searcher. The keywords may be
combined with Boolean operators (and, or, not); or grammatical operators(e.g. which specifies the desired proximity of keywords, or the truncation
operator — see below.)
Truncation Cfrerator - The truncation operator is part of the user
language. It is requested by typing the characters (?) at the end of a wholeor partial word. Internally the truncation operator retrieves the union of
documents containing any word in tlie vocabulary beginning with the letters to
the left of the (?).
Search Inquiry- A SUPARS/DPS search inquiry is one complete injxit into
the system. It consists of seme system ocnmands, an output specification
statement, plus one or more user chosen keywords combined with the operators
desired in order to represent concepts described by IRS. M l correctly input search inquiries will not necessarily retrieve output. For one information requirement, a user may typically enter several search queries.
Logically Different Search Inquiry - One of the arbitrary parameters of
the SUPARS system is the maximum number of retrieved documents vrtiich can be
typed out in response to any one search inquiry. Using the fuller output
format (LIST REOORD) each search will print a maximum of 10 documents. The
citation output format (LIST BRIEF) will print up to 100 document identifications per search. If a searcher wishes to print more than this maximum,151the search inquiry must be repeated, (although he need not type the key words
ard operators again.) Each of these repeated search inauiries will be counted
as one logical search inquiry- In order to have two logically different
inquiries, the user would have to change one or more keywords or operators.

Estimated Recall - Recall, as a measure of performance, is defined as
the retrieved proportion of relevant documents in the data base. For large
data bases it is difficult to know how many documents in the total database are relevant (which is tlie denominator of the recall ratio). To estimate a recall ratio, we have assumed that each of the ten documents identified as relevant by the writer of an IRS represents 10% of the total numberof relevant documents in the entire data base.
Total Retrieval - This is siirply the number of documents retrieved by
one search inquirv. It can be used as the denominator of the precision
ratio. By using total retrieval instead of nrecision, we were able tosnecify how many documents a user will have to scan through in order to
fin:! the relevant ones. In this sense, total retrieval is a measure ofeffort or cost of labor of the system. In this study, total retrieval is oneof our major measures of system cost. It should be noted, however, that theeffect of this cost, (effort) on users of a particular system, probably
varies. It is not considered here.
Costs - Many factors contribute to the cost of a retrieval system.
King ^4j nas identified the majo£ ones. Many of these factors are installation dependent (e.g. salaries, whether or not an intermediary is used,
whether retrievals are mailed to requester, etc.) For this reason we will
limit our analysis to two measures of cost: (1) total retrieval — whichis reasonably independent of the particular installation, (2) computer costsin terms of CPU time and the number of I/O chainel executions. Though theseare dependent upon particular hardware configurations, comparisons amongalternative computing systems can be made. CPU time and I/O executions areeasily obtained and we chose them because they constitute a common standard
for comparing different computer systems.

2.

METHODThe major objective of the study outlined below is to obtain an
estimate of the cost-performance relationship for the current versions of
the SUPARS/DPS.
At the end of our analysis of the 1970 SUPARS system, we concluded that
oerformance oould be improved by some form of vocabulary control. The VDE
and the SDB were constructed as an initial attempt to "control" the vocabulary, but retain the free-text component of the system.
I The VDB allows a searcher to determine if his keywords are in the
vocabulary. Another option available with the VDE retrieves all vocabulary
items with the same initial letters. We had hoped that this option could be152used to reduce the total retrieval of a search incruiry. If, for example,
a searcher plans to use the truncation operator, prior use of tlie VDR might
identify unrelated words which the truncated keyword would encompass. By
eliminating the non-related words frcm lois incruiry, the total retrieval will
be reduced. This method of using the VDB could also improve the proportion of
relevant documents retrieved by identifying synonym-like2 keywords in the
vocabulary.
The SDB was designed priinarily to aid the searcher to obtain synonyms.2
By typing as input into the SDB a keyword which is not general enough (i.e.
does not retrieve enough documents), the searcher can retrieve all keywords
and all stored searches which contain his keyword. On option, tlie searcher
can have the entire retrieved incruiry printed — including the operators
which link the keywords. Another possible input into the SDB is the identification of a known relevant document; the output would consist of all
searches which retrieved that document. M l of these options were designed
to make the vocabulary and logic of any searcher anonymously available toany other searcher — hopefully increasing a users ability to search the
DDB more effectively.
In order to test the effect of the DDB and the VDB on performance we
conducted a laboratory e^cpferiment very much similar to the one conductedlast year. In the. experiment searching stimuli were obtained by hiring•subjects to write a statement of their information needs (IRS). Uponcompleting his IRS each subject manually searched the published version of
Psychological Abstracts to find documents relevant to his information needs.
The ten most relevant of these documents were used in the study.
Expert searchers read each IRS and used SUPARS/DPS to retrieve 9 of
the 10 documents previously identified as relevant. Each retrieved relevant
document was taken to be an estimate of another 10% of all relevant documentsin the DDB. Experts were restricted in their search inquiries, however, to
one or more of the three data bases. (DDB, VDB, SDB.) Through proper
experimental design we had hoped to obtain clear measures of the relevant
effectiveness of the two new data bases.
a.

Experimental Design

Our original plans called for four treatment conditions (controllingthe data base permitted to be used by a search expert), four search experts
and sixteen IRSs. These variables were arranged in a 4 x 4 Latin Square
design replicated 4 times.

2

The term "synonym" is used here as a general name for all words whichniay refer to the same class of objects. True synonyms fit in this classas do those items listed under a "See also" heading in an index. For
example, synonyms of "human" which may help a user, are male, female, boy,girl, child, adolescent, student, subject, teenager, etc. — plus allof their plurals.

153

For various reasons,3 we had to delay the start of the experiment andconsequently one of the experts liad to resign fron the study because oforior craTiruttinents. The study was redesigned with three treatment conditions,
three experts, and 15 IRSs arranged in a 3 x 3 Latin Square replicated 5 times.
This design is presented below in Table XLI.
TABLE XLI. EXPERIMENTAL DESIGN

IRS

SearchExpert

8

7

10

15

12

16

9

2

4

3

13

14

11

5

1

A

S

D

V

S

D

V

S

V

D

S

V

D

v

S

D

B

D

\T

s

D

V

s

V

D

S

V

D

S

D

V

S

C V

c.

D

V

S

D

D

S

V

D

S

V

S

D

V

In Table XLI three treatment conditions identify the data bases eachexnert was permitted to use for a particular IRS: 'Df means the exnert was
restricted to the DDB, 'V means the exnert could use the ^ B as well as theDDE, 'S* means that the SDB and the DDB could be used.u
b.

IRSs

Subjects were needed to produce the required IRSs for the experts tosearch. They were contacted through advertisements asking for people withinformation needs in the social or behavioral sciences. A brief descriptionof the fifteen subjects whose IRSs were used and of the IRSs themselves ispresented in Table XLII.
Each subject was asked to write a detailed description of one of his
current information needs. He was not told to construct his IRS in terms ofthe content of Psychological Abstractsf in fact, the data base was not mentioned at this stage of the experiment. The instructions given to the subjects
are presented in the Appendix. In general, each subject was asked to state
his information needs in such a way that we could go out and get the information for him.
3

Notably, the delays caused by the computing center's conversion from the
IBM 360/50 to an IBM 370/155 before this experiment was Planned to be
carried out.

^In the original 4 x 4 design, the fourth treatment would have allowed asearch expert to search all 3 data bases. While it is unfortunate that thiscondition is missing frcm the study we can still estimate the effects of the
4 condition — if it turns out to be additive.

154

TABIE XLIICHARACTERISTICS OF SUBJECTS AND IRSs

1 Academic
IRS | Sex Status

Dept

Topic

i Number of
Relevant
Documents
Found

T~ 1M

Fac

Bduc

Vocational choice proaess of the disadvantaged

18

2 i F

Grad

Educ

Behavioral Theory, counseling, andeducation

43

3

M

Fac

Psy

Activity level of animals in response 1
to stiinulation

73

4

M

Grad

Law

Change in suburbs and voting patterns

16

5

M

Grad

Psy

Efficiency and disoriininability of
tests

17

7

F

Undgr

Nursing

Inaonpatability in marriage

33

g

H

Grad

Psy

Comprehensive services for unmarried pregnant teenagers

43

9

M

Grad

Psy

Development of human relationships in T group

42

10

M

Grad

Educ

Education of Exceptional Children
and Instructional Technology

18

11

M

Grad

Bduc

Instructional t.v, in education

65

12

M

Grad

Psy

Effects of intercultural contact
and bilingualism

13

13

F

Grad

Psy

Errorless learning in adultproblem solving

19

14 M j Fac

J

Lib Sci

Attitude of people confronted withtechnology

1715

M j Undgr

Psy

Police intervention in social/sexualdeviancy

54

16

M

Psy

Review of federally supported youth
work programs

42

Fac

155

c.

Identification of Relevant Documents

Upon completing the writing of his IRS, each subject was given the
February, 1970 — June, 1971 issues of Psychological Abstracts. These issues
were chosen for two reasons: (a) they constituted a data base of approximately
the same size as that used in last year's study, and (b) they were the mostrecently received issues of Psychological Abstracts (the IRSs were collectedin early Fall of 1971) which WDuld also be included in the DDB.
Each subject was asked to find the entries in Psychological Abstracts
which he judged to be relevant to his IRS. A subject could complete thistask in any manner he chose. We suggested that he at least look at eachmonth of the journal — but there were no directions specifying how to findthe relevant documents or how many to choose,5The final task for each subject was to rate each of the documents chosen
into terms of their relevance to his IRS. The 10 documents rated most
relevant were used in this study.6
In last year's study we used the 5 documents rated most relevant andthe 5 rated least relevant by the subjects. Upon analysis, we found a better
cost performance relationship for the most relevant documents. Because of
the plausability of this finding we did not think it necessary to replicatethat portion of the study this year.
d.

Search Experts

Three SUPARS staff members served as search experts. All had considerable
experience with SUPARS: they WDrked on last year's project, helped in thedesign of this year's system, made numerous searches with the system and prepared materials which taught others how to use the system.
Each expert was given the 15 IRSs and the identification (Volume and
Abstract No.) of the 10 documents judged relevant by the subjects. The
experts were instructed to use SUPARS as effectively as possible to retrieve
any 9 of the 10 documents.

5None of the fifteen subjects completed this test without identifying atleast 10 documents he considered relevant. If this had not hapnened, hevrauld have been asked to continue the task.
6

It was sometime after the subjects completed their job, when we discovered
the two gaps in the DDB — see footnote #1. At that time it was necessary
to check each of the ten documents paired with each IRS. If the chosen
document fell in one of the gaps of the DDB, the next most highly rated
unused document not in the gap was substituted.

156

Their job was to use the system to maximize the retrieval of relevant
documents with minijral total retrieval. To achieve this goal they werepermitted to use any of the SUPARS/DPS operators available. The only
restriction
the experts were to observe vas the treatment condition —the data base combinations permitted for each IRS. (Experts were permitted,
but not required, to use the SDB or the VDB when they were searching under
the 'S' or 'V1 conditions (see Table XLI.)*e.

Procedure

Each expert was given a packet of 15 IRS — each paired with a list of
10 documents judged to be relevant to that IRS. The order of IRSs in eachpacket was randomized. Unfortunately, however, it became essential for theexperts to begin their work before the 1971 searches were loaded into the
SDB. The loading of searches took nnch longer than expected; consequently
the three experts searched their 5 IRSs under the 'S1 treatment condition
last. That is, the effect of the 'S1 treatment condition is confounded witha practice effect. This will be discussed, at more length, later.
All of the experts' terminal interactions with SUPARS were saved. For
each IRS, the first 9 relevant documents retrieved ware marked, and thecumulative number of non-relevant documents retrieved before each relevant
document was counted. In this way, 9 measures were obtained for each of
the 45 expert-IRS combinations. The mean number of non-relevant documents
retrieved before the first relevant document was retrieved, (across all fortyfive expert-IRS combinations) provides the estimate of total fall-out, (orcost of screening) at the 10% recall level.
Wfe also counted the number of search inquiries made because the computer
cost of running SUPARS depends mainly upon the num^oer of inquiries, not the
number of documents retrieved. CXir second measure of cost (CPU time andI/O executions) is obtained from data collected by STATPAC. For each searchinquiry an expert enters for an IRS, STATPAC stores the CPU time and I/O
executions (among other things )> Summing these through each relevant
document retrieved produces the second measure of cost.
f.

Analysis

The Latin Square design was analyzed as a cross-over design. The
partitioning of sources of variation is shewn in Table XLIII. An analysis
of variance procedure, followed by hypothesis testing or interval estimation
will determine whether or not there is a significant difference among the 3treatment conditions. There probably will be real differences among experts
and IRSs.
•Experts were given the option of using SDB under the f S' condition or the
VDB under the 'V1 condition.

157

TABLE XLIIISOURCES OF VARIATION AND DEGREES OF FREEDOM IN CROSS-OVER DESIGN

Sources of Variationdf

Among Treatment

2

Among Experts

2

Among IRSs

14

Residual1

26

Ibtal

44

1
Ihe degrees of freedom for residual is not as large as desired. Ouroriginal design called for four 4 x 4 Latin Squares. The degrees of
freedom in that design WDuld have been more satisfactory.
3. RESULTS
The findings of this study are presented in 4 sections: First, abrief look at the treatment variable to see if search experts actually used
the tvo optional data bases (VDB, SDB) as searching aids. Second, anotherlook at the dependent variable, computer cost. Third, we will compare thecost-performance relationship of the different treatments. Finally, anattempt will be made to compare last year's and this year's systems in termsof cost-performance.
a.

Use of Optional Data Bases

As noted earlier each search expert was assigned five IRSs he could
search with the VDB as an aid. Five other IRSs could be searched with the
SDB, and the remaining 5 IRSs could not be searched with any aid — only the
DDB was permitted to be used.
Since any differences between the use of different data bases isprobably cumulative, Vve only need to examine the total number of searchinquiries made to achieve the 90% recall level. The search frequencies
and average use of the three data bases is presented in Table XLIV.
The DDB was necessarily used in all forty-five IRSs because it contained the documents to be retrieved. Of the 824 logically different inquiries
158

CO
in

m

en

3

oo

•9
en

i

S MM

J
00
o¥>

uu

CO

o\P

o
o

o

CM

*S3 6

-I

*W 0 , 0 ,

CM CM

*y

oo

CM
CO

rrt
r-H
CO CM

VD

in

.c

CO
CO

CO
CO

§ §

CM

00

g g
^•s

(0

(0

(f)

te

00
00

C\00

•6 -6

£3i

%

u-t
U-4

v./

w

v-, _ ^>

log calltestis

•H

Q SI
H

IM

tf

en

<Tj 10

4J

m

in

O

in

U

%
co

38

OQ

QQ

CQ

M-J

3 4

in

H (]1 S8 g 83 a s
159

>-

call

CM

iC

-H

S$
4-f
M-4 4-»

0 -H

5&
5 CO

c
•H

:s
3 as

3 ,#*

made (to achieve a 90% recall level for 3 experts each searching 15 IRSs) v
691 (84%) were to the DDB. The remaining search inquiries were made to the
VDB (11%) and the SDB (5%). Fifteen IRSs (5 for each expert) were searched.
under the VDB condition. Two experts each chose not to employ the ^/DB
for one of their IRSs, Averaged over the 15 IRSs in which the VDB was permitted to be searched, experts chose to use it over 27% of the time.
The SDB had less usage. Search experts found at least one IRS in which
they did not employ the SDB. Moreover, average use of the SDB dropped toapproximately 20% of all inquiries nade by the experts under this treatment
condition. In each of the 5 times the SDB was not used, the experts achieved
the 90% recall level with fewer total searches than the othet experts. Acrossthese 5 IRSs, the median difference between the fewest number of searches
arri the next fewest was 7 — i.e. experts who chose not to use the SDB
achieved a 90% recall level with an average of seven fewer searches than
the next best performing expert.7 Seert in this light, it appears thatexperts who are using a wsrkihg strategy for retrieving relevant documents
may not have the need to use the SDB for help. Perliaps they would turn to
the SDB when and if their list of searching strategies became depleted.
However, when asked, the search experts did not recall consciously lurking
in the manner described.
There is an obvious need for further study of the use of the SDB; more
will be discussed in this report. At this time, it is sufficient to pointout that the SDB was infrequently used. The dilenroa affecting an assessmentof the performance of the SDB is apparent: (1) If the performance of the
SDB does not differ from that of the DDB it may be due to its infrequent
application rather than any intrinsic uselessness on its part; or (2) if tlie
performance of the SDB does differ from that of the DDB it may be due tothe fact that all 3 experts searched their SDB IRSs last -- so an increasein performance because of practice, or a decrease in performance because of
fatigue or boredom is possible.

7

A similar pattern is apparent, looking at the two IRSs which did not esmnloy
the VDB. One expert, not using the VDB achieved the 90% recall level withfewer logically different searches than either of tlie other experts. The
other expert, achieved a 90% recall level with the second fewest numberof searches — but the total number of searches made (7) was small in
absolute terms. The rationale postulated in the text supports the nonuse of the VDB as well as the non-use of the SDB.

160

b.

Gomputer Costs

There are 4 different, but related, measures of computer cost collectedby ST&TPAC. (1) The CPU time is the amount of processing required to execute
one search. This includes input and output processing plus accessing theinverted file and using the search operators. CPU time will be presented
in seconds. (2) The EXCP count is a measure of I/O activity plus any
internal channel executions.8 It will be given as a frequency count.
(3) The number of completed searches is another indicator of computer costs.
In general, only logically different, completed search inquiries werecounted. (4) Finally, we have combined the CPU time and the EXCP count fora search into a dollar figure. How to combine these into one index of costis somewhat arbitrary • We. have chosen to use the rates currently being
charged for the university's IBM 370/155: $360 per hour of CPU time plus
$1.50 for every 1,000 EXCPs.9
Our plan was to retrieve frem STATPAC these four cost measures for allof the completed inquiries made by the search experts. Unfortunately,
several disc tracks were irretrievably lost, making it impossible to get
accurate post figures for searches in 23 IRSs. Eleven IRSs were missing
cost figures for all;searches, 12 IRSs had some cost figures, and the remaining 22 IRSs had cost figures for all searches. Two of the group of 12
IRSs with partial cost information had less than 20% of their searches
missing. The missing costs in these two IRSs were estimated using the mean
cost per search (See Table XLV). This raised the number of IRSs with complete cost data to 24. These are divided into seven complete IRSs searchedunder the 'D' condition, 12 searched under the 'Sf condition, and 5 searchedunder the 'V1 condition.10 While each expert contributed at least one IRS
8

The operating system's teleprocessing package executes one EXCP for every
carriage return on input and output — i.e. one EXCP for every line (regardless of size) of input and output. In addition, there are two EXCPs
for every keyword in a search inquiry (one to the dictionary and one to the
vocabulary.) Plus, there is approximately one EXCP for every use of a label
as a keyword in a search inquiry. It is impossible to determine the exact
nuuber of EXCPs without using STATPAC or some computer monitoring system.
Truncation operators, the length of internally generated temporary computation strings, and the size of buffers have an effect on the number of
EXCPs.

9

CPU time and EXCP count are only twD of four components which determine
the total oonfxiter charge for a search at Syracuse University. There isalso a charge for (1) core residency (i.e. number of bytes of core required per hour); and (2) the length of time a terminal is connected tothe computer. Neither of these are included as cost figures in thisreport as they contributed less of our understanding of the operating
characteristics of SUPARS/DPS.

°1hese 3 experimental conditions refer to the design outlined in Table XLI.
Each condition was a restriction upon wliich data base an exnert could

161

4->

TJ 0

Stand
Devia

\M

I

CM
O

•

o

o
•

O

o

s

•

1
8

•K

rH

s-

i

rH
O

•

r•
rH

$

I
i —

00

0)

o

3

o•

to these data bases
ee ** and ***£iStandard
Deviatio]

C!

!
i
1

m
•

CO

•

O

•H
•
O

to to
•P 1
rrt
E to
•r-J ••

i M
*

*
; CQ

O

I A
1 w

.

S

•KT
CM

•

O

in
•^*
•

CO
r^

8
Q

1 1
•H

H

in

10

0

to

to

I

as

o

M

i
8

4-* CO

•

•H

0

%

u

CO

5
&

TJ 0

1*
n3 rd

r-\

C\

r-

vo
•
in

o\
•

<j\

CM
00
CM

o

•

i fie

*

8

o

Q) 0

§

in
•
m

*^4

CN

•
ro
•^r

in
00

VD

rH

•
m
rH

•

rH

to

u Is
D1
8 8^
-P

M

$8

oQ) M

S

O)

0)

I
c
0)

|b J° | °
4J

S3 a.

Doll
Cost

!

CO

r-l CO

w

1^

P

I

co
c^
•
ro

fe

g

I

•H

c

Stan
Devi

03

1

!

CO

! «
i

A

o

i

^
-P

i*

U

M
0

•8
-J

H

s

3

$ M
O rH

1 a a.

162

to a)

CO 0)

3%

3S

-P <D

4J 0)

E CO
3

CQ

3S

35 *85*

under each of the tliree (conditions, these 24 IRSs aire not counterbalanced
across IRSs, treatment conditions, and search experts. Tlie cost findingspresented represent less controlled estimates of cost than would be* desired,
and the reader is strongly cautioned to study the acccnpanying text material
which will specify the basis for all estimates.
Table XLV presents cost estimates to the 3 data bases. All three measuresof cost decrease as we go from the DDB to the SDB to the VDB. • Since the DDE
and the SDB are true data bases as defined by DPS, the differences in costshould be solely due to the differences in size of each data base. If the
SDB contained as many searches as the DDB contains documents, we would
expect the cost of searching each to be about equal.
The VDB, however, is not a true data base; it is merely part of the inverted file of the DDB. Searching it requires a much simpler set of computations. Cost for searching the VDB should remain quite low unless thenumber of words in the vocabulary increases by several magnitudes. The consistency of the cost of searching the VDB is shown by the very small standard
deviation.
c.

Cost-Performance•Hie relationships between 5 measures of cost and performance (in termsof recall ratios) for each data base are presented in Tables XLVI-XLVIII and
Figures 10-11. Extreme care must be taken in interpreting these data. In
Tables XLVI-XLVIII, the first column of data contains cumulative total retrievals — that is, the number of irrelevant documents retrieved before thenext one of the ten relevant ones were found. This data is based upon all 3
experts searching 15 IRSs — 45 IRSs in all.
The other columns of data in Tables XLVI-XLVIII, though accurate, areless reliable, because they are not based upon a complete set of experimental
data. The costs of the DDB were based upon 7 IRSs, those of the SDB werebased upon 12 IRSs, and for the VDB, the data came from five IRSs.
The average cost per logically different search in the DDB is $1.16
according to Table XLV. That figure can also be computed from Table XLVT
by dividing the dollar cost by the number of searches (i.e. 18.39-rl5.85).
This correspondence between the two tables does not hold up with the othersearch. Under 'D' only DDB could be searched, under 'S' both SDB and DDB
could be searched, and under 'V1 both VDB and DDB could be searched. It is
important to distinguish between the cost of searching a data base and thecost of searching under a specific experimental condition. Only under the'D1 condition are they the same. Table XLV presents cost estimates forsearching the three data bases. Tables XLVI-XLVIII present cost estimates forsearching under the three treatment conditions.

163

j

!

1

-P
C
0)

M-l

a

- H CQ

Q Q

1!
•H Q
•H

w

en

•s-8

o
o
^r•

rm

•

<*?

^r
rH

rm

vo•

vo•

"d1
rH

CO
CN

•

o•

00

r-i

^
r-^
CM»

r*

rH

rH

in

en•

m

CO

•
"5

rA

I
••

Q

•*

i
1B

M

M

4J

•H en

en
<D

8

! -6 rH&
^ rH

00
CN

rsj
r-i

o
o^

rH

V£>

in

cr.

^

m

LO

vo

00

9a

rH
CM
rH
rH

rm

«*
rH

r*
^r

ON
CO

en
en
pL

in

CO
rH

in

r-i

rH

8

rH

«H

jjj

•P

I

* s

1

8

Bas

H3
0)iL>-

V)

5

rH
VO

in
1

-

in

r-

00

.H

rH
"00
rH
CN

ro
O

^
CM

^r
en
en
en

*?
Is*

o
TT

<*r

XT
CO

CO
VO

"«*
in

K*

m

VO
rH

vo

-P

c
flj

1
<u

8

(ii
C/j

i

00

00

o

rH
rH

rH
CT>

00
VD

rH •

^
c*•

en•

VO•

00•

C^•

CN

CM

CN

CO

o•

rH

00
in

m
cc

rH

CN

o

CO•

en•

*r

KD

r^

VO•

S
o

rH
Q4

00

O

Pn

1
s

i

*•

1

E *

»'

eved

DOCU

1 c

-H

1 *S it
•

K

1
r-

O

r-•

*T

CN

c^
(N

vo

•

en
in

ro
o^

O
CN

ro

r r•

m•

•
**
^
r~i

r-i
VO

S

•^r

r*»

ro
VO

o
^r

r-*
00
00
r-\

o

**
i n•
o
en
en

r*

CO

VO

in

o•
o

i n•

CN

in

m
o
r*

1
en

•H

rH

8

rH

rH rH

S2

1 g«i

<*>

O
r-i

c#>
O
<N

0*1

O

ro

dP
O

<*P
O

•^

in

164

4P
CD
VO

<#>
o
r^

en
ctf>

O
CO

dP
O
<Ti

s
*

s
CQ

(N

vD.
VD

CM

CM

CO

O

uO

VD
CP»

m
co

in

O

in
oo

8
3

CQ

a

1

1/1
CM

o
o

in

o
o

vo

in

VD

VD

CO
VD

en

CO

ON

in

CM

CM

CO

CM

o
m

co
oo

oo
in

CO

oo

CO

CQ

a

S

CO
CM

VD
O

CM
CO

•

r-i

CO

CO

«
VD

8.

E 8
CM
CO

CM
CT*

cr

«H

CM

in

K3*

^r
^t
rH

I***

CO

r*

*
r
•n

r
*
ON

CM
.H

VD

CO

H

H

in
CM

«H
CO

o

CO

c*

o
>. r
in
00

b
•8
VD
CFs

CO

in

VD

o%

CO

cr>

o>
in

«y•
CM

CM

co

o

in
^

CM

o

co
co

O
tH

^
rH

CM
CM

o
o>

VD

•

o%

h

o

CO

in

m
m

82

o

o
CM

oo

VD
CO

CO

O
CO

165

r-

o

O
CM

o

00
00
VD

m

•

<#>
o
in

CM
CO

oo

CO

in

00

o

O

O

VD

r-

co

only

™

CO

CO

vc

o
o

CO

o
o

Q

rH

•

CM

•

co•

i n•

CO

o

a
CO i
(D
8

3s

O

o

•

—r

o
o

O
O

o

o

CO

CN

o>•

o•

o•

i n•

rH

rH

c
o

—

CQ

11 g

3J

£

M-l

CQ

H

O ,0)

8
S

14-1

t/J i • *u

pQ

o
o

«

VD

O

^r

•

o

o
o

o
^r

vo

•

VD

•

o

^

•

<N
rH

rH

o

o

**

VD

9

CO

•

o

8

o

CN

•

VD

rH

rH

r*
r•

CO
VD

VD

•
H
o

•

CO

•
o
CN

LO

•S
>

0)
CD

e

•6
&

CO

1°
P

(0
VO
CN

B

•H

IS
AH

in

o>

vo

r-

rH

rH

•

•

CO

co

o^

CO

•

•

CO

CO

co
rH

g

rr
o

M

|
!

«tf

o
C

i

|
in

in

rH
1

|
H-»

1
br

i
aVO
LO

-

H
1
ta

CT}
VD

r-

rro

CO

m
c
o
CN

CO

^r
H
ro

CN

8
5
I

1

ON
VD
KT

r*
0>

CN
VD
CN

cr.
CN

co

00

-tf

in

1

CN
rH

ro

CN
00
•

1

C

VD

•

c\

in

CN

O

cr»

ro

^

i

P

1
1

i

'

•

"d•1

i n•

<J\
VD

-r

r-

O

VD

<T.•

V

r-i

VD
CO

<»

Ch

ro

r-«
^

VD
o

c^
TT

in

VD

CD

i

1

I

r

_______
>
i
!

|

i § *

!

3

' 0 >

fi-S

I

;!

co
00

•

CO
VD

m -P

°s
Is

(D

o

VD

ro•
VD
rH

m

f-{

•

CN
«T
CN

ro

•

CTN
CN

r*»

Ch

o

-^
VD•

VD

rH

rH
CN
r-A

o>

o>•

r*

r-

<tf

CN

rrH

•

•
^
rH

in

o
CN

r-i

O
00

•

ON

rH

ro

i

8

HH

Recal
Level

CO

CO

<*o
O
rH

dP
O
CN

dP

C*>

dP

<*>

O

o

O
in

o
VD

co

166

•^

df
O

r»

<#> dfi
O
CO

O

o>

data bases. For the SDB the average cost per search is $0.13 (Table XLV),
but according to Table XLVTI it equals (7.31-rl3.58)=$0.57. The explanation
for this difference lies in what is being counted in each table. Table XLVdata is based upon searches to the SDB only, while in Table XLVII thedata comes from searches completed under the 'S' condition. Under the 'Sf
condition, an expert was permitted to use the SDB as an aid in retrieving a
document, lb achieve this goal the expert was required to use the DDE toretrieve documents. The data in Table XLVII is taken from 12 IRSs under the'S' condition. Four of these did not include any searches to the SDB. Of
the 163 searches only 37 were to the SDB while 126 were to the more expensive DDB.
A similar situation occurs with the VDB. In Table XLV the average
cost of a search to the VDB is $0.08. But from Table XLVIII it is $0.72(15.04T20.80). Again this difference occurs because Table XLVIII includes
73 searches to the more costly DDB as well as 31 searches to the VDB.
One tentative finding of this analysis of the average cost per searchis that use of the VDB decreases the need to use the more expensive DDB andis, therefore, a valuable searching aid. This is evident when we look at
the last..column of Table XLVI and Table XLVIII, and vdien we cxsnpare theaverage Cost. Sihce tinder both condition searches to the DDB had to be
made, the only way the average cost per search to the VDB ($0.72) can beless than that to the DDB ($1.16) is if the use of the VDB reduces the needto search the DDB. At present this is a tentative finding because it isbased upon incomplete unbalanced data. However, as we sliall see, otherevidence of a more reliable nature supports this finding.
A similar argument can be made for the SDB, but because the 'S'
condition was confounded with a learning or practice effect, it is impossible
to tell how valid it wuld be.
Another way of saying this is that the average number of logicallydifferent searches at all recall levels is about the same for the three
data bases. The most they differ is apprximately seven searches (at the
90% level.) While the average number of searches seems comparable, theother three estimatesof computer costs differ widely across the 3 experimental conditions. In general, the 'D' is the most expensive, the f S' the least
expensive and the 'V extends the full range between them. This generalfinding is true for all three measures and across the nine levels of recall.
Figure 10 presents graphically the cost-performance differences betvaeen the
3 in computer dollar costs.
Upon inspection, we determined that the radical decrease in performanceof the 'V condition between the 30% and the 50% recall levels was due
solely to one IRS. This may be artificial; if the remaining 10 IPSs in the'Vv condition were available, the performance decrease could conceivably beaveraged down. A similar pattern emerges if we plot either of the othertwo computer cost measures.
It should be noted that all of the computer cost measures of the 3
conditions (with the possible exception of 'V') appeared to be log-normally

167

ESTIMATED RECALL RATIO

10%

20%

^0% 40% S0% r,0% 70*

80%

90<

90.00
80.00
70.00
60.00
50.00
40.00
30.00

20.00

*D' Condition*

PER

jg)'V

g

8
>
H

10.00
9.00
8.00
7.00
6.00
5.00

Condition**•S* Condition***

4.00
3.00
&l
•

2.00
- -*

s^
1.00
*Based upon 7 IRSs**Based upon 5 IRSs***Based upon 12 IRSs

Figure No. 10. Conputer Cost per Search To Achieve Nine Levels of Recall:
A Conparison /\mong Three Treatment Conditions

168

ESTIMATED RECALL RATIOS
10,0009,000
8' 000
7,000
6,000
5,000

10%

20%

30% 40% 50% 60% 70%

80%

90%

D'Condition*

4,000
3,000
2,000

1,000
900
800
700
600
500
400
300

200

100
90
80
70
60
•Based upon 15 IRSs in Cortplete Study
Figure No. H » Nuntoer of Documents Retrieved (Total Retrieval) to
Achieve Nine Levels of Recall: A Comparison Among
Three Treatment Conditions.

169

3.50

As one logical plus
two redundantsearches

4.00

2.50

2.00

1.50

As one logical search
1.00 0

100

200

300

NUMBER OF DOCUMENTS RETRIEVED*This splits the mean cost per search of $1.16 (Table XLI) into
$1.12 for input processing and searching, and $0.04 for printing
100 LIST BRIEF citations (25 lines of output).
Figure No. 12. Two Methods of Computing Dollar Costof Searching on DDB* Using LIST BRIEF Output Format

170

distributed. Since the computer cost estimates are derived from a portionof the total experimental design, it was impractical to test this finding
statistically. For the same reason, we did not attempt to determine the"significance" of the differences between the data bases on any of the computer cost measures.
This is an appropriate place to explain the rationale underlying ourreliance, this year, on logically different searches. As noted in thedefinitions, logically different searches are somewhat independent of tlie
arbitrary system parameter which specifies the maximum amount of printed
output. In this sense, logically different searches represent a good estimate of the minimum computer costs of operating SUPARS/DPS.11 Figure 12
presents a plot of costs as a function of number of documents retrieved bya search. For each logically different search there is the initial costof input and processing, plus a steady slight increase as a function of thenumber of documents printed. Under the current system's parameter with
LIST BRIEF output, the user is required to pay for the input and search
processing after every 100 documents are printed. Costs under the current
system can be described as a step-function where the size of the step depends
on CPU processing, and the number of steps for a given amount of output
depends upon an arbitrary parameter of the system. By counting logicallydifferent searches, we attempted to simulate the minimal cost conditions inwhich the 'output parameter is set-to the maximum.
Up to this point in the discussion of cost-performance characteristicsof the 3 experimental conditions, we have been solely concerned with various
estimates of computer costs as our dependent variable. The findings havebeen useful to get some idea of the cost of making different types of SUPARS/DPS searches. Comparisons among the conditions and data bases were considered tentative because of tlie incomplete nature of the data.
The major dependent variable, number of documents retrieved, has notas yet been discussed. This variable is based upon the complete experiment,
and except for the interpretation of the performance of the SDB, is morereliably interpretable. Total retrieval for the 3 data bases is presented
in the first column of Tables XLVI-XLVIII. It is the same measure used toevaluate the performance of last year's version of SUPARS.
Figure n graphs these data on a log-normal grid. A statistical analysis
supports an inspection of the data; there is a difference betaken the treatment means at the higher recall levels (70%, 80%, 90%.) The analysis ofvariance sutirary tables are presented in the Appendix.
Figure 11 supports the notion that use of the VDB as a searching aid
materially helps inprove the cost-performance of the system. This is particularly true at the higher levels of recall. Figure 11 also suggests
the value of the SDB, as it has the best performance curve. However, because
This is a slight underestimation as we need to add $0.0015 for every lineof output above that permitted by the system's operator.

171

of the lack of control over the 'S' condition, a plausible alternative
explanation is tliat search experts learn how to perform tetter over time.
It is unfortunate that this study is incapable of adequately determining thevalue of the data base of searches.
d.

Comparing the Cost-Performance of TMD Years of SUPARS Operation

Two comparisons are of interest here. First, a comparison of costperformance in terms of computer cost. This is useful because the tvro years
of SUPARS operation included a major change in computer svstems. Second,
a comparison of cost-performance in terms of total retrieval. This will
give us some idea of the stability of the DDB, IRSs, and experts1 ability
to search efficiently.
As would be expected, many changes have occurred between Fall, 1970
and February, 1972, which will affect this comparison. The major changes
are outlined in Table XLIX. As we can see from the Table, it will beiiipossible to make straightforward comparisons. Three factors affect thecomparison of computer cost; the third one affects the comparison of total
retrieval; (1) the charges for computer service (from 360/50 to 370/155
increased 80% for CPU use and 36% for EXCP activity, (2) the performancecapability of the twD computer systems differ,12 and (3) as noted in thelast three columns of Table XLIX, there are differences in terms of what
searches were counted and how EXCPs were treated.
In addition, there are several other differences between the two
operations which may have an affect on any comparison. The size of the
DDB grew from 35,874 in 1970 to 46,828 in 1972. Moreover the free-text
vords were coded differently: 16 bits per word in SUPARS I, and twice thatin SUPARS II. This coding difference permitted an increase in the number of
unique wDrds in the inverted file from 64,534 to 106,702. The extent towhich these changes affect any comparison between the two systems is notknown.
To obtain a rough estimate of the average cost per search to the DDB
from this year's data, which would be comparable to that of SUPARS I, we
recommend the 3 computer cost estimates of searching the DDB reported inTable XLV by counting all 246 searches in the 7 IRSs. Of the 246 searches
111 are logically different (reported in Table XLV) and the remainder arethe "redundant11 ones.13 Table L gives these cost figures.
2

Depending upon the nature of the data and the type of operation being performed, the 370/155 is 3-4 tinres faster than the 360/50 in terms of computation. The 3330 disc devices have a transfer rate of 2.6 times fasterthan the 2314s. Seek time across a cylinder is 50% faster and time withina track is 33% faster in the newer discs. We cannot determine what thisincrease in ocmputer performance means to SUPARS specifically. As a data
retrieval system, the improvement in disc performance probably affects
SUPARS more strongly tlian raw computation power.
3
Since these make up 7 complete IRSs the relative proportion of logicallydifferent searches to "redundant" searches should be reasonably representative of the use of the DDB.

172

H

3

Mh*lsa

8 £6

0)

W

M w M

(D

o

•

•H - H8 a a .3 as
1
843

tp • 4J UW*

U § Q) 0}

HI

B

SB J

£8S

mi

<3j ^ ^o^e^

U-4

° 0 «0S8.8- a

<D

m

i

in fi
in

3

r- Q
m
M CO

eg

i

§

O
1^

as
M

fn

iH

CM

Cfl

i
173

•8 a

11

5

9§
cin
</*

TABLE L
ESTIMATES OF THE COMPUTER COST OF SEARCHING TODDB COUNTING ALL ENTERED SEARCHES (N=246)

MeanStandard
Deviation

CPUSeconds

6.45

5.79

EXCP
Count

493

290

1.38

1.00

DollarCost
Using 1972
Rates

174

ESTIMATED RECALL RATIO

10000
900^
8000
7000

10%

20%

30%

40% 50% 60% 70%

80%

90%
SUPAR1 II,
TABLE XLVISUPARf I*

6000
5000
4000
3000

2000

1000
900
800
700
600
500
400
300

200

100
90
80
70
60
*Based upon SUPARS I data averaged at 9% simple and 91%
advanced levels of search logic.
Figure No. 13. Number of Documents Retrieved (Total Retrieval) to
Achieve Nine Levels of Recall: A Comparison Between
SUPARS I and SUPARS II Searches to the DDB

175

The costs in Table L are scmewhat liigher than those in Table XLV. Thisis likely due to the additional internal processing required of many of the"redundant" searches. Search experts frequently used redundant searches toskip quickly down the list of retrieved documents. To do this they added totheir search query an additional carmand specifying, in effect, how far toskip. This additional corrmand was probably the.cause of the cost increases.1
The estimated mean cost per search inquiry last year was $0,589 forsearch inquiries earploying simple logic, and $1,097 for inquiries using
advanced search logic.*5 A search inquiry employed the simple level of logic
if it only used the AND or OR Boolean operators. As used by the search experts in last year's study, the advanced logic level included, for the mostpart, search inquiries employing the NOT and the Truncation Operators as wellas the two simple level operators.
In order to determine the type of logic used by search experts thisyear, we counted all 293 inquiries completed under the 'D' condition. Simple
logic operators made up only 9% of all the searches, and advanced logic
operators appeared in the remaining 91% of the inquiries. Therefore, a hypothetical search inquiry to last year's DDE which voald be comparable to thelogic used by the experts this year, would be 9% simple and 91% advanced.
The cost of such a hypothetical inquiry in terms of the rates charged on the
IBM 360 would be $1.05.
The $1.05 represents the cost for an average search inquiry to last
year's system which corresponds to the type of inquiries made this year.
Our goal is to convert this figure to one based upon the IBM 370 charge
rates (see Table XLIX). When this conversion is completed we can compare
it with the actual charge for an average search to the DDB of SUPAPS II,
which is $1.38 (Table L ) .
To convert the $1.05 to one determined according to the IEM 370 rates,16
we increased 40% of $1.05 by 80% (the rate change for CPU time,) and 60% of$1.05 by 36% (the rate change forEXCP.) The result of this arithmetic gives
us an estimate of the cost per search inquiry last year, computed at thisyear's rates: it is $1.63. This year's dollar cost of $1.38 is about 14%
lower than last year's.17 Most of the increased efficiency this year is^There is a minor reduction of cost of redundant searches because the entire
search query does not have to be retyped. This decrease in the number of
input EXCPs is too small to have a noticeable affect.
5

These figures came from Table 7 of Section 5 of last year's Final Report.

6

7

The dollar oost of an average search inquiry at last year's charges can be
broken down as follows: 40% of it is due to CPU time and 60% is due to
EKCP activity. To compute this corresponding rate for this year, we merely
increased the appropriate portion of the dollar cost by the percentage rateincrease for CPU and EXCP.

Ihe arithmetic did not take into consideration t*;o factors which raise theconverted cost for last year slightly above $1.63; first, that last year's
176

probably due to the new liardvare configuration (see Footnote 12 J
We can compare the two SUPARS systems in terms of total retrieval.
Figure 13 graphically demonstrates the equivalence of the two systems when
number of documents retrieved is used as a measure of cost. This suggeststhat on the average, experts, IRSs, and the DDB did not change between thetwo systems in any vay to seriously affect total retrieval at any level of
performance.
Finally, we can compare the best cost-performance relationship of
SUPARS I, with SUPARS II. In this year's study experts did not choose torestrict search inquiries of the DDB to the simple level of logic — which
had the best cost-performance relationship last year. In fact, only 9% of
their inquiries to the DDB used at most simple logic operators. Given thesestrong preferences of the experts, it would be worthwhile to see how use of
the SDB and VDB improved the efficiency of searching SUPARS II, comparedwith the optiiral simple logic strategy. Figure 14 plots these curves.
Figure 14 duplicates Figure 11 with the additional cost-performance curve
illustrating the most efficient version of SUPARS I. Without testing thedifference among these curves, it appears that experts searching under the\D' condition did not.perform as efficiently as possible. This decreased
efficiency, however, can be offset through judicious use of the VDB andpossibly the SDB — especially at the higher tefcall levels.

4.

DISCUSSIONa.

The VDBThe remarkable finding of this study is the discovery of the power ofthe VDB. Though many others18 have called for an on-line thesaurus with interactive retrieval systems, we \*ere pleasantly surprised at its cost andits cost-performance characteristics.
Since our VDB is, in essence, a portion of the inverted file of the
DDB, it does not take additional space. There are some user interactioncommands to and from the VDB, plus searching routines which had to be
written to access the inverted file, but those are relatively easv programningtasks, and the new instructions take little space. From a systems design
point-of-view, the modifications of DPS to make tlie VDB accessable are notexpensive.
The contribution of tlie VDB to tlie system seems quite clear. On two
measures of cost (computer dollar cost and total retrieval) use of the VDB
helps a searcher to use the DDB more efficiently. And, this help is mostobservable at the higher recall levels when the "going gets rough" for a
EXCP count was based solely on output printing charges, and second that last
year's data base size and vocabulary size were less than this year's. In
other words, this year's dollar cost for an average search inquiry is atleast 14% lower than last year's.

177

ESTBTATED RECALL RATIO

10%

20%

30% 40% 50% 60% 70%

800 1
700)
600 1

80%

90%

Condition,
SUPARS II>g

Simple Logic
Level,SUPARS I

5001
4000

*V Conriition,
3000' Condition,
SUPARS II

2000

g 1000

P 900
g

800

8 700
t, GOO

O

400
g

300

En

200

100
90
80
70
60

Figure No. 14. Number of Documents Retrieved (Total Retrieved) to Achieve
Nine levels of Recall: A Comparison Among the SUPARS II Conditions
and the Most Efficient Use of SUPARS I

178

searcher. Apparently, use of the VDB decreases the need to search the DDE.
This is supported by the data presented in the last columns of Tables XLVIand XLVIII.
The search experts used the VDB primarily as an adjunct to the truncation operator. With a free-text system, a truncation operator is a great
help pulling together irony vords with the same root, but with different
suffixes. Tb search a system such as SUPARS without a truncation operator
might be more efficient (See Figure 13), but it would be more arduous forthe searcher. This is suggested by the fact that approximately 90% of thesearch inquiries to the DDB included the truncation operator — even tlough
the experts were admonished to muiimize total retrieval. Use of the truncation operator occasionally decreases the precision of a search because it
may retrieve documents containing irrelevant keywords with the same rootas the desired keywDrd. By entering the root into a search inquiry of the
VDB, one can identify those irrelevant keywords and eliminate them fromthe inquiry to the DDB. This pairing of the truncation operator with the
VDB allows an expert to keep the searching easy, but maintain a high precision ratio. An added benefit to the pairing of these two searching aids
is that the cost of searching the VDB is quite low (See Table XLV); searchers
can be encouraged to use the VDB liberally with marginal increases in cost.
Finally, it should be noted that the cost-performance of SUPARS II
under the 'V1 condition is better than that of the most efficient versionof SUPARS 1.19 Thus, the relationship between tlie truncation operator andthe VDB is not siitply a trade-off — what one loses in performance, theother gains. Rather, the VDB appears to be a worthwhile investment regardless of the use of the truncation operator.
b.

The SDBThe SDB was developed as a means of synonym control working within theconstraints of a free-text system.20 The SDB is one way to develop a listof synonym-like words when such a list is not available frcm either the
publisher of the data base or professional indexers.
As shewn in Figures 10 and 12, the 'S' experimental condition proved
to be most efficient. However, search experts are unanimous in their agreement that use of the SDB was not noticeably helpful. This, nlus the fact
8

For example, see J. II. Williams, Jr. "Functions of a Man-riachine Interactive Information Retrieval System.1' Journal of the American Society for
Information Science, Volume 22. 1971. Pages 311-317.

9

Though we have not tested this statistically. As shown in Figure 14, thecurve for the 'V1 condition is lewer than the curve representing the mostefficient version of SUPARS I, and tliis is true for all nine levels ofrecall.

°Another use of the SDB is to provide a data base for those who wish to study
how searchers interact with an on-line retrieval system, but tliis is not ofinterest here.
170that the 5 IRSs under the 'S' condition were searched last by the experts,
leads to the conclusion that the cost-performance curves depict a learning
effect. While not an implausible finding, it is useful to know that experts
can improve their searching abilicy noticeably over time.
In terms of evaluating the SDB as a searching aid, the evidence suggeststhat in its present form it was not useful. The usefulness of the SDB may
increase with its size. The current data base consists of 4,235 searches,
which is not a large number considering the numerous topics covered in the
15 IRSs searched by the experts (See Table XLII). Another factor which
lessened the use of the SDB was the knowledge of the search experts. As
frequent users of SUPARS, with formal training in psychology, the 3 searchexperts many times found it easier to generate synonyms frcra memory thanfroii a small SDB. Perliaps less frequent users of the system or less knowledgeable users in the subject area would have a greater need for the SDB.
Since the SDB is a true data base, a system designer has to allocatea large amount of storage for it. And, as its size increases he would
expect the cost of a search inquiry to the SDB to rise to that of an
inquiry to the DDB. Thus, a SDB is a relatively large investment for a
retrieval system which might only pay off after several months or years when
the total number pf searches to the DDB is large enough to constitute a
viable SDB. If a SDB is too costly, a system designer should consider alternative means to make synonym-like words available to the searcher of a freetext system.21c.

Differences Among IRSs and Experts

The 15 IRSs differ according to the ease with which experts can achieve
a 90% recall level. The magnitude of the difference is staggering. Threeexperts achieved a 90% recall level for IRS 7 with an average total retrievalof 242 documents. The corresponding figure for IRS 2 .was 11,321 — 47 times
as many documents. The difference has little to do with SUPARS per se andprobably cannot be reduced (toward the lower number) by the system designer.
Tliree interrelated factors contributed to the difference: (1) the generality
and/or specificity of the IRS as written by the subject, (2) the stringency
with which the relevance criterion was applied when the subject identified
the relevant documents and (3) the breadth of topic areas published in
Psychological Abstracts — each topic differing in the specificity of its
technical language.
It is likely that the cost-performance of SUPARS/DPS would improve if
DPS has one such alternative. The system designer can load a list ofsynonyms or equivalent terms into a DPS file. The searcher can use thisfile, on option, to augment the keywords in his search inauiry. Since
SUPARS does not have such a list available, we could not make use ofthis option.

180

its DDD covered fewer topics, and each topic had fevv^cr terms referring toeach important concept.
As one would expect, the 3 experts differ in their ability to use SUPERS
efficiently. The size of this difference might be unexpected however. At
the 90% recall level the total retrieval, averaged over 15 IRSs, ranged
from 707 to 7,840.22 Search experts, equally trained and equally knowledgeable have characteristically different ways of attacking an IRS. At tliis
time we do not know what the differences are among the experts which accountfor the differences in total retrieval.
Once these differences are identified, one can choose search experts
or intermediaries more selectively. Or, if the system is available to a
general population, the system designers can use this information when they
develop the user interactive language and vrtien they prepare training materials
for the potential user.
If differences between trained users of a retrieval system are of thismagnitude generally, then intensive study into this matter is needed, because
better training of users or a more adaptive interactive language might
contribute more to improving the cost-performance characteristics of asystem than costly sof tware developments.

5.

SUMMARY OF FINDINGS

Several of the major findings of this study of the cost-performance
characteristics of SUPARS II cure listed belcw. The listing is only meant
as a partial surrmary and as such does not include explicit limits, exceptions,or explanations of the findings.
a. Search experts did not find the SDB to be a useful aid in retrieving
documents. The 'S' experimental condition did have the best cost-performance
characteristics, but this was attributed to learning effects, rather thanto any intrinsic value of the SDB.
b. The VDB is a good investment in a free-text system such as SUPARS.
It is relatively inexpensive to add to the system, and search inquiries into
the VDB are the lowest costing of all 3 data bases and should remain so as
the data bases grow. More importantly, use of the VDB decreases the need tosearch the more expensive DDB. Therefore, inquiries carried out under the•V1 condition had a better cost-performance relationship than searches
under the 'D' condition.
c. Searches carried out under the 'D' condition had the poorest
cost-performance curve.
22

Uie differences among experts sure statistically significant at all recall
levels except the lowest (10%).

181

d. The cost-performance curves of searches to the DDB in SUPARS I andSUPARS II are remarkably similar insofar as total retrieval is concerned.
DDB search inquiries in SUPARS II are about 14% less expensive in terms ofdollar cost than similar inquiries carried out in SUPARS I (using the samecharge for computer services.) The decreasing cost is attributed to theijnrprovement of hardware performance in the new IBM 370/155.
e. Search experts found the truncation operator too valuable to discard, even though the evidence fron SUPARS I shows it decreases cost-performance. The use of the VDB paired with the truncation operator seems to
be a useful match: It meets the searching style of the experts and improves
the cost-performance beyond that of the best sub-system tested in SUPARS I.
f. Some means of synonym control is needed. The SDE is one means ofachieving it when other alternatives are not available. However, the valueof the SDB may not become evident until the number of stored searches isouite large. For many systems this requires a large initial investment
with no consistent pay-off for several months.
g. Some study into the differences between searching styles may conceivably contribute greatly to the cost-performance of a system.

182

APPENDIX V I I
INSTRUCTIONS TO SUBJECTS183

APPENDIX VII
INSTRUCTIONS TO SUBJECTS
Choose a specific or general topic you need information for right new.
If you are doing a paper or planning a talk you probably have a topic inmind. If you don't have any current topic you are working on, oonsider one
you are familiar with.
In order to acquire this information for your topic we want you towrite down your information requirements as if you ware talking to a colleague who understood the field as well as you do.
(1) Start off by making a broad general statement.
your topic.

Give an overview of(2) When that is oorpleted give as much specific information on yourtopic as you would give to a colleague.
(3) After you have written the suirmary, make certain that you havegiven as much specific information as possible. Write down such things as
the major author and/or people in your topic area, recent publications,related concepts to the topic, or any other clues that relate to your topic.
In writing this sumrary give us enough of your thoughts so that we cantheoretically go out and get this information for you.

184

APPENDIX VIIIANALYSIS OP VARIANCE SUMMARY185

Source of Variation

Sum of Squares

df

Mean Square

Among
Treatarent Conditions

5041702252085Experts

5230882261544

IRSs

2136598

14152614

Residual

316221226

121623

Tbtal

6326068

44

Standard Error of the ?tean = 90.05
Standard Error of the Difference Between Means = 127.34

Figure No. 1. Differences Among Three Treatment Conditions
at the 10% Recall Level

186

Souroe of Variation

Sum of Squares

df

Mean Square

Among
Treatment Conditions

14069882703494

Experts

222325521111628

IRSs

549348314392392582649326

22409614950219

44

Residual
Totalv"Standard Error of the Mean » 122.23
Standard Error of the Difference Between Means » 172.86
Figure No. 2. Differences Among Three Treatment Conditions
at the 20% Recall Level

187

Source of Variation

Sum of Squares

df

Mean Square

Among
Treatment Conditions

11715502585775Experts

35589312

1779465

IRSs

10116553

14722611

Residual

688439426

264784

Total

21731428

44Standard Error of the Mean = 132.86
Standard Error of the Difference Between Means = 187.89
Figure No. 3. Differences Among Three Treatment Conditions
at the 30% Recall Level

Sum of Squares

df

Mean Square

6700022335001

Experts

10826397

25413199

IRSs

23116099

141651150

Residual

19091616

26

734293

Total

53704115

44

Source of Variation
Among
Treatment Conditions

Standard Error of the Mean = 221.25
Standard Error of the Difference Between Means = 312.90
Figure No. 4. Differences Among Three Treatment Conditions
at the 40% Recall level

188Source of Variation

Sum of Squares

df

Mean Square

Among
Treatment Conditions

453835422269177

Experts

23623521

211811761

IRSs

43113614

143079544

Residual

32147044

26

1236425

Total

103422533

44

Standard Error of the Mean • 287.10
Standard Error of the Difference Between Means • 406.02
Figure No. 5.

Differences Among Three Treatment Conditions
at the 50% Recall Level

Sum of Squares

df

Mean Square

Treatment Conditions

850422524252113

Experts

33061006

216530503' •".IRSST ..".

71507851

145107704

Residual

47738801

26

1836108

Total

160811883

44

Source Of Variation•AncSng

Staaidard Error of the Mean = 394.87
Standard Error of the Difference Between Means * 494.79
Figure No. 6.

Differences Among Three Treatment Conditions
at the 60% Recall Level

189

Source of Variation

Sum of Squares

df

Mean Square

Treatment Conditions

42632120221316060

Experts

77944731

238972365

IRSs

138288573

149877755

Residual

83901321

26

3226974

Total

342766745

44

Among

Standard Error of the Mean = 463.82
Standard Error of the Difference Between Means = 655.94
Figure No. 7. Differences Among Three Treatment Conditions
at the 70% Recall Level

Sum of Squares

df

Mean Square

Treatment Conditions

124757078

262378539

Experts

194712111

297356056

IRSs

327989517

14

23427823

Residual

289814769

26

11146722

Total

937273475

44

Source of Variation
AmongStandard Error of the Mean = 862.04
Standard Error of the Difference Between Means = 1219.11
Figure No. 8. Differences Among Ihree Treatment Conditions
at the 80% Recall Level

190

Source of Variation

Sum of Squares

df

Mean Square

Treatment Conditions

195418483

297709242

Experts

382735553

2

191367777

IRSs

485520111

1434680008

Residual

320914847

26

12342879

Total

1384588994

44

Among

Standard Error of the Mean = 907.12
Standard Error of the Difference Between Means = 1282.85
Figure No. 9.

Differences Among Three Treatment Conditions
at the 90% Recall Level

191