i i mi.

•••,

DOCUMENT CONTROL DATA - R & D{Security elms siticmtion of title, body of abatract and indexing annotation must be entered when the overall report Is claeeltiad)». O R I G I N A T I N G ACTIVITY (Corporate author)
2a. REPORT SECURITY CLASSIFICATION

Syracuse University
School of Library Science
Syracuse, New York 13210

UNCLASSIFIED
26. CROUP

N/A

S. REPORT T I T L E

FREE 'TEXT RETRIEVAL EVALUATION
4. DESCRIPTIVE NOTES (Type of report and inclusive dates)

Final Report 1 February 1971 - 31 January 1972
ft. A O T H O R ( S ) (Firmt name, middle initial,

taal name)Pauline Atherton
Kenneth H. Cook
Jeffrey Katzer
ft. REPORT DATE

7a. TOTAL NO. OF PACES

da. CONTRACT OR GRANT NO.

7b. NO. OF REFS

196

July 1972

6

9a. ORIGINATOR'S REPORT NUMBER(S)F30602-71-C-0185

None

Job Order No. U59UOOOO&e. OTHER REPORT NO(S) (Any other number* that may be memiffd
thle report)

RADC^CR-72-159
10. DISTRIBUTION STATEMENT

Approved for public release; distribution unlimited.
II'. SUPPLEMENTARY NOTES

12. SPONSORING MILITARY ACTIVITY

Rome Air Development Center (IRDT)
Griffiss Air Force Base, New York 1 3 ^ 0

S NoneII. ABSTRACT

The basic problems this research effort investigated were (l) the development,
implementation, and evaluation of algorithms to improve recall levels in interactive,
free-text retrieval using a modified version of IBMfs Document Processing System(DPS), (2) the development of techniques for increasing the vocabulary capacity of
DPS, and (3) unobtrusive, statistical data gathering of system use, growth, and cost.
through a previously developed computer program. A free-text document data base
(DD?) of U6,828 bibliographic citations and abstracts from Psychological Abstracts'
was developed. Also, two interactively accessible data bases were developed andimplemented to provide free-text vocabulary control and recall improvement directly
to the user. No intermediaries were used in the retrieval process. These two algorithms were (l) a Vocabulary DataBase (VDB) containing the 106,7027unique freetext terms from the inverted file of processed documents in the DDB, and (2) a Search
Data Base (SDB) containing previously submitted user search inquiries to the DDB. A
two-month period of experimental use of the entire system with all three data bases
by students, staff, and faculty of Syracuse University in fall 1971 provided the required "real-life" field environment. A total of 2399 search inquiries were submitted Via the 27^1 terminals. The system operated under both the 360/50 and the370/1155 operating systems. The capacity of the DPP vocabulary was increased by thedevelopment and successful implementation of computer programs that revised the DPS
coding structure of the vocabulary file. The newly developed structure changing
the 16-bit coding to 32-bit coding resulted in increasing the vocabulary capacity

DD w«1473\0~

UNCLASSIFIED
Security Classification

u n v s j j j t o o x r ±r*ij
Security Classification
K EY

WORDS

Abstracts
Information Retrieval

13.

Abstract (continued)

from the former single-file limit of 65,53^ to over k biklion terms
An qxtensa ve
user-oriented public relations/publicity, i n s t r u c t i o n , and edijjcatiorj package wasdeveloped and implemented. This user emphasis r e s p i t e d i n a si ignifijcantljf great er
number of registrants and actual users in the systJem, which di]d not use searchintermediaries, than during a similar fall 1970 pefriod, Results o f |a conH r o l l e dcost-performance (recall) study indicated that use) of t h e VDB [or tftd SDB jfieldeq
better cost-performance levels, especially at highjer
HP rec(all pejrcentelges, t]|han bjj
use of the DDB alone. These results are the i n i t i k l f i n d i n g s and would Require
additional testing to substantiate their validity, Othepr evaJjuativd techniques
included a Semantic Differential attitude scale fob interactive retrieval systems,
a structured telephone interview of users, and a s p e c i a l number useils couJjd c a l ifor help in developing search inquiries. The STATIPAC program for unobtrusively
monitoring and retrieving data on system use, grov(th, arid cost! wasandsuccessfully used to provide evaluative data, General conclusions aire thdt i n t e r active, free-text cost-performance (recall) levels) sin b|e improved tjy direjct us^r
control of algorithms providing vocabulary control] ReaR - l i f e ! a p p l i c a t i o n ofthese algorithms suggests that the recall improvemjent avkilabU e thrqugh cq ntrol3|ed
vocabulary or indexing systems might be obtained rieadiljrt
the free-- t e x t searcher
who is provided with techniques such as those imp]Jemente(d i n tjhis MsearcH e f f o r t .

SACr-Griffiss AFB NY

ft

UNCLASSIFIED
Security Classification

N O T I C E
THIS D O C U M E N T HAS S E E N R E P R O D U C E D
B E S T COPY F U R N I S H E D US BY THE
AGENCY.

FROM THESPONSORING

ALTHOUGH IT IS R E C O G N I Z E D THATTAIN P O R T I O N S ARE I L L E G I B L E .

CER-IT IS BEING R E -L E A S E D IN THE I N T E R E S T OF MAKINGAS MUCH INFORMATION AS P O S S I B L E .

AVAILABLE

FREE TEXT RETRIEVAL EVALUATION
Pauline Atherton
Kenneth H. Cook
Jeffrey Katzer
Syracuse University

Approved for public release;
distribution unlimited.

I Cy

FOREWORD

This report was done by the Syracuse University Psychological Abstracts
Retrieval Service (SUPARS) Research Group at the Syracuse University Schoolof Library Science, under contract F30602-71-C-0185, Job Order Number
45940000, for Rome Air Development Centerf Griffiss Air Force Base, New York.
Mr. Nicholas M. DiFondi (IRDT) was the RADC Project Engineer.
This report represents a continuation of work conducted under contracts F30602-69-C-0013 and F30602-70-C-0190 during the period 1 July 1969
and 31 January 1971.
The current reseach contract covers work accomplished during 1 February1971 to 31 January 1972, and deals with the development, iitplementation, andevaluation of new algorithms to iinprove recall in an interactive, on-line,
free-text retrieval system.
•Individual authors of different sections include: Section I, Kenneth
H. Cook; Section II, Lynn Trurtp and Mr. Cock; Section III, Sandra Browning,
June Brewer, Jeffrey Katzer, Patricia Moell, and Peggy Mucia; and Section
IV, Mr. Katzer.
Adoidwledgernent is given to the following individuals: Dean itoger C.
Greer and Dr. Allan F. Hershfield of the School of Library Science who provided continued support for the SUPARS project; Mr. William J. Jones, Director, Syracuse University Carputing Center, for his cooperation and assistancein the inplementation of the on-line system; and to Mrs. Brenda Lefebvre,
tthose overall editorial and typing made this report possible.
This report has been reviewed by the Information Office (01) and isreleasable to the National Technical Information Service (MTIS).

Approved:

NICHOLAS M. DIFONDI
Technical Evaluator,?£)<

Approved^XpfcANZ H. DETTMER
C o l o n e l , USAF
C h i e f , I n t e l & Recon

Division

FOR THE COMMANDER
FRED I . DIAMOND

Acting Chief, Plans Officeii

ABSTRACTThe basic problems the current research effort (February 1, 1971-January 31, 1972) investigated were (1) the development, implementation, andevaluation of algorithms to inprove recall levels in interactive, free-text
retrieval using a modified version of IBM's Document Processing System (DPS),(2) the development of techniques for increasing the vocabulary capacity of
DPS, and (3) unobtrusive statistical data gathering of system use, growth
and aost through a previously developed computer program. A free-text document data base (DDB) of 46,828 bibliographic citations and abstracts from
Psychological Abstracts was developed. Also, two interactively accessible'lata bases were developed and implemented to provide free-text vocabularycontrol and recall improvement directly to the user. No intermediaries wereused in the retrieval process. These two algorithms were (1) a Vocabulary
Data Base (VDB) containing the 106,702 unique free-text terms from the inverted file of processed documents in the DDB, and (2) a Search Data Basei(SDB) containing previously submitted user search inquiries to the DDB. A
tfco-nonth period of experimental use of the entire system with all three
data bases by students, staff, and faculty of Syracuse University in fall1971 provided the required "real-life" field environment. A total of 2399
3earoh inquiries were submitted via the 2741 terminals. The system operated
underfeoththe 360^50 and the 37§/l55 operati^ ^The capacity of
the DPS vcfcabulary was increased by the development and successful implementation of aomputer programs that revised the DPS coding structure of the
vocabulary file. The newly developed structure changing the 16-bit coding
to 32-bit coding resulted in increasing the vocabulary capacity from thefarmer single-file limit of 65,534 to over 4 billion terms. An extensive
user-oriented public relations/publicity, instruction and education package
was developed and implemented. This user enphasis resulted in a significantly greater number of registrants and actual users in the system, which did
not use search intermediaries, than during a similar fall 1970 period. Results of a controlled cx>st-performance (recall) study indicated that use of
the VDB or the SDB yielded better cost-performance levels, especially athigher recall percentages, than by use of the DDB alone. These results are
the initial findings, and would require additional testing to substantiate
their validity. Other evaluative techniques included a Semantic Differential
attitude scale for interactive retrieval systems, a structured telephoneinterview of users, and a special number users could call for help in developing search inquiries. The STKTPAC program for unobtrusively monitoring£nd retrieving data on system use, growth, and aost was modified and successfully used to provide evaluative data. General conclusions are that interactive, free-textrosfc-performanae(recall) levels can be inproved by directuser control of algorithms providing vocabulary control. Real-life application of these algorithms suggests that the recall iinprovement available
throu^i controlled vocabulary or indexing systems might be obtained as readily by the free-text searcher who is provided with techniques such as those
iKplonented in this research effort.

••*
inEVALUATIONThe o b j e c t i v e of t h i s study was to develop, implement
and evaluate methods for increasing vocabulary f i l e space andimproving the r e t r i e v a l effectiveness of a f r e e - t e x t indexedo n - l i n e document r e t r i e v a l system at Syracuse U n i v e r s i t y .
Thesystem f i r s t operated on an IBM 360/50 computer and recently
on a 370/155 computer.
The document data base consisted of
46,828 b i b l i o g r a p h i c c i t a t i o n s and/or abstracts from
Physchological A b s t r a c t s .
Vocabulary storage space wasincreased by developing computer programs to convert thehalf-word ( 1 6 - b i t ) coding scheme as defined by the IBM/DPS
program to f u l l word (32 b i t ) coding*Methods of improvingr e t r i e v a l e f f e c t i v e n e s s include a vocabulary data base anda search data base as o n - l i n e searching a i d s .
Results arereported in terms of nine l e v e l s of Recall (the portion ofr e l e v a n t documents r e t r i e v e d ) * t o t a l r e t r i e v a l (the number of
documents r e t r i e v e d to achieve a s p e c i f i c Recall l e v e l ) , andCost-performance (the cost incurred to achieve a s p e c i f i c
A v
Recall l e v e l ) .
*There are several s i g n i f i c a n t conclusions derived fromthe r e s u l t s of t h i s e f f o r t :1.
The conversion from 16 b i t to 32 b i t coding has
increased the l i m i t on a single vocabulary f i l e size fromapproximately 65,000 words to over 4 b i l l i o n words.
Withoutt h i s c a p a b i l i t y , upon reaching the 65,000 word l i m i t a new
vocabulary would have to be defined and then created by thef r e e - t e x t indexing program r e s u l t i n g in i n e f f i c i e n t use ofcore due to redundant information between f i l e s and longer search
cycles.
2.
The use of the vocabulary data base as a searching
aid r e s u l t s in b e t t e r cost performance than using the other databases,* Since the vocabulary data base is a portion of thei n v e r t e d f i l e developed from the o r i g i n a l processing of documents,i t is r e l a t i v e l y inexpensive to add to the system and reduces
cost performance by decreasing the need to search the more expensive
document data base.
3. S t a t i s t i c a l l y s i g n i f i c a n t differences in t o t a lr e t r i e v a l at a l l levels except the 10% level of Recall r e f l e c t e dv a r i a t i o n s in e f f i c i e n t use of the system by search e x p e r t s .
Eachexpert was equally knowledgable in the subject a r e a , equallyt r a i n e d in the use of f r e e - t e x t r e t r i e v a l , and used the same
Information requirement statements to formulate his searches.

v

Preceding page blankHowever, each chose a different search strategy in the hopesof minimizing total retrieval. This finding indicates that
it is difficult for system experts to find or establish efficient
search methods. Since most users may be knowledgeable in theirfields but not necessarily experts in the use of on-line
retrieval, attempts by novice users to formulate efficient searchstrategies may be difficult to achieve.
4. Total document retrieval is very high at alllevels of Recall. Although use of the vocabulary data baseor the search data base does reduce total retrieval considerablyfrom that achieved by using the document data base* too muchtime and effort would be required of the user to scan for relevant
documents.
As a result of this study future research can be directed
toward establishing better search methods to reduce total
retrieval, continue work on the Recall improving algorithms
to insure total retrieval improvement does not negate their effecton system effectiveness, and identify differences in theexperts methods of searching to determine the impact on thegeneral population of users and make adjustments accordingly.

NICHOLAS M. DIFONDI

vi

TABLE OP CONTENTSSECTION
I.

TITLE

PAGE

INTRODUCTION
1.

Previous Research Efforts

1

2.

Objectives of Current Wbrk (SUPARS II)

2

3.

Philosophy and Approach

3

4.

CXrarview of Wbrk Acoarplished:
February 1971 - January 1972

4a*Development of New Searching Algorithms(1)

The Search Data Base

5(2) The Vocabulary Data Base

9(3)fBbegratian o£ Ml
in Searching ;-'•

Three Data Bases
10(4) Alternatives Considered but not
implemented

11

5.

Modification of the Free-text Vocabulary File

12

6.

Documenting.Cost, Use, and Grcwrth of SUPARS/bPS II
au

Documenting Cost of SUPARS /DPS II

13b.

Documenting Use of SUPARS/DPS II

14

(1)

14

c*

7.

Structured Phone Interview(2) Attitude Measures

14

(3)

15

User Initiated Calls

Documenting Growth of SUPARS/DPSn(1) Growth of the Three Major Data Bases

15

(2)

Profiles bf User Registrants

15

(3)

STATPAC16

Summary

16

vii

TITLE

PAGE

1.

Docunent Data Base

17

2.

Vocabulary Data BaseDATA BASE

a.

b.

3.

Special Programs Developed for a Vocabulary
Data Base

22

Expanding the Capacity of the Vocabulary
Data Base24

The Search Data Basea.

Definition of Data Base Description27b.

Modifying Search Module and Reformat
Programs

27

Specification of Oiitput Forms

29

c*
4.

SuKirary

29

5.

Implications and Projections

29

APPENDICES
Appendix I

System Overview

32

Appendix IITranslate Psychological Abstracts

34

Appendix III Search Reformat

49

THE USER
1.

Profile of Registrants
a.
b.

57

Cumulative Growth of the Registrant
Population

58

Demographic Data60

• •. (1) University Status(2) Departmental Status(3)

Time S£ent on Teaching, Research andOther Activities
viii

60
60

60

SECTION

TITLE

c.

(4) Previous SUPARS Registrationand Use64

Use of Psychological Abstracts byRegistrants •64(1) Reaent Use of Psychological Abstracts

64

(2)

68

Future Use of Psychological Abstracts(3) Reaent and Future Use of Psychological
Abstracts for Preparation of a Term
Paper, Thesis, or Speech(4)d.

Specific Need far Psychological
Abstracts

71
71

Ccnputer Experience of Registrants

76(1) Previous Scperience with Computer,TB3tdnals
-

76

(2)

e.

PAGE

Previous Experience with Oonputer-Based
Retrieval Systems

77
77

Summary

2. Publicity

78

3.

4220 LOGS

78a.

Description of Log Summary B

79b.

Summary80

4. STATPACa.

Summary 1

85b.

Summary 2

88c.

Summary90

5. Telephone Interviews of SUPARS Registrants

91a.

The Sample

93b.

Completed Versus Non-completed Interviews

93

ix

SECTION

TITLE
c.

Users: Description

PAGE
94

d. Users: General Usage Patterns

95e.

Users: Reactions

99

f. Nonusers: Description

101

g.

Sunmary

10 4

6. Semantic Differential

106

a.

Description of Procedure

106

b.

Data Organization

112

c. Results
d*

7.

113

CarpariLson with the Results of Last Year's
Study

113

e„ * Conclusions .-118

f. Discussion

118

Inplicattons and Projections

119

a.

User Orientation

119

b.

Publicity and Instruction Fwictions

119

c.

User Control of Interaction

120d. Obtaining User Response

121

e.

121

Conclusion

APPENDICES
Appendix IV
Appendix V

Program Description of SUPARSSIATPAC122

SUPARS telephone Survey Question
and Registration Form

134•

Appendix VI

Introduction to Semantic Differential
Questionnaire147

x

SECTION

IV

TITLE

PAGE

CO!3T-PI3RFORMRNCE ANALYSIS

150

1. Definitions

150

2. Method

152

•j.f

a. Experimental Design

153

b. IRSs

154

c. Identification of Relevant Documents

156

d. Search Experts

156

e. Procedure

157

f. Analysis

157

Results

158

4

a. Use of Optional Daba Bases

158

b. Computer Costs

161

c. Cost-Performance

163

d. Comparing the Cost-Performance of Two
Years of SUPARS Operation

172

4. Discussion

177

a. The VDB

177

b. The SDB

179

c. Differences Among IRSs and Experts

180

5. Summary of Findings

181

APPENDICES
AppenSlx VII Instructions to Subjects

183

Appendix VIII Analysis of Variance Suntnaxy

185

XI

SECTION
V.

TITLE

PAGE

CONCLUSIONS AND RECOMMENDATIONS

192

1. Major Conclusions

192

2.

a.

Rocall-Iraproving Algorithms

192

b.

Improvir^ SUPARS/DPS Vocabulary Capacity

193

c.

Computer Programs Recording System Use,Ccst, and Growth

193

d.

Publicity, Instruction aixl Education

193

e.

Systematic Evaluation of User Attitudes
and Reactions

193

f.

Growth of the Document Data Base

193

g.

Cost Comparisons Between SUPARS I andSUPAR&ai

194

Reccniriendaticns Madea*

194

Continue Development of User Control
of Vocabulary and Synonyms

194

b.

Maintain a Non-Reentrant Search Module

194

c.

Improve User Access to Hardware

194

d.

Investigate Searching Styles and Techniquesof Free-Text Users

195

Develop Interactive Algorithms Based onUser Styles and Techniques

195

e.

REFERENCES

196

xii

LIST OF FIGURES AND TABLES
PAGE
Search Inquiry Using Search Data Base

6

Output Example of "LIST SEARCHES" in Search
Data Base

7

Output Exairple of "LIST WORDS" in Search
Data Base8

Vocabulary Data Base Input: Determining Statusof Single Wbrd

8

Vocabulary Data Base Output: Determining Statusof Single Word10

Grcwth Rate of SUPARS/lDPS n Document Data Base

20

Curtulative Gropfch of the SUK®S/t3PS Vocabulary
Data Baste--.
.
v

23

Modification of Characters in Search Data Base

28

Cumalative Grcwth of SUPARS II Registrant
Population

59

Carrputer Cost per Search to Achieve Nine Levels of
Recall: A Conparison Among Three Treatment
Conditions
168
TVo Methods of Conputing Dollar Cost of Searchingon DDB Using LIST BRIEF Output Format

170

Number of Documents Retrieved (Total Retrieval)to Achieve Nine levels of Recall: A Comparison
Among Three Treatment Conditions

169

Nunber of Documents Retrieved (Total Retrieval)to Achieve Nine Levels of Recall: A Comparison
Between SUPARS I and SUPARS II Searches to the DDB 175
Nimber of Documents Retrieved (Total Retrieved)
to Achieve Nine Levels of Recall: A Conparison
Among the SUPARS II Ctonditions and the Most Efficient Use of SUPARS I

xiii

178

PAGE

TABLE

I

loading History of SUPARS/DPS II Document
Data Base

19

II

Track Allocation and Usage of Document Data Base

21

III

Track Allocation and Usage of Search Data Base

26

IV

Growth of Search Data Base

25

V

Number of SUPARS Registrants by university
Status61

Number of SUPARS Registrants by Major Departmental
Categories

62

Number of SUPARS Registrants by Departmental
Categories

63

Percentage of Tine .Qigaged in Teaching and/or
Learning

65

IX

Percentage of Time Engaged in Research

66

X

Percentage of Time Engaged in Other Activities

67

XI

Registration for SUPARS Last Year

68

XII

Usage of SUPARS by Last Year's SUPARS Registrants

69

XIII

Average Usage of Psychological Abstracts in the
2-3 Month Period Preceding Registration

70

Anticipated Usage of Psychological Abstracts inthe 2-3 Month Period Following Registration

72

Past Use of Psychological Abstracts for Paper,
Thesis, and Speech Preparation

73

Anticipated Future Use of Psychological Abstracts
for Paper, Thesis, and Speech Preparation

74

VI
VII

VIII

XTV

XV
XVI
XVII

Anticipated Types of Use of Psychological Abstracts 75

XVIII

Previous Experience with Computer Terminals

76

XIX

Previous Experience with Cotputer-Based Retrieval
Systems77

xiv

TABLEPAGE

XX

Log Summary A

81

XXI

Log Summary B

84

XXII

STATPAC Summary I

87

xxfii

Mean CPU lime and Cost Over the Three Data Bases

89

XXIV

Frequency of Completed Interviews

92

XXV

Academic Status of Users

94

XXVI

Frequency of Attempted Use of SUPARS

96

XXVII

Average Tine Spent Using SUPARS in one Session
at the Terminal

96

Frequency and Percentage of Estimated Number ofSearches Mad&"~^97

Problems Experienced' by Users Which H&npered
Attenpts to Sign On

98

Problems experienced by SUPARS Users After
Signing On

99

Frequency and Percentage of Responses:"Have you located more relevatn information
with SUPARS?"

100

XXXII

Academic Status of Nonusers

101

XXXIII

Attitudes Itoward and Cbntact with SUPARS

1Q3

xxxiv

Canaepts Used in the SUPARS Semantic Differential
Packet107

Exanple of a Semantic Differential Used in the
Present Study

108

Classification of Respondents by Ctxrpletianof Serantic Differential

109

Classification of Respondents by Number ofSearches Made

110

Number of Days Students Used SUPARS112

XXVIII

XXIX

XXX

XXXI

XXXV

XXXVI

XXXVIIXMCVIII

xv

PAGE

TABLE
Concept Means

114

Standard Deviations for Concept Means

116

Experimental Design

154

Characteristics of Subjects and IRSs

155

Sources of Variation and Degrees of Freedom in
Cross-over Design

158

Use of Three Data Bases to Achieve the 90%
Recall Level

159

Estimates of the Ccnputer Cost of Searching theThree Data Bases

162

Estimates of the Mean Cost of Searching tinder
the 'D' Experimental Condition at Nine Levelsof Performance (Recall)

164

Estimates of the Mean Cost of Searching Under
the 'S1 Experimental Condition at Nine Levelsof Performance (Recall)

165

Estimates of the Mean Cost of Searching Under
the 'V Experimental Condition at Nine Levelsof Performance (Recall)

166

XLIX

Differences Between SUPARS I and SUPARS II

173

L

Estimates of the Computer Cost of Searching
to DDB Counting All Entered Searches (N=246)

174

xmx
XL
XLI
XLII
XLIII
XLIV

XLV

XLVI

XLVII

XLVIII

xvi

LIST OF FIGURES FOR APPENDICESFIGUREPAGE

APPENDIX I
System Overview

33

1

Input Record Description

35

2

Translate Psychological Abstracts Logic Diagram

36

3

Translate Psychological Abstracts Flow Chart

37

4

Reformatted Data

41

5

Reformat Psychological Abstracts Logic Diagram

42

6

Refarwat Psychological Abstracts Flow Chart

44

1

Data Base Description

51

2

Input Record Description

523

Output Record Description

53

4

Refonrat Searches Module 1 - KEBF1DS

54

5

Reformat Searches Module 2 - POEMSRCH

56

1

Data Flow of SUPARS log

124

2

Program Flow for Producing STAT Output

125

3

Program for Conversion of Log to STAT Usable Form

126

4

MACRO - General Outline of STAT Programs

129

5

PROCESS - Main Wbrking Section of STAT Programs

130

1

APPENDIX IIAPPENDIX III

APPENDIX IV

xvii

APPENDIX V
1SUPARS Registration Form

146

Differences Among Three Treatment Conditions
at the 10% Recall Level

186

Differences Among Three Treatment Conditions
at the 20% Recall Level

187

Differences Among Three Treatment Conditions
at the 30% Recall Level

188

Differences Among Three Treatment Conditions
at the 40% Recall Level

188

Differences Among Three Treatment Conditions
at the 50% Recall Level

189

Differences Among Three Treatment Conditions
at the 60% Recall Level

189

Differences Among Three Treatment Conditions
at the 70% Recall Level

190

Differences Among Three Treatment Conditions
at the 80% Recall Level

190

Differences Among Three Treatment Conditions
at the 90% Recall Level

191

APPENDIX VIII
1

2
3
4

5

6

7

8

9

xviii

USAGE OF TERMS

Because the definitions of terms used in the information technology
field are not completely standardized nor consistent the terms in this report are e^lained below. An effort has been made, where possible, tofollow the most consistently used and reasonable meaning to convey a concept. Where specialized or nore specific usages of these terms are enployedin specific sections of this report, an explanation will be given by thesection author.
(a) Delta: The character(upshift "H") on the 2741 keyboard that isused in SUPARS/DPS user interaction to access the document database and initiate a search inquiry.
(b) Delta S: The character "delta" and the letter "S" ( S) which areused in SUPARS/bPS user interaction to access the search database and initiate a search inquiry.
(c) Delta V; ihe character "delta" and the letter 'V" ( V) which areused in SUPARS/DPS user interaction to acaess the vocabulary database and initiate a search inquiry.
(d) Dictionary; The internally stored list of unique free-text terms
processed by DPS and the document frequency count for each work.
The DPS dictionary forms one part of the inverted file.
(e) pocunent: In this study, the term "document" stands for the bibliographic citation and abstract that are used as a surrogate of
the original journal, article, proceeding, book, etc.
(f) Document Data Base (DDB): Consists of SUPARS/DPS processed docur
irents of Psychological Abstracts which are interactively accessible
on-line by users- The DDB is one of three data bases available
to the SUPARS/t>PS user (others are vocabulary data base and searchdata base).
(g) Document Processing System (DPS): The IEM free-text, batch mode
programs that convert machine readable textual data into searchable and retrievable data sets organized in inverted file structure.
(h) Free-Text; Specifically refers in this study to the Document
Processing System. The general reference is to an organized system alleging the indexing and retrieval of documents or theirsurrogates by any of the terms used in a defined text, rather
than terms detived by a controlled set of terms*(i)Information Requirement Statement (IRS); The verbal or written
statement of an individual's interest area as generally related
to documents or their surrogates. The IRS is the publically verifyable indication of the internally held construct, "informationxix

(j) Label (labelled line): Ttje portion of a search inquiry, such as
IA, 12, etc. that identifies and stands for the search words andoperators used to act on those wards; a label can be used itself in
an irquiry as a search ward.
(k) Operator; The user language accepts as input one or more keywords
which represent the IRS of the searcher. Keywords nay be combined with Boolean operators (AND, OR, NOT) or grammatical operators (those which specify the desired proximity of keywords within a sentence or those which specify the root of a word).
(1) Search: The search inquiry, the user/computer interaction, andthe printed output, if any. The beginning of a new search inquiry marks the end of a search.
(m) Search Data Base (SDB): Consists of the previously stored andprooessecFsearch inquiries made to SUPARS/DPS. The SDB is one ofthree data bases available to the user and was newly developed
during the current research. (Others are the document data baseand the vocabulary data base.)(n) Search Inquiry: The user arrangement of words, word combinations
and logical operators in A form acceptable as input for machine.processing. A SUPARS/DPS search inquiry would consist of thefree-text terms combined with Boolean and other logical operators,
the request for output, and an "end" statement. Examples of searchinquiries are given in Section II.
(o) Search Word: Free-text word(s) or term(s) used as part of a searchinquiry. A KEYWORD is a synonym for a search word.
(p) STATPAC: The Statistical Package used in con junction with SUPARS/DPS to unobtrusively collect, store and retrieve the elements ofuser interaction and other system parameters such as time, terminal number, cost, etc. STATPAC includes a highly flexible retrieval system in itself which allcws the operator to specify andretrieve various ccnbinations of data reflecting user interaction
or system performance. In addition to standard summaries printed
periodically, the operator oould request, for exanple, a listingof the oonputer time used for all searches of the document database by one-time users after a oertain calendar date.
(q) SUPARS/bPS: Syracuse University Psychological Abstracts Retrieval
Serviae/Document Processing System. The modified DPS program developed at Syracuse University which allows on-line, interactive
searching of free-text data. SUPARS/DPS I refers to the research
work aoixlucted from July 1969 to January 1971. SUPARS/bPS II
refers to the work oonducted from February 1971-January 1972.
(r) Voccibulary: The on-line, interactively accessible dictionary thatis stored by DPS. The term "vocabulary" rather than "dictionary"is used to connote the words and terms accessible to the userxxthat can be used as free-text index tents.
(s) Vocabulary Data Base (VDB): Consists of the on-line, interactively
accessible DPS dictionary of free text terms. The VDB is oneof three data bases available to the user and was newly developed
during the current research. (Others are document data base andsearch data base) •xxi