Index

Aberystwyth off-shelf test, 137, 142, 149, 150,
151
Aboutness, concept of, 47, 54, 299
AGRIS, 125
Alphabetical subject headings, 136, 147, 221,
259, 264
Analysis of results, 11, 18
failure, 16
in Medlars, 20
in Cranfield 2 test, 20
semantic, 318
stages, 18
syntactic, 318, 320
Anomalies, 42
A NOVA procedures, 96, 97
Archetypal retrieval test, 11
analysis, 18
components of, 12
costs and time, 15, 19
coverage and currency, 15
documents and requests, 13
experimental design, 13
explanatory variables, 15
failure analysis, 16
performance limits, 16
performance measurement, 14
recall and precision, 19
relevance assessments, 17
sampling, 13
system, 12
Artificial intelligence work, 172
Assumptions, 52
ASTIA Uniterm test, 261
Automatic document classification, 323
Automatic extraction indexing, 21
Automatic indexing, 218, 219, 317, 321
non-statistical, 242
not using relevance information, 239
tests, 226
using relevance information, 241
Automatic keywords, 221

Automatic term classification, 323
Automatic systems, in laboratory testing, 156178
advantages, 165
benefits of, 159
data collection, 163
fallibility of apparatus, 175
model building, 170
objectives, 166
programs, 163
program correctness, 173
realism, 167
speed of, 159
technical faults, 160
test collections, 160
Automation, 2
Averaging, 86

BACKFILES, 187
Bayes' theorem, 35
Behavioural evidence of relevance, 196
Behavioural intervention, 122
Benefits of systems, 10
measures of, 107
Bias, 26, 246
in Cranfield I, 269
BOOKS test, 247
Boolean logic, 237, 286, 289, 303, 309, 312
in INSPEC, 290, 295, 311
British Library Research and Development
Department, 285
Browsing facilities, 149, 169, 171, 191
Bureau of Ships study, 220, 222, 272

Case-Western Reserve University test, 139,
146, 219, 220, 272
Cataloguing policy, 125
Categories, 83
Changes in system, costs of, 125

343

344

Index

Characteristics of systems, measurements of,
144
Chemical abstracts, 144, 244
Clustering bases, 227
Communication,
between indexers and searchers, 123
failures in laboratory testing, 160
Computers, 1,218,324
evaluation of processing, 116interaction with user, 165, 171
output, 83
search costs, 307
statistical packages, 99
Computer searching,extraneous sources of variation, 75
Thomas, 21
Concepts, 44-57
confounded, 47, 55
definitions of, 45difficulties, 44
ineffable, 44
significance of, 53
interactions, 48
text-related, 46, 51,54
user-related, 55
Conceptual simplication, 160
Control of tests, 214, 225
Coordinate indexing, 259
Costs, 37, 69, 106, 192, 228, 248, 259
in evaluation, 110, 120
INSPEC experiment, 306
justification of, 107of changes, 125of experiments, 202
reduction of, 77
sensitivity of, 192
simulation, 192
Cost-benefit, 70, 107
Cost-effectiveness, 15, 19, 70, 107, 110, 114
INSPEC, 307, 311
Cost-performance-benefits analysis, 107
Coverage, 15, 114
as extension of recall, 114
Cranfield projects, 49, 55, 202, 219, 220, 222,
256-283, 321
co-ordination level function, 189
importance of, 256
index languages, 220, 224, 264
precision and recall, 263
specificity in, 278
test collections, 161
Cranfield 1 j see Cranfield WRU tests
Cranfield 1, 2, 68, 136, 152, 215, 225, 257-272
administrative decisions, 261
bias in, 269
criticisms of, 267
design of, 258, 268, 270
essential features, 258
failures, 264
human performance and, 151
hypotheses from, 143

Cranfield 1 (cont.)
importance of, 271
indexer/searcher memory, 269
indexing statistics, 261
indexing systems, 259
indexing time, 260, 268
index language, 138, 143, 144, 260, 269
influence of, 272
limitations, 271
parallel tests, 265
precision and recall, 137, 268, 271
failures, 152
relevance in, 271
searching, 262
search strategy problem, 146
source documents, 274
summary of, 266
uniterms, 257, 264
variables, 259
Cranfield 2, 2, 19, 21, 60, 136, 138, 143, 157,
225, 272-282
aims, 273
analysis, 20
basis of, 273, 275
conclusions, 278
co-ordination strategy, 238
criticisms, 279
defects of, 281
design, 274
failures, 280
impact of, 2indexing, 274
index language, 139,143,144,233,272,278,
282
influence of, 282
methodology, 282
recall and precision, 223, 271, 273, 276, 278,
281,282
relevance measurement, 20
results, 228, 277
searching, 236, 275
search strategies, 276
source documents, 274
source of queries, 74
specificity, 144
statistics, 279, 281
test collection, 22, 163, 243variables, 230, 275
Cranfield Western Reserve University test,
221,222,256,263,265,269
comparisons, 225
criticisms, 270
index language, 224
methodology, 229
precision in, 271
recall, 223
results, 228variables, 230
Cross-reference provision (linkage), 144
Currency, 15Current awareness, 10

IndexData,
accuracy of, 114
analysis of, 85
implementation, 99
medium, 99
collection of, 83, 163
comparison, 94
estimation, 92
exploring relationships, 97
presentation of results, 100
scale of measurement, 92
statistical inference, 91
Database, 61
building, 70
characteristics of indexing, 71
choice of, 70
collection coverage, 71
commercial, 73
constraints on testing, 130
coverage of, 114
form of surrogates, 71
INSPEC, 73
medium of, 72
search, 133
size of, 70
SMART, 72
structure of, 71
Definitions, 9Delphi technique, 85
Descriptive methods of testing, 85
Destination's relevance, 47
DIALOG, 129
Discrimination model, 324
Discrimination value, 324
DOAE test, 233, 256, 282
Documents, 13
assessments of relevance, 74
automatic classification, 323
clustering, 157, 159,239
collection of, 61
heterogeneity of, 61
definition, 9delivery test, 119delivery time, 185
description, 161
forms of, 61
for tests, 287
identification, 1
identification of, 181
indexing, 317
in SMART, 319
management, 1
ranking, 87
surrogates, 71
weighting surface, 188
Document access information, 216
Document-term incidence array, 89
Effectiveness of systems, 10, 32-43, 228
concept of, 48

345

Effectiveness of systems (cont.)
definition, 106, 168
description approach, 39
evaluation of, 133, 134, 173
limits of retrieval, 40
measurement, 33, 38
online systems, 133, 134
optimal retrieval, 35
predictive approach, 39
probability ranking principle, 35
relevance-based, 168
relevance feedback strategies, 40
SMART, 327
theoretical foundations, 33
Efficiency of systems, 10, 69
improvement of, 77
Electric Electric Library, 265
Environment of retrieval systems, 316
EPSILON, 84, 139, 144, 148, 149, 233, 234
design, 237
results, 153
users, 231
Estimates, making of, 199
Estimation, 92
standard error, 92, 94
Evaluation, 4, 66, 105-127, 222, 228, 245, 323
accuracy of data, 114
analysis and interpretation, 117
application of results, 123
behavioural intervention, 122
compromises, 119, 121
computer processing, 116
controlled experimentation, 121
control of variables, 115
cost effectiveness, 110
cost factors, 107, 110, 120
costs, 114
coverage, 114
criteria, 109
definition of scope, 115
design of, 117
effectiveness, 133, 134, 173
execution of programme, 117
input/output/outcome distinction, 109
indexing, 116
in experimental and operating environment,
121levels of, 106
macro- and micro-, 108
novelty, 114
objectives, 115
precision ratio in, 120
problems of, 115
proof and insight-oriented studies, 122recall and precision see also Recall andPrecision, 111, 113
recall ratio, 121
requirements for, 286
response time, 114, 117
searching in, 116, 236
simulation and, 119

346

Index

Evaluation (cont.)
SMART, 321
speed of response, 111
steps in programme, 115
subjective and objective, 105
technical information intervention, 121
tests, 230
validity, 137
time criteria, 110
user co-operation, 118, 119user effort, 113
value of pretest, 118
work statements, 116
Exhaustivity, 139
Expected fallout, 34
Expected recall, 34
Experimental design, 11, 13, 29
analysis of data, 85
collection of data, 83
databases, 70
document collection and database, 61
evaluation, 66
exploring relationships, 97
indexing characteristics, 71
indexing languages and procedures, 62
number of queries, 81
operationalizing variables, 61
people, 65
processing queries, 73
queries, 63
search process, 63
source of queries, 73
treatments assigned to units, 78
Experimental units, treatments assigned to, 78
Experimentation, 145
consistency, 245
controlled, 121
controlling searching, 145
definition, 4, 213
determination of, 59
efficiency of, 59
methodology, 100
pragmatics of, 59-102
presentation of results, 100
purposes of, 92
quality of, 3, 247
reliability of, 59
two-factor, 79
usefulness, 326
Explicitly probabilistic systems, 203
Explicitly utility-theoretic systems, 203
Extrapolation problems, 160

F-test, 81
Faceted classification, 136, 147, 219, 221, 259
Failure analysis, 16
in Medlars, 20
Fallout, 66, 190
expected, 34

False drops, 38
Farradane's relational indexing, 234
Fatigue factor, 81
Feedback, 9
relevance, 29Filtering capacity of system, 111

Gedanken experimentation, 199-209
definition, 199
development of, 200
indexing and,204
requester and, 205
system designers and, 206
Generality of theory, 53, 66
Godel's theorem, 41
Greco-Latin squares, 80

Harter, S.P., theory of, 21
Hawthorne effect, 148
Heisenberg principle, 148
Heuristic systems, 149
Hierarchial term arrangements, 318
Homeosemy, 48, 52
Human aspects of information services, 105
Human performance, 151
Human preferences, 151
Hypotheses, 29, 143,214
descriptive and explanatory, 215
evidence, 166
testing, 94

Input/output/outcome distinction, 109
Indexers,
as gedanken experimenter, 204
communication with searchers, 123consistency between, 63
Indexer/searcher memory, 269
Indexes,
inverted file, 72
printed, comparisons, 142
search, 133
Indexing, 9, 167,317
accuracy of, 63
as gedanken experiment, 182
automatic,
non-statistical, 242
non-using relevance information, 239
tests, 238
using relevance information, 241
automatic extraction, 21
characteristics of, 71
choice of, 144
criteria for, 12
evaluation and, 116
exhaustivity of, 62, 236, 265, 275
in Cranfield 2, 274

IndexIndexing (cont.)
manual, 2
compared with automatic text searching,249
person consistency, 151
pre-coordination of terms, 63
probabilistic, 21
procedures, 62
relational, 234
sophisticated, 243
specificity of, 62
statistics, 261
tests, 235
times, 260, 268
unweighted, 205
Indexing experiments, 139
Index languages, 62, 78, 136, 143, 219
artificial, 248
choice of, 144construction of, 19
controlled, 233, 235
Cranfield 1,260, 269
Cranfleld 2, 272, 278, 282
evaluation, 116
exhaustivity, 236
manual, tests, 230
natural, 233, 235
orderings, 80
sophistication of, 221
specificity, 231
tests, 139,220, 229, 232
overview, 224
EPSILON test, 84
Index terms, 157
Inference chains, 49
Inferential methods of testing, 85
INFOLINE, 129
Information, 45
generation of, 183
need for, 45, 46, 50
identification, 181
philosophy of, 180
relation to text, 47
Information systems,commercial, 77
control of factors, 125
core of, 216
current state of understanding, 248
definition of, 9, 180
designers as gedanken experimenters, 206
effectiveness, 134
essential function of, 129
explicitly probabilistic, 203
explicitly utility theoretic, 203
human aspects of, 105
importance of, 316
interaction with user, 193
control of, 126
libraries as, 193
measurement of characteristics, 144
outcome of, 108, 109

347

Information systems (cont.)
probability theory in design of, 202
prototype development, 159
Informativeness, 51
Insight oriented studies, 122
I M S , 125
INSPEC, 73, 139, 142, 146, 150, 282, 285-315
arrangement of groups, 296
arrangement of terms into groups, 295
boolean comparison, 303
compilation times, 293
conclusions on, 310
costs, 306
computer search, 307
intellectual effort, 306
cost-effectiveness, 307, 311
criticism, 313
database, 134
documents, 287
free and controlled-language comparison,309
list of search terms, 295
methodology, 311
number of queries, 312
profiles, 286, 297
profile compilation, 290
modification times, 294
standard tasks, 291,294
times, 293, 309
queries for, 287
ranked output comparison, 300
recall figures, 300
relevance assessments, 296
results, 299
retrieval performance, 299
retrospect, 311
search software, 299
search strategies, 288, 294, 301, 311
search time, 309
standard tasks, 291,294
term groups, 289, 313
term weighting, 288, 290, 296, 313
user participation, 312
relevance assessment by, 297
Interrelationships, 192
Investigation, definition, 4, 213
INSILT, 139, 140,143,144,151, 234, 243, 282
indexing language, 231, 233

Justification studies, 105

Knowledge, anomalous state of, 46, 50, 53
KWAC, 142, 144
KWIC index, 233

Laboratory tests, 11, 60, 156, 241
characteristics of, 156
communication failures, 160

348

Index

Laboratory tests (cont.)
conducting, 145
control of, 159
extrapolation problems, 160
manual systems, 136-155
controlling searching, 145
design of, 152
human performance and preferences, 151
indexing and searching experiments, 139
index language comparisons, 138
laws, rules or principles?, 143
measurement of characteristics, 144
performance criteria and measures, 149presentation clarity, 142
printed index comparisons, 142
recording problems, 148reliability, 153
requirements, 137
search diagnosis techniques, 152
search time, 150
types, 138
validity, 137
methodology, 157
limitations, 159
recruitment of searchers, 146
results of, 166
statistics, 28
with automatic systems, 156-178
advantage of, 165
benefits of, 159
data collection, 163
fallibility of apparatus, 175
model building, 170
objectives, 166
programs, 163
program correctness, 173
realism, 167
speed of, 159
test collections, 160
Latin square design, 79, 96, 152
Laws, 143
LEADERMART system, 47
Libraries as information systems, 193
Linkage, 144
Logic of techniques, 164, 236
boolean, 237, 286, 289, 290, 295, 303, 309,
311,312
tests, 238
Logical surface, 188
Macroevaluation, 108
Man-machine interaction, 171, 286
Manual systems,laboratory tests, 136-155
controlling searching, 145
design of, 152
human performance and preferences, 151
indexing and searching experiments, 139
index language comparisons, 138
measurement of characteristics, 144

Manual systems,laboratory tests (cont.)
performance criteria and measures, 149presentation clarity, 142
printed index comparisons, 142
recording problems, 148reliability, 153
requirements, 137
search diagnosis techniques, 152
search time, 150
types, 138
validity, 137
recording and analysing, 142
Manual tabulations, 9<pMark sense cards, 84
Mathematical theories, 170, 171, 173
Meaning, concept of, 47
Measurement, 11, 23
costs and times, 15, 19
coverage and currency, 15difficulties, 33
effectiveness, 38
explanatory variables, 15
failure analysis, 16
performance, 14
performance limits, 16
precision, 32
problems of, 33
recall, 32
Mechanized retrieval system, development of,103
Medlars, 20-21, 116, 144, 187, 215, 219, 222,
224, 230, 322
analysis of failures, 20
evaluation, 119
MEDLINE, 187
Medusa, 21, 139, 148, 151,238
MeSH terms, 241
Methodology of experiment, 9-31, 100, 128.
157, 322
achievements in, 246
Cranfield 2, 229, 282
CWRU, 229
defects, 227, 243, 247
inadequacies, 225
in simulation, 196
limitations, 159
SMART, 323
Microaveraging, 87
Microevaluation, 108, 164
Models, 42building, 170
discrimination, 324
Modularity, 77
Morse's browsing model, 185, 186
NASA study, 256, 282
Numerical algorithms Group, 186
Oddy, R.N., Thomas Program, 21

Index
Office for Scientific and Technical Information, 285
Off-shelf test, 137, 142, 149, 150, 151
Online systems, 72, 249, 250, 286
observing user behaviour, 131
record of searches, 84
software, 77
testing, 128-135
aspects that warrant testing, 132constraints on, 129
database search and print, 133environment of, 129
index search, 133
opportunities of real life systems, 131
search effectiveness, 133
system and database restraints, 130
search formulation and checking, 132
system effectiveness, 134
types of, 133
user constraints, 130
Operational definitions, 49
Operation tests, 11
Optical character recognition cards, 84
Optimization, 180
ORBIT, 129
Outcome of service, 108
Output,form of, 65
from computer, 83
indicativity of field, 65
measurement of, 108
ranking, 226, 231, 234
variety of, 78

People,collection of data, 84
training and experience of, 65
Performance,
absolute and relative, 320
criteria, 113
limits, 16
measurement of, 226
Personnel, 65
Pertinence, 47, 68
Philosophy of information, 180
Portable test collections, 22
Post-coordinate indexing, 218, 219, 226
Uniterm system, 136
Post-coordinate natural language terms, 2
Post-coordinate searching, 2
Post-coordination, 249
PRECIS, 142, 144
Precision, 32, 36,66, 111, 113, 128, 143analysis of failures, 118
averaging, 86
calculation of, 32, 66
problems, 67
Cranfield 1,268,271
Cranfield 2, 271, 273, 276, 278, 280, 282
definition, 19, 34

Precision (cont.)
differences in, 223
evaluation, 241
in experimental environment, 120
in SMART, 318, 320, 323
limits, 40
relation to recall, 67, 224, 263
results, 242
sensitivity of, 188
test ranges, 234
use of, 189
user's requirement, 112
values of, 223
Precision-boosting power, 206
Precision Engineering study, 282
Precision ratio, 111, 117
in evaluation, 120
limitations, 112
selected, 150
Prediction, 98
Presentation clarity, 142
Pretests, value of, 118
Principles, 143
fundamental, 143
Probabilistic systems, 21
explicit, 203
Probabilistic theory, 157, 169, 187, 202
advantages of, 208
Probability change factor, 206
Probability ranking principle, 35, 40
Problematic situation, 46
Profile compilation, 290
modification times, 294
standard tasks, 291
times, 293, 309
Programs,
structured, 77
testing and proving, 174
Proof-oriented studies, 122
Quantum mechanics, 33
Queries, 13,44,46,63,222
artificial, 27
formation of, 28
bogus, 225
definition, 10
formulation of, 180
for tests, 287
negotiation, 66
number of, 81
optimization, 180, 182
processing, 75
response of searcher, 64
source of, 73
statement of, 75
statistics, 25
treatment of, 243
Query-term incidence array, 90
Query terms, 231
selection, 239

349

350

Index

Questionnaires, response rate, 84
Questions,interactions, 49
relation with text, 48
Random number simulation technique, 191
Random retrieval, 41
Ranked output, 234
Ranked retrieval, 39
Rankings,
evaluation of, 38
residual, 40
Recall, 32, 36, 66, 111, 113, 128, 143, 190, 22
236
absolute, 69
analysis of failures, 118
averaging, 86
calculation of, 66, 69, 222
problems, 67
coverage and, 114
Cranfield 1,268
Cranfield 2, 273, 276, 278, 280, 282
definition, 19, 33, 226
evaluation, 241
exhaustive, 234
expected, 34
failure, 38, 152
in experimental environment, 120
in INSPEC, 300
in SMART, 318, 320, 323
limits, 40
measurement of, 32
relation to precision, 67, 224, 263
relative (proportional), 69, 112, 223, 234
results, 242
results, 242
sensitivity of, 188
use of, 189
user's requirements, 112
Recall-precision graphs, 87, 190, 240
calculating curves, 88
Recall ratio, 111, 117
in evaluation, 121
limitations, 112
Regression techniques, 98
Relationships techniques, 98
Relationships, exploring, 97
Relevance, 14, 15, 17, 36, 45, 47, 59, 231
assessments, 17, 74, 222, 287
in INSPEC, 296
in SMART, 319, 322
scales of, 67
behavioural, 197
binary, 67
concept of, 67, 68
Cranfield 1, 271
Cranfield 2, 20
definition, 47
differences in, 118
in Medlars, 20
judgement of, 167, 195

Relevance (cont.)
logical, 47
meaning aboutness, 299
meaning pertinent, 299
objective and subjective concept, 32
probability of, 169, 187
ranked, 67
recognition of, 181
relation to usefulness, 36
situational, 47
subject competence and, 75
three-value, 67
understanding of, 169
weights, 67, 243
Relevance feedback, 29, 157, 165, 170, 231,
237,241,242,321
effectiveness of, 40
Reliability of tests, 153
Requester see User
Residual ranking, 40
Response rates, 84
Response time, 111, 114
evaluation, 117
Results,presentation of, 100
Retrieval,archetypal test see Archetypal retrieval test
definition, 9fundamental issues, 44
limits to, 40
random, 41
ranked, 39
retrospective, 10set, 39
theory and experiment in, 201
Retrieval effectiveness see EffectivenessRetrospective retrieval, 10
Rules, 143
Sampling, 13
adequacy, 25
size, 25
statistics, 29
Satisfaction, 201
concept of, 48, 55
definition of, 129
level of, 106
Scope, definition of, 115
SDI, 99
SDI profiles, search strategy variations in see
also INSPEC experiment, 285-315
Searchers,
behaviour of, 237
communication with indexers, 123
individual performance, 151
interaction with user, 66
recruitment of, 146
Searches,
control of strategy, 125
diagnosis techniques, 152
logic tests, 238

Index

Searches (cont.)
performance criteria and measures, 149
recording problems, 148
response to query, 64
rules for termination, 148time, 150,309
Searching, 9, 116
controlling, 145
Cranfield, 1,262
Cranfield 2, 275, 276
evaluation, 116, 236
formulation, 9, 132
logic, 236
strategy, 236
tests, 139,232,236-238
Search process, 63
Search statements, 63, 317
specificity and exhaustively, 64
type, 64
Search strategy variations, in SDI profiles see
also INSPEC experiment, 285-315
Selective dissemination of information, 10, 15
Semantic analysis, 318
Sequence effects,
fatigue factor, 81
magnitude of, 79
Set retrieval, 39
Significance levels, 94
Significance tests, 25, 29
Sign test, 95
Simulation, 119, 179-198
advantages of, 120
definitions, 179, 182, 183, 184
examples of, 185
goals of, 195
methodology, 196previous work applied to retrieval, 192
random number, 191
reasons for, 184
virtual attribute technique, 196
SMART, 3, 72, 140, 161, 165, 175, 219, 233,
282, 316-329
assumptions, 319, 327
automatic text analysis, 227
collections, 243, 244
design of, 318
document environment, 319
early results, 319
effectiveness, 327
extended, 321
feedback, 241
form, 240
methodology, 322, 323
processing options, 119
recall and precision, 318, 320, 323
relevance, 231
assessments, 319, 322
relevance feedback, 242
results, 229, 321
theoretical insights, 323
users and, 322

351

Software, 175
development of, 77
for INSPEC, 299
online, 77Specificity, 139, 144
Cranfield, 2, 278
SPIRES/BALLOTS, 77
STAIRS, 77, 163
Statistical inference, 91
Statistical Package for the Social Sciences, 185
Statistical packages, 99
Statistics, 23-30, 36, 225
approach to tests, 225, 227
averaging, 86
comparisons, 94
effect of bias, 26estimation, 92
experimental design, 29
in Cranfield 2, 279, 281
laboratory tests, 28
limitation of, 30
parametric, 25
non-parametric, 25
reasons for, 23
samples, 25
sample size, 25, 29
significance levels, 94
techniques, 240
time and variables, 26
two populations, 24
value of, 239
variables, 27
STATUS, 163
Structured programs, 77
Subsearches, 146
Surrogates, 71
Syntactic analysis, 318, 320
Syntactic control, degree of, 63
Synthema, 48, 52
Syntol,219,221,224
System restraints in testing, 130

Taube's uniterms, 219
Technical information intervention, 121
Terms,
arrangement into groups (concepts), 295
automatic classification, 323
choice of, 148
in INSPEC, 295
Term associations, 232
Term clustering, 239
evaluation, 232
Term discrimination value, 62Term groups, in INSPEC, 289, 313
weighting, 237
Term phrases, 325
Term precision, 325
Term weighting, 157, 161, 226, 231, 237, 239,
243, 265
in INSPEC, 288, 290, 296, 313

352

Index

Tests
1956-1978, 213-255
biases, 246
characteristics, 60
choice of, 60
descriptive, 85
differences of design, 247
forms of, 216
implications, 216
inadequacy of, 245
inferential, 85
non-parametric, 95, 97
numbers of, 245
objectives, 216
parametric, 97
quality of, 247
results, 222
Test collections, 160, 195
multi-, 243, 247
numerical representation, 162
portable, 22
sizes, 244
use of, 243
Test decisions, 7Testing, 60
conclusions of, 143
fundamental principles, 143reasons for, 10
requirements of, 137
Text, 9informativeness, 51
relation to need, 47
relation to question, 48
Text-related concepts, 55
Text searching, compared with manual indexing, 249
Theory of retrieval, 201
generality of, 53
paucity of, 201
Thesauri, 219, 221,240, 318
classes, 325
EJC type, 275, 285
INSPEC, 309, 314
Thomas program, 21, 171
Time, statistics of, 26
Time criteria, 110
Top down development, 77
Treatments assigned to experimental units, 78

UKCIS investigation, 231, 233, 238, 242, 243,
244
Uniterm, 147, 148, 219, 221, 259, 270
ASTIA test, 261
compared with UDC, 223
Cranfield 1,257,264
Uniterm system, 136
Universal Decimal Classification, 136, 147,
221,259
Usefulness,concept of see also Effectiveness, 48

Usefulness (cont.)
probability of, 36
relation to relevance, 36
User(s),as gedanken experimenter, 205
behaviour of, 231
constraints in testing, 130
co-operation in evaluation, 118, 119
definition, 10
desire of, 45
satisfaction of, 55differing needs, 110
for precision and recall, 112effort of, 113
goals of, 168
interaction with computer, 165, 171
observation of behaviour, 131participation in INSPEC, 285, 287, 312
relevance assessment, 297role of, 44
SMART and, 322
studies of, 228
User-related concepts, 55
User-system interaction, 193
control of, 126
Utility, 48
theory, 202
Variables, 7, 32, 248
comparisons, 218
control of, 115,230
Cranfield 1,259
Cranfield 2, 275
detection of changes, 27
distinctions, 230
document collection and database, 61
explanatory, 15
identification of, 214
independent, 61
isolation of, 54
operationalizing, 61
relationships, 97
statistics of, 26, 27
variety of, 61
Vector product rule, 203
Vector space models, 157
Virtual attribute technique, 196
Vocabulary,
degree of control, 62
linkage in, 62
Vocabulary weighting, 240
Washington International Conference on
Scientific Information, 218
Western Reserve University Test see also
Cranfield-Western Reserve University
test, 138
index language, 220
Wilcoxon-Mann-Whitney test, 95
Wilcoxon signed ranks test, 152, 303
WUSCS, 144