'•Indexing of the remaining cases in the experiment will be performed by machinefrom full text, using the Type I list of discard words and the Type II list to prepare an analysis of the frequencies related to index-word space. Instead of
selecting specific words as indexing terms, concepts will be selected (statistically)
as volumes in index-word space* A rough physical analogy to this process would
be to toss pennies at the previously mentioned grid so that, for every Type II
word in the source document, a penny lands at its proper slot on the grid. Where
the pennies heap up in a pile, you have a concept. If"Searching will be carried out essentially by indexing a question presented
narratively, determining the concept volumes that represent the question, andsearching those volumes in document space for the relevant document numbers.
Since the 'edges' of the concept volumes are determined statistically, output canbe listed in order of probable relevance; as an option the question could beaccompanied by a request that 'at least 100 references be supplied', in which casethe concept boundaries would be adjusted to provide that number. " }J
It will thus be noted that the proposed indexing and search program begins on aderivative basis to establish for one-half the experimental material the significant words,
next combines word frequency with significant word distance data to derive probabilistic
association factors between words, then develops clusters, and finally indexes the items
in terms of the clusters rather than words so as to provide assignment rather thanextraction of index terms.
7.

PROBLEMS OF EVALUATION

We have noted, in the introduction to this report, that several fundamental andhighly controversial questions can be raised with respect to the feasibility and evaluationof any automatic indexing scheme and with respect to the evaluation of any indexing
systems whatsoever. Yet if automatic indexing procedures are to be based upon previous
human indexing or if their results are to be compared with human results, then thequestions of the quality, the reliability and the consistency of human indexing are crucial
ones indeed. Thus, Solomonoff warns:"The finding of exact languages for retrieval is also made l e s s likely, in viewof the fact that the categorizations of documents that are presented to the machineas a training sequence will not be performed altogether consistently by the humancataloger. " Zj
Montgomery and Swanson ask whether human indexers are in fact self-consistent andconsistent with each other, and they suggest:yEldridge and Dennis, 1963 [ 182], pp. 97-99.
Solomonoff,

1959 [ 5 6 2 ] , pp. 9-10.

143

"If the answer turns out to be 'no 1 , we might reasonably conclude that the only
reliable and effective kind of human indexing is that which is already machinelike in nature. f l ! /With a few noteworthy exceptions, there has been very little serious investigation of these
problems and there is very little comparative data,
O'Connor has been making a series of studies, with considerable emphasis upon how
one might measure the products of machine indexing and how one might derive machine
rules for automatic index ing from systematic review of documents indexed by people.
Cleverdon and his associates at the ASLIB Cranfield project have extensively tested
several different indexing procedures. Painter, MacMillan and Welt, Slamecka andZunde, and others report findings on intra-indexer, and inter-indexer consistency - unfortunately, on the basis of quite small samples. Various alternate approaches to theevaluation of automatic indexing results have been considered by Borko, Doyle, Swan son,
Savage, Giuliano, and others. In addition, some data bearing on these questions havebeen reported in connection with analyses of selective dissemination (SDI) s y s t e m s .
Some data from other sources, such as studies of user preferences with respect tovarious reference and search tools, is also pertinent.
The most generally accepted criterion for appraising the effectiveness o f indexingis that of retrieval effectiveness. But, in general, this is merely the substitution of
one intangible for another, entailing a string of as yet unanswerable or at least unresolved questions.^' Retrieval of what, for whom, and when? How can effectiveness bemeasured except by the elusive question of relevance judgments? How can human judgements of relevance and value be measured and quantified?
We shall try to distinguish here, insofar as possible, between the core problems
that make the evaluation of indexing as such an extremely difficult task, the available
data on human indexer reliability, and the possible advantages and disadvantages ofautomatic indexing techniques.

yMontgomery and Swanson, 1962 [ 4 2 l ] , p. 366.

If
Compare Swanson, I960 [ 5 8 2 ] , pp. 2-3: "The performance of retrieval experiments when relevance judgments per se cannot be consistently a s s e s s e d by humanjudgment would seem to represent overly vigorous pursuit of a solution before
identifying the problem." Similarly, see Black, 1963 [ 6 4 ] , p. 14: "Finally,
when one is faced with an existing collection of indexed materials, how does one
a s s e s s the effectiveness of any retrieval system? Suppose that one receives 20
documents as a result of a query to the system. Suppose further that all 20 documents are quite pertinent to the topic of interest. Is there any way to a s s e s s theamount of pertinent information still unretrieved from the file? Or is there any
way of learning whether the retrieved information is more pertinent than the unretrieved information ? The answer is 'No! f - - the use of any retrieval systemi s , then, an act of faith in the quality of indexing. "

144

7, 1

Core Problems

F i r s t and foremost of the core problems implicit in the question of evaluation of anyindexing scheme, whether applied by man, machine, or man-machine combinations, arethose of i n t e r p e r s o n a l communication itself, which in turn r e l a t e to fundamental problems
of epistemology. These a r e , first, the problems of language as a means of com municating perceptions, apperceptions of relationships between present observations andp r i o r experience, and value judgments based thereon, and, secondly, even m o r e fundamentally, the question and the veridicality of language representations of r e a l transactions
and events. Serious investigators in the field, including many who have themselves contributed to automatic indexing techniques, have made such typical acknowledgments of the
difficulties as the following:"The imprecision connected with discussion of r e t r i e v a l effectiveness and of
relevance is not due to lack of understanding of the relatively straightforward
r e t r i e v a l p r o c e s s e s , but is due to our lack of basic understanding about language,
meaning and human communication itself. " —'"Fundamentally, the study of inquiry p r o c e d u r e s is a problem in the general
psychology of cognitive functioning. Relevant problems concern the way
problems a r e recognized and formulated into questions, the way a s e a r c h planis developed to find answers to questions, and finally, the way it is decided
whether or not a possible answer matches the specifications of a question. M —A second core problem is the heterogeneous and somewhat arbitrary development of
n a t u r a l languages t h e m s e l v e s . It is much the same fundamental p r o b l e m whether men or
machines a r e to read text and determine the-"meaning" (at least, in the sense of communication intent) of m e s s a g e s expressed in a natural language. However, the p r o b l e m s
a r e aggravated if men themselves must know enough about language and its conveyancesof m e s s a g e content to specify p r e c i s e l y to a machine what it is to look for and to u s e .
Salton enumerates some of these difficulties as follows:"No well-defined set of rules is known by which the individual words in thelanguage a r e combined into meaningful word groups or sentences. Specifically,
the c o r r e c t identification of the meaning of word groups depends at l e a s t in p a r t
on the p r o p e r recognition of syntactic and semantic ambiguities, on the c o r r e c tinterpretation of homographs, on the recognition of semantic equivalences, on
the detection of word relations, a£'d o n a general awareness of the backgroundand environment of a given utterance." 3 /1/
Giuliano, 1963 [ 2 3 0 ] , p . 6.
Stone, 1962 [ 5 7 6 ] , p . 1.
3/
Salton, 1963 [ 5 1 9 ] , p . 1-2.

145

Similarly, Baxendale states:"We are confronted with difficulties which arise from the multiple ways inwhich words and sentences are put together to convey meanings and shadesof meaning - - i. e . , to represent ideas and concepts. Research into thisproblem - - drawing upon psychological and logical analysis - - i s scarcely

begun. n J/

A third core problem is the proper choice of appropriate selection criteria ifcondensed representations of document content must be used for scanning, search, andrelevance decisions. Swanson suggests that the price paid for brevity of representation
so that searching operations can be efficiently managed is the l o s s of at least some,perhaps most, of the information in a collection or library. He notes also that:"It is another obvious but seldom remarked fact that the extent of suchinformation loss for existing libraries is not only unknown but has neverdefined in measurable terms. " zJ
This l o s s is lived with, today, in many practical situations involving abstracts, index
term s e t s , selective-dissemination notices, and even mere author-title listings inannouncement bulletins or search output products from either manual or machine
searches. Yet the sheer increase in volume of the total number of items to be covered
and of the number of items potentially responsive even to a single individual's interests
has severely stretched any individual's capacity to scan or skim, much l e s s read, thepresumably pertinent material - - documents themselves, abstracts of other documents,
listings of documents available - - already accumulating on his desk.
Condensation, reductive representation, becomes more and more imperative.
Concurrently, while conventional tools may be lived with, after a fashion, the sub stitution of machine-compiled or machine-produced alternatives, even though they givethe same information in the same volume, number of pages to be scanned, may becauseof such things as inferiorities of page and line formatting, size of type on the page,
limitation of ttypography to upper case and a few other symbols, make the problem of how
adequate the user judges the selection and condensation to be, that much worse.
A fourth problem in evaluation, therefore, is the question of whether or not thebenefit to u s e r s is worth the cost. For example, despite the arguments for concept
rather than word indexing, for assignment of labels rather than m e r e extraction of a few
words used by the author himself, at least some data on the use made by scientists ofvarious sources of information on material which might be of interest to them suggests

U
Baxendale, 1962 [ 4 2 ] , p. 68.

U
Swanson, I960 [ 5 8 2 ] , pp. 5-6.

146

that subject indexes a r e not the m o s t important s o u r c e , n o r even a major s o u r c e . H e r n e rfound, for example, that only about 16 p e r c e n t of his respondents reported u s e of indexes
and a b s t r a c t s as p r i m a r y tools in l i t e r a t u r e s e a r c h e s . He r e p o r t s , for the use of tools in
becoming a w a r e of c u r r e n t s o u r c e s of information, 477 ol 3832 r e s p o n s e s indicating theuse of indexing and abstracting publications as against 486 using footnotes or other citedr e f e r e n c e s , }J 291 using l i b r a r y acquisition l i s t s , and 212 using s e p a r a t e bibliographies(Herner, 1958 [ 2 6 5 ] ) .
These data, and s i m i l a r findings of Fishendon that 17 p e r c e n t of scientists queriedconsidered the scanning of titles in accession l i s t s and announcement bulletins a principalmeans to find information of i n t e r e s t , U suggest that KWIC type indexes may be adequate
for many p u r p o s e s . On the other hand, the KWIC index to the U . S . Government R e s e a r c hReports made available to the public on an experimental b a s i s through the Office ot
Technical Services was discontinued after a year of subsidized operation because too few
of the u s e r s indicated willingness to pay a fee in o r d e r to have the s e r v i c e continued ona subscription b a s i s .
The evaluational p r o b l e m h e r e involves the lack of information on indexing c o s t s ,the relatively few quantitative and objectively validated studies that have been made ofu s e r n e e d s , the question of whether what the u s e r says he does or wants is what he reallywants or does, and the m a t t e r of defining " i n t e r e s t " for different u s e r s with differingp u r p o s e s and r e q u i r e m e n t s . The concept of " i n t e r e s t " is taken to mean the motivationsof a p a r t i c u l a r u s e r or group of u s e r s at a p a r t i c u l a r time, while the equally i m p r e c i s e
notion of " r e l e v a n c e " r e f e r s to the value judgments made by the u s e r as to the relationof an item to his query or i n t e r e s t .
A final core problem, then, is that of the question of relevancy itself, involving
recognition that "relevancy is a comparative rather than a qualitative concept . . . (and). . . that a docToiment of little relevancy in the eyes of X might well be highly relevantin the eyes of Y . " zJ Mooers s t a t e s , s i m i l a r l y , that:" T h e r e is no absolute 'Relevance' of a document. It depends upon the person
and his background, the work and the date. What is not relevant today may be
relevant tomorrow. " —'Good d i s c u s s e s various possible m e a s u r e s of ' r e l e v a n c e ' - logical m e a s u r e s , frequency
m e a s u r e s , r e f e r e n c e s to, citations of, i n t e r e s t m e a s u r e s , linguistic m e a s u r e s , 2 /17Note that H e r n e r ' s data and those of d l a s s and Norwood, 1958 [ 2 3 2 ] , reporting
6. 9 p e r c e n t u s e of c r o s s - c i t a t i o n s in another paper as the method of learningof important work as against 1.2 percent using an indexing s e r v i c e , appear tor e - e n f o r c e the claims of those who advocate citation indexing.

II
Fishenden, 1958 [ 197], p. 163.

II
B a r - H i l l e l , 1959 [ 3 3 ] , p. 4 - 8 . 4 .

±1
M o o e r s , 1963 [ 4 2 3 ] , p . 2.
5/
Good, 1958 [ 2 3 4 ] , pp. 7-9.

147

but except for the obvious statistical criteria, the problems of how to measure relevancy
remain largely unresolved.
At least some data on the variability of relevance judgments is available in reportsof the performance of an SDI (Selective Dissemination of information) system. In such
systems, the indexing terms or tags assigned to a new item are compared with a file of"user-profiles" that i s , with a pre-prepared listing of terms or topics in which aparticular user is interested. Where the term-profile of a new item matches that of auser, a notification of the acquisition of that item is sent to him. Barnes and Resnick
report tests of such a system in which pseudo-notifications selected randomly were
included with those produced from the matching procedure. Account was kept of whichnotices were regarded by the users as meeting their interests and which were not. They
found that 58.1 percent of the non- random notifications were regarded as relevant, butthat so also were 26. 8 percent of the random ones. 2 /Katter comments on findings that the inter subjective agreement of typical users
with respect to value judgments of condensed representations of text is low. He
suggests:"One source of this low inter subjective agreement among u s e r s may be that it isoften not clear what is intended by the words relevant and representative. Considerations such as the validity of the material, its usefulness, stylistic qualities,
understandability, conceptual prefer ability, e t c . , can all enter their judgments inunknown amounts. " 2/Corroborating evidence is available from other sources. Swan son, in his tests ofa natural language text searching technique, had first used subject matter specialists torate the relevance of each of the text documents to each of 50 questions. Two individuals
rated each item, and if they disagreed significantly, a third person was asked to reconcile
the difference.
In spite of this, 8 percent of the cases of failure to retrieve ''relevant 1 'documents were ascribed to incorrect initial judgments of relevance, and 15 percent of thepresumably "irrelevant" documents were finally judged to be relevant after all (Swanson,
1961 £586 1 ). In Swanson's words: "The question of formulating criteria for judging therelevance of any document to the motive, purpose, or intent which underlies a request for
information is profound and l i e s at the heart of the matter. " A'1/
Barnes and Resnick, 1963 [ 3 6 ] , p. 2.
Katter, 1963 [ 3 0 8 ] , p. 24.
Swanson, I960 [ 5 8 7 ] , p. 1099.

148

7. 2

Bases and Criteria for Evaluation of Automatic Indexing Procedures

What should the bases be for the evaluation of existing or proposed indexing systems
that rely, to a greater or l e s s e r extent, on machine generation of the indexing or c l a s s i ficatory labels? Since the evaluation of quality of indexing per se raises such fundamental
and elusive questions, can these questions be begged for the case of automatic indexing as
they are in fact for almost all manual systems? If so, the obvious bases are those oftime, cost, availability of alternative possibilities, and customer acceptance. Here again
we are faced with a dearth of objective data, even for the inter comparison of any two
manual s y s t e m s .
In the two years preceding the ICSI Conference, the Program Committee openly
solicited papers that would provide comparative data for operating information systems
and that would develop and discuss criteria for the comparison of systems. U Nevertheless, of the papers received only two were responsive to this invitation: the special
case of comparing the conventional file against the inverted file approach to the searchingof chemical structure data (Miller et al, 1959 [419])i and an early report by Cleverdon
on the ASLIB Cranfield project for the inter comparison of indexing systems, under agrant from the National Science Foundation (1959 C 1261).
There had been an earlier comparative experiment, generally conceded to be thefirst of its kind,in which 98 search requests were run by ASTIA personnel using a
conventional catalog and by personnel of Documentation Inc. , using a coordinated Uniterm
index. Warheit says:"Unfortunately, the conditions of the test were very poorly designed so that,in the final analysis, each group was the sole judge both of the scope of theoriginal request and of the adequacy of the bibliographies produced. The
resulting claims are of course contradictory. tf _'1/
See "Proposed Scope of Area 4 , " Proceedings, ICSI, 1959 [ 4 8 l ] , pp. 665-669.

DCompare, for example, Gull, 1956 [ 2 4 6 ] , p. 329: "When one considers that afairly thorough search of the literature indicates that this comparison of two
reference systems is the first undertaken so far, it is not surprising that theresults reveal clerical errors and an incomplete design of the test. "II
Warheit, 1956 [ 6 3 1 ] , p. 274.

149

However, some of the findings are pertinent to our present questions of evaluation.
Thus, of 492 items selected by Documentation, Inc. , that ASTIA considered pertinent but
had not selected, 98 were m i s s e d by them although the proper subject heading was
searched and the catalog card had adequate selection clues, 89 were missed because not
all applicable subject headings were searched, 21 were missed because the original
subject heading assignments had been inadequate, 7 were missed because neither title nor
abstract provided indication that the report itself was pertinent to the request, and 102
were missed "because the subject heading did not occur to the searcher or because there
were so many cards under the subject heading that the searcher was discouraged 11 . 1 /Similarly, Gull reports, of 318 items selected by ASTIA that Documentation, Inc.
personnel considered relevant but had not themselves selected, 97 were missed because
the searcher did not consult the proper terms.
7. 2. 1 The Cranfield Project
The inauguration of the Cranfield project is itself indicative of a prior lack ofobjective standards as applied to the measurement of effectiveness of informationindexing, selection and retrieval systems. j?/ Beginning in 1957, and still continuing withrespect to individual indexing devices such as synonym controls and role indicators, thiswork has attempted to compare different indexing systems (e. g. , UDC, Uniterm, etc.)
under different indexing conditions (e. g. , type of training of indexer, length of timeallowed to index) against proposed measures of "retrieval effectiveness". These
measures are, respectively, the recall ratio, or the percentage of relevant documents
retrieved as against the total number of relevant documents known to be in the collection,and the relevance ratio, or the percentage of relevant documents among those actually
retrieved.
In the first Cranfield tests, on 18,000 documents, it is reported that the recall ratio
ranged between 75 and 85 percent for all four indexing s y s t e m s . 2 / These results are
1/
Gull, 1956 [ 2 4 6 ] , p. 329.

1/
Compare, for example, Randall, 1962 [ 4 9 2 ] , pp. 380-381: "Prior to 1957, theproponents of the various indexing and classification schemes, the universal
decimal system, the alphabetic subject heading, the Uniterm system and facetedclassification touted their own system on the bases of subjective evaluation andtheoretical investigations. There were many claims and much supposition about
the relative merits and benefits . . . but there was no body of data from which an
objective evaluation could be made. . .Many observers believe that the Cranfieldstudy constitutes the most important work done in the field of cataloging inrecent times. "IICleverdon, et al, 1964 [ 1 3 0 ] , p. 87.

150

rather better than reported by others — and have been subjected to specific criticisms
although these first tests were limited to the recall of the source documents on which
the test questions were based. For non-source documents there would of course also
be questions relating to the core problem of how relevance is to be judged. Thus Markus
says:"Despite investigations by Cleverdon in England, and by many others, there istoday no generally accepted method of comparing the effectiveness of different
types of indexes. The needs of index users vary so greatly that even the mostcarefully planned tests of retrieval efficiency can be challenged. " zJ
Notwithstanding such criticisms, however, and in spite of the fact that the Cranfieldtests have so far been directed principally to indexing systems applied manually, certain
findings and conclusions reached by Cleverdon and his associates are pertinent to thequestions of evaluating automatic indexing procedures. Examples are:
"The fact is that no indexing sleight of hand, no indexing skill, can produce asystem in which a figure for recall can be improved substantially without
weakening the over-all relevance, i. e . , the number of documents that arereally relevant compared with the total number retrieved.
"The majority of the failures (60 percent) were due to inadequacies and inaccuracies (carelessness rather than lack of knowledge) in the indexing process.
However, supplementary tests, in which the staff of outside organizations carriedout the indexing revealed that the Cranfield indexers were achieving a standard
above average. This seems to indicate a certain inevitability of human weakness
and error in the indexing process and lends some support to the many current '.
research projects that are investigating the feasibility of automatic indexing. " —7. 2. 2 O1 Connor's Investigations
As O'Connor has cogently observed on a number of occasions, the question ofwhether or not automatic indexing i s possible is not the real question. Rather, theproblem is whether or not indexing by machine is capable of producing results that are"good enough" for retrieval purposes, raising in its turn the still more basic question of
how "good retrieval" can be evaluated. His own approach in detailed investigations has
1/See, for example, Johnson 1962 [ 300], p. 90: "The amount of meaningful
information that can be retrieved is too small. There are few available studies
on this subject. But these s e e m to indicate that, under some indexing schemes,
meaningful retrieval can run as low as 10 and 15 percent and that the most thatcan be optimized for any of them, even under highly motivated conditions, isaround 70 percent. "2/Markus, 1963 [ 3 9 4 ] , p. 16. See also Kochen, 1963 [ 3 2 7 ] , p. 12: "The outstanding large-scale and realistic experimental work is that of Cleverdon.
Unfortunately, his results are not very decisive. "3/

Cleverdon et al, 1964 [ 130], pp. 8 6 - 8 7 .

151

been to study an existing system (e. g . , using Merck, Sharp and Dohme data) with respect
to indexing terms such as "penicillin, l f "toxicity, l f and "mode of action." He thenattempts to define various possible machine assignment rules, and then to determine theprobable over-and-under assignments that would result from the application of theserules.
Typical results pertinent to both questions of word-indexing evaluation and of interindexer consistency showed that for 23 documents indexed under the term "toxicity, " 1 1
did not contain the stem "toxi. . . " at all; th*at 17 items indexed under "penicillin" contained
the word at least once; that none of 34 randomly selected documents not indexed under"penicillin" contained the word, but that 7 of 28 items not so indexed but selected asprobable candidates from title and other clues did contain the word. (O'Connor, 1961[447])
Typical suggestions, comments, and conclusions made by O'Connor include thefollowing:"It might be required that the mechanized indexing permit as good (or no worse)
retrieval as existing human indexing, because it is desired to free the subjectskilled indexing personnel for other work. Or poorer retrieval (than possible
with human indexing such as is presently done of comparable material) might beaccepted from computer indexing, because poorer retrieval is better than none .
and there i s a shortage of subject-skilled people to do the additional indexing. " —"Such considerations as the following are relevant. Over-assigning can increase
input costs and storage (to an extent dependent on the storage system), butmechanizing indexing might be worth the cost. Over-as signing might also
increase the number of irrelevant documents retrieved, but the increase might
be insignificant. " zJ". . .Suppose terms A, B, and C each correctly characterize five percent of aten thousand document collection, each term is overassigned to another five
percent, and over-assignment of each term occurs independently of the correct
assigning and over-as signing of the others. Then about nine documents will be
extra for the search question A & B k C. " zJ
"The question of permitting some under-as signing, that i s , the computer failingto assign [ a term] T to some document which should have it, is more delicate.
Human indexers sometimes underassign. If we knew the rate of ounder as signingby human indexers for a term T, we might consider allowing the computer asimilar rate. However, some c a s e s of under as signing might be more important
than others and if the computer made more important mistakes than the humanindexers, retrieval might not be 'good enough 1 . "yO'Connor, I960 [ 4 4 4 ] , p. 3.
O'Connor, 1 9 6 l [ 4 4 8 ] , p. 199.

1'

O'Connor, I960 [444], p. 6.

4/Ibid, pp. 6-7.

152

Other typical points made by O'Connor include the possibilities that the u s e ofautomatic indexing techniques might free trained technical people for other work, that it
might p e r m i t m o r e indexing than is now possible with available r e s o u r c e s , that it might
cost l e s s , and that it might produce a better or m o r e consistent indexing product.zJ
With r e s p e c t to the l a t t e r point, however, he points out that g r e a t e r consistency might not
in itself be a virtue, since the product although generated m o r e consistently might berelatively worthless by comparison with the inconsistent human product. z\! Especiallypertinent to the question of judgment factors in evaluation was a comparison of the mostfrequent words selected by the Luhn "auto-encoding" technique as applied to an ICSI paper
against a q u a s i - r a n d o m word list for the same paper produced by selecting the l a s t noncommon word on every page, and the first such word on every second page. He r e m a r k s :"The important point of this q u a s i - r a n d o m l i s t for my p r e s e n t purposes is toemphasize that first i m p r e s s i o n s might not be at all a good way of judging
the adequacy of an index set, f f zJ
7. 2. 3 Questions of Comparative Costs
The paucity of objective data on the effectiveness of indexing s y s t e m s generally
extends to even such obvious questions as costs of indexing and time required to index.
These very questions might, in fact, be decisive with r e s p e c t to choice between manualand machine s y s t e m s . It has been estimated by some that the costs of manual subject
indexing amount to close to 75 percent of the costs of operating an information selection
and r e t r i e v a l system, ZJ yet very little actual data on costs has been reported in thel i t e r a t u r e . —' Exceptions a r e , for the most part, limited to r a t h e r special c a s e s , suchas the following examples:1.

A total cost of l e s s than $30, 000 is r e p o r t e d for a 10, 000 document
collection at Aeronutronic. F o u r m a n - y e a r s of effort w e r e required.
On average, 12. 6 a c c e s s points w e r e provided per document, of which
9.2 w e r e subject-indicating d e s c r i p t o r s chosen, with some modifications,from the second Edition of the ASTIA T h e s a u r u s . "This favorable figure
was possible because an adequate r e a d y - m a d e thesaurus of indexing t e r m s
was available and because the ' p e e k - a - b o o 1 type equipment used was much

1/
O'Connor, 1962 £ 4 4 7 ] , p. 267.

l)
1'
±1

O'Connor, 1963 [443J, p. 16.
O'Connor, 1962 [.447J, p. 270.
O f Connor, 1963 [ 4 4 2 ] , p. 1.

5/

See, for example, A. D. Little, Inc. ( 1963 [ 2 3 ] , p. 5): " P e r f o r m a n c e and cost dataon existing large documentation s y s t e m s a r e surprisingly s p a r s e , and cost data
have r a r e l y included adequate overhead and depreciation accounting. "153

l e s s expensive than most other devices offering comparable speedof operation and search logic possibilities," 1/
2.

"The experience of libraries that have gone through indexing using
links and role indicators and careful editing shows that indexing takes
about one-half hour per document (or $4.00) and costs an additional$1. 00 for routine processing. n JJ

3.

In an investigation of the comparative merits of manual indexing of 2, 000
documents using the UDC classification system as against a KWIC index,
Black gives the figure of approximately $1400 for the UDC case compared
to about $600 for an in-house computer operation to produce KWIC listings,
and somewhat more for a KWIC index compiled by a service bureau.—'

Time required to index, which directly involves cost, is reported by Clever don tovary widely:"Few reliable figures have been given for current practices, although a particularlyhigh figure is the 1 1 / 2 hours average quoted for indexing reports for the catalogueof aerodynamic data prepared by the Nationaal Uuchtvaart laboratorium in
Holland. It appears from personal discussions that an average of 20 minutes fora general collection of technical reports i s the top limit, and this has been taken
as the maximum indexing time to be used in the project."Insofar as such meagre data i s indicative, there does not appear to be any particularcost-advantage for machine-compiled and machine-generated indexing other than the titleonly KWIC indexes. Thus, Olmer and Rich report, in part:"The program • . • lends itself to a variety of applications. One of these . . . isestimated to cost roughly $4.00 per document for cataloguing, putting on tape,printing and making any necessary corrections." 5/This is for a case where the indexing (cataloging) is done manually.
For a specific proposed automatic indexing system, employing a modified versionof the Luhn word-frequency counting selection principle, Gallagher and Toomey reportthat:yLinder, 1963 [ 3 6 l ] , p. 147.

U
Lockheed Aircraft Corp. , 1959 [ 3 6 9 ] , p. 93.

IIBlack, 1962 [ 6 5 ] , p. 318.

±1
Cleverdon, 1959 C 126], p. 690.

I!
Olmer and Rich, 1963 [ 4 5 4 ] , p. 182.

154

"For the documents in our system, we estimate that processing time will be
about 20 seconds per thousand words • . . The cost is approximately $3. 50
per minute when averaged between prime and extra shift. n A/
This means that the cost of processing a 3, 000-word document would be $3. 50 , exclusiveof the costs of keypunching the input text which, conservatively estimated, costs not l e s sthan 1-2 cents per word. £/ Swanson similarly assumes either that machine-us able text
is already available or that editing and keystroking efforts are separate costs in arriving
at an estimate of $1. 00 per item for automatic indexing. zJ
These quantitative estimates bear out the more subjective conclusions of suchinvestigators as Bar-Hillel, O'Connor, and others. Examples are:
"It is very likely that manual Uniterm indexing by cheap clerical labor will still,on the average, be qualitatively superior to any kind of automatic indexing, and
it is very unlikely that the cost of automatic indexing will ever be l e s s than thiskind of manual Uniterm indexing, unless the automatic indexing is to be of suchlow quality as to totally defeat its purpose." Z'"Most of these techniques require that the full texts of documents be in machinereadable form. At present this usually requires keypunching which is much
more expensive than a specialist's indexing efforts." zV
Gallagher and Toomey, 1963 [ 2 0 5 ] , p. 52.
21'Compare, for example, Ray, 1961 [ 4 9 6 ] , p. 55; Swanson, 1962 [ 5 8 4 ] , p. 470:
The cost is roughly one or two cents per word which by standards of what isnormally spent for even the most thorough indexing and cataloging, is
exorbitant." Mersel and Smith report 1964 [ 4 1 5 ] , p. 10A) typical TRW costsof keypunching as two cents per word for Russian technical text, and one centper word for English. They also cite cost figures as low as half a cent per
word at the CIA-Georgetown Keypunching Center in Frankfurt and at IBM, but
this is exclusive of overhead and computer processing ( e . g . , editing program)costs, so that the one cent figure appears minimal as of today. However,
Kochen reports (1963 [ 3 2 7 ] , p. 7): "While keypunching of text cost roughly one
cent/word, new means for recording spoken (and writte'n) text using a stenokeyboard tied to a photodisc storing a Stenocode-English dictionary could possibly
reduce the cost to 1/3-cent per word. tf

uSwanson, 1962 [ 5 8 4 ] , p. 471.

M

Bar-Hillel, 1962 [ 3 5 ] , p. 418.

5/

O'Connor, 1963 [ 4 4 3 ] , p. 1.

155

7.2.4

Summary:

Potential Advantages as Bases for Evaluation

In view of the difficulties engendered by the underlying core problems, thecriticisms that can be brought against tests of "retrieval effectiveness", the general lackof comparative data and standards of measurement, the question of evaluation of automatic
indexing procedures largely reduces to the weighing of potential advantages and disadvantages. In the case of such procedures as KWIC and citation indexing, some of these
possibilities, both pro and con, have been discussed previously. In general, suggested
bases for evaluation reflecting operational considerations may be summarized as follows:1.

Speed and timeliness

2.

Relative economy

3.

Consistency and reliability—'

4.

Elimination of the need for further human intellectual effort after
initial planning and programming has been done.

5.

Providing a product that could not otherwise be obtained.

6.

2/
Ease of updating and revision of indexes so produced.—'From the point of view of possible operational advantages, these may be combined
into the single criterion:
The achievement of a more effective and more economical balance between themeeting of the objectives of the indexing system and the utilization of available
resources.

yuCompare McCormick, 1962 [ 4 0 9 ] , p. 182: "A computer is objective in its operations
and it can be repetitive. If given a certain amount of information about a document,
it i s always able to index the document in a consistent manner. This consistency isdesired so as to avoid the situations where a person might index a document differently on various occasions, or where it would be indexed differently by anotherperson when there appears to be no good reason for a difference. " Note, however,
O'Connor's point previously mentioned, (1963 [ 4 4 3 ] , p. 16): "It has been argued
that mechanized indexing has the advantage of consistency. • • However this argument by itself says very little in favor of mechanized indexing. For two humanly
produced index sets for a document which differ somewhat may both be quite useful,though imperfect, while the index set which the same program will always reproduce
for the same document may be worthless. nSee, for example, Youden, 1963 [ 6 5 8 ] , p. 332:
"The facility with which indexes may be updated and the ease of selecting items for
special bibliographies will result in the majority of indexes being computer produced
before many years. "156

However, the question of the objectives of the system brings us back full circle to thequestions of purpose in terms of particular requirements, of quality, and of how tomeasure either purpose or quality. Thus we may determine that an automatic indexing
procedure produces a product at least as rapidly, at least as inexpensively, at least as
consistently as human indexing operations would, and with substantially l e s s investment of
manpower resources. However, will this product be as useful or as "good11 as the human
product?
In view of the many caveats about the present quality of indexing systems—' and thelack of standards for measuring quality, 2 / it is important to recognize that we should
compare the products of automatic indexing methods "not with hand-crafted excellence, butwith the average, the routine output of the over-burdened subject analyst working with thedeficiencies of any other indexing system 11 . 3 / Such deficiences include the criticalquestion of how well and how consistently the system, whatever it i s , is applied in practiceby the human analysts.
7.3

Findings with Respect to Inter-Indexer and Intra-Indexer Consistency

Very few objective studies, despite the obvious relationship to the general questionsof quality, pertinency, and reliability of indexing, have as yet been made of inter-indexer
and intra-indexer consistency. Perhaps the first investigation both to obtain experimental
data and to analyze the observed types of failures to achieve correct assignments was thatof Lilley. 4 / He took the answers made to 6 questions by 340 students entering a graduate
library school, wherein they were asked to write down the subject headings which theywould expect to be applied to other books on the same subject as 6 "sample books" in asystem such as the Library of Congress card catalog. Lilley reports:\J

See, for example, in addition to comments by O'Connor and others previously quoted,
Helyar, 1961 [ 2 6 2 ] , p. 110: "The general current of feeling of the meeting as r e flected both in the papers and in the discussion is that the standard of indexing is notnearly adequate;" Artandi,
1963 [ 2 2 ] , p. 1.: " . . . 'Good indexing 1 as such has
not been defined satisfactorily and is the function of many variables, some known, others not yet identified"; Tritschler, 1963 [ 6 1 0 ] , p. 5: ". • . 'Good*indexing is e x tremely difficult to describe and 'perfect' indexing is impossible to define ormeasure. "2/

See Cleverdon, I960 [ 124], p. 429: "The most important requirement in information
retrieval is a recognized standard of measurement and after that we need a s a t i s factory method of measuring. Only when these have been found will it be possible toknow for certain whether any new system of indexing or retrieving information i s an
improvement on previous methods. At present all those trying to solve the problems
of information retrieval are working very much in the dark, uncertain as to the real
problems and quite unable to apply any measurements to their proposed solutions. "1/

Kennedy, 1962 [ 3 1 1 ] , p. 126.

4/

Lilley, 1954 [ 3 6 0 ] : See also Vickery, I960 [ 6 2 6 ] , p. 4.

157

"A total of 2245 headings were suggested, averaging 1. 1004 headings per book per
student. These headings represented 373 different varieties, of which 368 weredifferent from the headings traced on the Library of Congress cards for the sample
b o o k s . . . As an average 62. 17 different headings were suggested for each book. • •"When the 368 different varieties of incorrect headings were analyzed in accordancewith certain criteria that had been set up, it was found that incorrect specificity wasa factor in 93. 48%, incorrect terminology in 79. 08% and incorrect form of entry in
72. 28% of the headings. • • Over half of the incorrect headings (54. 62%) had somecombination of two errors, and almost half (49. 73%) could have been converted into
•correct 1 headings only by changing the level of specificity, and by revising the t e r m inology, and by altering the f o r m . . .
"It was also found, contrary to the general assumption that failure in specificity
almost always means that the reader is approaching his subject from too broad apoint of view, that of those headings in which an incorrect level of specificity was af a c t o r . . . 64. 82% were too broad and 35. 18% were too narrow. f l \JLilley then asks the rather plaintive question as to what would happen, given that his quite
homogeneous group of subjects, all of them college graduates and all seriously interested
in librarianship, could come up with more than 62 different headings, on average, forevery heading actually used in the catalog, if his test group had included a larger numberof subjects with more heterogeneous interests?
In 1961, Macmillan and Welt investigated the duplicate indexing of 171 papers in alimited area of the medical sciences (1961 [389]). In only 18 percent of the cases was theindexing identical or nearly so. About a third of the papers had been indexed so differently
that there was no common correlation. For the rest, terms were used in one case thatwere m i s s e d in the other.
Some brief data on inter-indexer consistency is also provided by Kyle (1962 [342])
for two indexers applying her classification system to 246 arbitraily selected French andEnglish items in the field of political science. Of these, 160 were indexed the same way
by both indexers, for a consistency figure of 70 percent. Tritschler noted that no items
were indexed the same way a second time as they were the first, in small-scale experiments involving 20 documents independently indexed by 7 different people. 2 /Painter (1963 [460]), in her study of problems of duplication and consistency of
subject indexing of the reports handled by the Office of Technical Services, proceeded by
selecting items from the announcement bulletins of agencies contributing to OTS, having
these items re-indexed in the various agencies, and comparing the results with the original indexing assignments. At ASTIA, 94 items were re-indexed, with 1, 239 terms having
been assigned to them originally and 1, 119 assigned on the re-run. Overall, 62 percent of
those terms originally assigned were also assigned the second time, and 69 percent of thesecond-time terms had also been assigned originally. However, 111 of the starred d e s criptors (which are of the most significance in the ASTIA system) were used the first timeand not the second, while 98 were used the second time but not the first.
1/

Lilley, 1954 [360], pp. 42 and 43.

2/

Tritschler, 1963 [610], p. 5.

158

At AEC, 96 items were re-indexed to the subject heading scheme used in Nuclear
Science Abstracts, There had been 249 headings assigned to these items originally and406 were assigned on the second run, for an overall consistency rate of 54 percent, butwith 53 percent of the headings used the second time not haying been used the first. The
sample checked at OTS consisted of 32 items to which 346 descriptors had been assigned
the first time and 413 the second. The consistency was 65 percent with respect to the firstrun and 54 percent with respect to the second. Finally, at the National Agriculture Library
99 items were checked, with results showing a high consistency rating and a similarity ofindexing between the two runs of 86 percent. Painter concludes:"The consistency rates are not encouraging. Apparently there is little difference
between preparation for a manual system and that for a machine system. The percentages indicate that there is no significant difference between consistency where
two or three headings are assigned and where twelve or sixteen are assigned.
Therefore, we are left with the fact that regardless of these variables, consistency
rates range between 60 and 72 per cent. f f \J
Jacoby and Slamecka report even l e s s encouraging data (1962 [293]). "In general,
the inter-indexer reliability was found to be low (in the vicinity of 20 per cent), the intra indexer reliability somewhat higher (about 50 per cent). " For a s e r i e s of tests of indexingof a group of chemical patents by three experienced and three inexperienced indexers, theyfound that the beginners had average matchings among the terms assigned by them to thesame documents of only 12. 6 percent and that even for the experienced indexers theaverage percent of matching terms was only 16. 3 percent. 2 / In other studies, these investigators have explored the effects of various indexing aids upon the reliability andconsistency of indexing, concluding that the use of prescriptive aids such as authority lists
improves reliability and inter-indexer consistency from 8 or 9 percent to 33 percent, while
those aids such as thesauri and association lists "which enlarge the indexer's semantic
freedom of term choice" are detrimental (Slamecka and Jacoby, 1963 [560]).
Rodger s in a study of intra-indexer consistency reports data for the re-indexing, by
the same person at a later date, of 60 documents dealing with the United Arab Republic
taken from The New York T i m e s . She reports that the average consistency over all 60
documents was 59 percent. 3 / In a further study of inter-indexer consistency, 20 papers
from Area 5, ICSI, were key-word indexed by 16 people all of whom were familiar withthe subject matter, (although only 8 completed all 20 papers). Results are given in termsof the proportions of the total number of unique words chosen by 100 percent of the subjects(. 008) half of them (. 14) and only one of them (. 52). 4 / Study of the results in terms of
the proportion of words selected in common by any pair of these indexers to the total
number of different words selected by them both gave a "grand mean agreement for all
two-person combinations for the 8 s u b j e c t s . . . [ o f ] . . 24 percent against all 20 articles. " 5 /The mean percentage of overlap between Luhn's word-frequency selection technique (asapplied to the same papers) and any one or more indexers who agreed was • 15.
yPainter, 1963 [ 4 6 0 ] , p. 94.

2/

Jacoby and Slamecka, 1962 [293], p. 16.

3/Rodgers, 1961 [ 5 0 4 ] , p. 12.

4/Rodgers, 1961 [503], p. 50.

5/Greer, 1963 [ 2 3 9 ] , p. 10.

159

Still further studies of indexer consistency investigated at the Information Systems
Operation division of General Electric have just recently been reported (Korotkin andOliver, 1964 [331, 332]). In particular, the investigators report on the effects of subject
matter familiarity and on the use as a job aid of a reference list of suggested descriptors
upon inter-indexer consistency* The material for test consisted of 30 abstracts drawn
from Psychological Abstracts, to be indexed by 5 psychologists and 5 non-psychologists in
two sessions, with and without use of the "job aid11. Results in terms of mean percent
consistency were reported as follows:
Session I
Session II"Group A (Familiar)
39.0%
53* 0%
Group B (Non-familiar)
36. 4%
54. 0%" \J
Corroborating evidence of a generally low rate of inter-indexer consistency isprovided by noting instances of duplicated indexing that may occur in regularly issued
announcement bulletins. During current awareness scanning of the DOC (ASTIA) "TAB"in recent months, members of the staff of the Research Information Center and Advisory
Service on Information Processing have caught more than 20 cases of duplicate and even
triplicate indexing of the same item. (Two examples can be discovered in Figure 8 a andb). For the 52 independent assignments involved, for these items the average interindexer consistency is only 46. 1 percent.
On the general subject of indexing consistency, Black comments as follows:"There have been enough experiments to indicate that there is no consistency, orvery little, between one indexing performance by a given individual and anotherindexing performance, at a later date, by the same individual. The same inconsistency has been discovered among different individuals all indexing the same documents. Thus there i s neither inter-indexer consistency nor intra-indexer c o n s i s tency in any system that depends on human performance. " 2 /There can be little doubt that the quality and consistency of most human indexing,
practically available today, is not good. Much of it, because of time and other pressures,
is either directly a word-extraction process, or it i s inconsistent in assignment of many
relevant descriptors and subject category labels. On the other hand, today's indexing,
whether accomplished by man or machine, is probably no better and no worse than any
other classificatory or indexing procedures. The only excuse, therefore, for choice
between man and machine is the cost/benefit ratio which i s related on the one hand to
specific operational considerations and on the other to the question of whether or not
various indexers, and various u s e r s , would agree with the machine as much as they agreewith each other.
Before turning to some of the operational considerations affecting the cost-benefit
ratio, however, certain special factors should be briefly mentioned.
7. 4

Special Factors and Other Suggested Bases for Evaluation

The difficulties and problems of evaluation so far considered are generally applicable
to any indexing system, whether manual or automatic. Certain special factors arise, however when we consider some of the proposed automatic assignment and automatic c l a s s i fication techniques. In addition, the prospects for computer processing hold at least the
JJ7

Korotkin and Oliver, 1964 [ 3 3 1 ] , p. 7.

2/Black, 1963 [ 6 4 ] , pp. 16-17.

160

AD-408 841

Div.

32

Joint Publications Research Service, Washington,
D. C.
UNDERWATER FISHERY RESEARCH IN THE USSR,
by V. P. zaitsev. 2 Apr 63, U p . 18501
Unclassified report
Trans* of Okeanologiya (USSR). 1962. v. 2, no. 6,
pp. 961-969. Also from OTS for $.50 as rept.
63 2 H 8 1 .

TECHNICAL JOURNALS NO. 190 (BIOLOGY AND MEDICINESERIES).
29 May 63, 27p. 19470
Unclassified report
Consists of abstracts of articles from selectedscientific and technical journals of Bulgaria,
Poland and Yugoslavia. Also from OTS for $.75
as rept. 63-21948.
Descriptors: ("Abstracts), Bibliographies,("Biology)t (*Meditfine), Genetics, Blood,
Drugs, Pharmacology, Microorganisms, Biochemistry, Diseases, Neurology, Therapy,
Medical examination. Vaccines, Viruses,
Plants (Botany), Scientific personnel.
Toxicity.

Descriptors: ("Fishes), Scientific research.
("Oceanology), Marine biology. Ocean currents.
Diving, ("Oceanographie equipment), (•Underwater equipment). Submarines.

AD-408 849

Div.

32

AD-408 878

Div.

32

Joint Publications Research Service, Washington,
D. C.
TOWARD NEW PROGRESS OF SCIENCE AND TECHNOLOGY AND
IMPORTANT PROBLEMS OF SCIENTIFIC ORGANIZATION,
by M. V. Keidysh and M. A. Lavrent'ev. 20 May 63,
25p. 19283
Unclassified report

Joint Publications Research Service, Washington,
D. C.
ABSTRACTS FROM EAST EUROPEAN SCIENTIFIC ANDTECHNICAL JOURNALS NO. 187 (BIOLOGY AND MEDICINESERIES).
29 May 63, 20p. 19465
Unclassified report

Trans, of Akademiya Nauk SSSR. Vestnik, 1962,
v. 32, no. 12, p. 9 - U and 16-18. Also from OTS
for $.75 as rept. 63-21864.

Consists of abstracts of articles from selectedscientific and technical journals of Hungary.
Also from OTS for $.75 as rept. 63-21945.

Descriptors: ("Scientific research). (•Scientific organizations). Energy management,
Materials, Semiconductors, Chemical industry,
Agriculture, Computers.

AD-408 854

Div.

32

Deseriptorst("Abstracts), Bibliographies,("Biology), ("Medicine), Chemical analysis.
Drugs, Neurology, Surgery, Wounds and injuries,
Pathology, Diet, Public health. Infants,
Toxicity.

AD-408 8 8 7

Joint Publications Research S e r v i c e , Washington,
D. C.

Div.

32

Joint Publications Research Service, Washington,
D.

C.

ORDER CONCERNING COMMISSION FOR USE OF UNIVERSE
POR PEACEFUL PURPOSES NO. 36.
29 Apr 63, 2p. 18954.
Unclassified report

CYBERNETIC MACHINES: SELECTED ARTICLES.
27 Aug 6 2 , 1 5 p . 1 4 9 6 2
Unclassified report

Trans, from Slusbeni List, Belgrade (Yugoslavia)
1963 19:12, p. 163. Notice: Also from OTS for$.50 as rept. 63 21705.

Trans, from Ceninskoe Znamva (USSR) 1962, July;
Literaturnaya Gaseta (USSR) 1962, 7 July;
Pravda, Moscow (USSR) 1962, 5 July. Also fromOTS for $.50 as rept. 62-11760.

Descriptors: ("Space flight), ("Politicalscience), Scientific organisations.

AD-408 866

Div.

Descriptors: ("Cybernetics), ("Digital computers), Learning, Computer logic. Design.

32

Joint Publications Research Service, Washington,
0. C.
THE PAST TEN YEARS AT VINITI (ALL-UNION INSTITUTE OF SCIENTIFIC AND TECHNICAL INFORMATION),
by V. A. Polushkin. 29 May 63. 3p. 19482
Unclassified report

Contents:••Thinking** machines: friends or enemies, by
V. Trapesnikov
Machine rns to learn, by 6. Zelenko
Can a machine create a design, by Tu. Sinyakov

AD-408 937(TISTB/PCR)

Div.

32

Linguistics Research Center, U. of Texas, Austin.
THE CLASSIFICATION OF ENGLISH ADVERBIALS IN
CORPUS 05,
by Howard W. Law. Apr 63, 52p. LRC 63 WDE1

Trans, of Akademiya Nauk SSSR. Vestnik, 1963,
v. 33, no. 3, PP. 127-128. Also from OTS for$.50 as rept. 63 21950.
Descriptors: ("Scientific organisations).
Documentation, ("Communication theory).

Grant NSF GN 54
Unclassified report
Descriptors: ("Language, Analysis), Machinetranslation, Classification, Computers.

AD-408 877

Div.

32

Joint Publications Research Service, Washington,
D. C.
ABSTRACTS FROM EAST EUROPEAN SCIENTIFIC AND

Figure 8a.

Research conducted in connection with the classification of adverbials produced the survey presented in this paper. The resulting classification is tentative because, among other reasons.

Examples of Duplicate Indexing161

it deals only with data of a Halted corpus.
The scope of the problem and statements by someother authors are presented. The procedure ofinvestigation involved a study of adverbial
sequences and occurrences of adverbials inreference to verbals. Four classification sortings were used to aid the study. Tentative
adverbial function classes were assumed. The
results of the first three sortings were used tomodify the tentative function classes. Tentative
position classes were established. The fourthsorting was used to establish function-position
classes. (Author)

AO-408 938(TISTP/AW)

Div.

32, 15, 5Linguistics Research Center, U. of Texas, Austin.
INTRODUCTION TO FORMATION STRUCTURES,
by D. A. Senechalle. Apr 63, 17p. LRC 63 WTM2
Grant NSF 6N 54
Unclassified report
Descriptors! ("Language, Mathematical analysis), ("Communication theory, Language),
Theory, Sequences, Analysis.
This is the second in a series of papers documenting two years of mathematical research directed toward a theoretical foundation for linguistic
information processing algorithms which will be
generally applicable to natural and artificial
languages. (Author)

AD-409 050(TISTA/PCR)

Div.

32, 12

Booz-Allen Applied Research, Inc., Chicago, 111.
FURTHER STATISTICAL METHODS IN INDIRECT, BIOASSAY BASED ON QUANTAL RESPONSE,
by William S. Mallios. 28 Sep 62, 36p.
Contract D*18 064cml28l0, Task I
Unclassified report
No automatic release to foreign nationals.
Descriptorss("Statistical analysis, Biological, assay). Test methods, Tolerances, Distribution, Scientific research. Population*
In Section I, the moments of a normalized tolerance distribution are estimated by utilizing experimental technique deaths in the indirect assay. More precisely, the information gained byassuming that the probability of experimental
technique deaths is independent of dosage may,in general, yield an LD50 with greater precision.
Adjustments are given for nbneonstant natural
mortality over time. A preliminary report on bimodal tolerance distribution is also given.
(Author)

AD-409 119(TISTB/MS)

Div.

32

Linguistics Research Center, U. of Texas, Austin.
INTRODUCTION TO FORMATION STRUCTURES,
by D. A. Senechalle. Apr 63, 17p. Rept. no.
LRC63 WTM2
Grants NSF GN54 and G19277
Unclassified report

AD-409 120(TISTB/MS)

DiVi ?&$2

Linguistics Research-Center, U. of Texas, Austin.
THE CLASSIFICATION OF ENGLISH ADVERBIALS IN
CORPUS 05,
by Howard W. Law. Apr 63, 1v. Rept. no. LRC63
WDE1
Grant NSF GN54
Unclassified report

32

Joint Publications Research Service, Washington,
D. C.
BIOGRAPHIES OF SOVIET SCIENTISTS.
29 Apr 63. 38p. 18951.
Unclassified report

Descriptors: ("Vocabulary, Classification),("Language, Analysis).

Trans, of 18 selected biographical articles from
Russian periodicals, Also from OTS for $1.25 as
rept. 63 21703.

Research conducted in connection with the classification of adverbials is presented in thispaper. The resulting classification is tentative because, among other reasons, it deals onlywith data of a limited corpus. The scope of
the problem and statements by some other authors
are presented. The investigation involved astudy of adverbial sequences and occurrences ofadverbials in reference to verbals. Fourclassification sortings were used to aid thestudy. Tentative adverbial function classes wereassumed. The results of the first three sortings•were used to modify the tentative function
classes. Tentative position classes were established. The fourth sorting was used to establish
function-position classes. Criteria for de-

Descriptors: ("Biographies), ("Scientificpersonnel), Medical personnel, Personnel.
Contents: L. I. Andzhaparidze; 0. A. Baikonurov:
Yu. A. Chernikov: I.B. Galant, S.A. Gilyarevskii5
A.A. Itskovich, 6.1. Mirzabekyan: O.G. Plisan;
S.A. Poplavskii: P.F. Sams onov: A.A. SaidAkhmedov; B.M. Sosina: I.V. Tsimbler; Ya. V.
Bykov; L.A. Vulis, .I.V. Egyazarov; E.I. Zhukovskii; Nominations for positions of Academician
and Corresponding Member, Academy of Sciences
Armenian SSR.

Figure 8b.

32, 15

Effort is directed toward a theoretical foundation for linguistic information processing
algorithms which will be generally applicable tonatural and artificial languages. (Author)

Descriptors: ("Space flight, Space medicine),("Spacecraft, Space communication systems),("Astronauts, Training), ("Astronautics,
Periodicals) (Spacecraft cabins, Geology,
Space biology. Launching, Space capsules).

Div.

Div.

Descriptors: ("Language, Mathematical analysis), ("Vocabulary), Theory.

Foreign Tech. Div., Air Force Systems Command,
Wright-Patterson Air Force Base, Ohio.
AVIATION AND COSMONAUTICS (Aviatsiya i Kosmonavtika).
Sep 62, 138p.
FTD Rept. no. ST 62 9
Unclassified report

AD-409 059

AD-409 090(TISTB/AAR)

Examples of Duplicate Indexing162

promise of more objective measures of performance or quality than evaluative techniques
available today.
Examples of the special factors involved in assignment indexing techniques andautomatic classification include the question of the amount of computation required in the
inversion and other manipulations of large matrices \J and the concommitant problems of
how large a vocabulary of clue words can be used effectively and of whether some documents cannot be indexed at all because they contain none of these words. 2 / There is, as
Needham says, "no merit in a classification program which can only be applied to a coupleof hundred objects. I f 3 /In the various techniques for automatic clustering or categorization of documents,
there are serious questions of whether the groupings can be conveniently named or d i s played for the benefit of the user. 4 / Another example of special factors in the appraisalof an automatically generated classification scheme is as follows:"Operational testing is displeasing in that it puts off any verification until right at theend; it is expensive; there is not much experience on how to do it in a realistic way;
and it is ill-controlled in the sense that the practical performance of a system isinfluenced by many other factors than the classification it embodies. ! l 5/
Examples of suggested bases for evaluation made possible by machine processing
itself include proposals by Doyle and Garvin, among others. Doyle in particular suggests
the substitution for the elusive concept of "relevance" of criteria based on "sharpness of
separation of exploratory regions in which the searcher finds documents of interest fromthose in which he does not find such documents. " 6 / He further emphasizes the need fordiscriminating a particular document from other topically close documents (Doyle, 1961
[ 166]) and suggests that "this decision can never be made by a human---only by a c o m puter, which is the only agency capable of having full consciousness of the contents of alibrary. " 7 / Garvin considers the more general problems of language and meaning, andsuggests that there are two kinds of "observable and operationally tractable manifestationsof linguistic meaning", ---namely, translation and paraphrase, and that these may beinvestigated by techniques of linguistic data processing. 8 / Edmundson, however, points
out that while there i s in general only one translation of a document, there may be a s many
abstracts (and, by implication, index sets) as there are u s e r s . 9 / Thus we are back again
at the questions of purpose and relevance.
1/Compare Williams, 1963 [642], p. 162.

2j

See Mar on and Borko, various references.

3/

Needham, 1963 [ 4 3 3 ] , p. 8.

4/See, for example, Doyle, 1963 [ 1 6 2 ] , p. 6: "Several researchers have tried togroup topically close articles, usually by statistical means, but it is rather difficult
to get any benefit from this grouping unless you can represent these groups forhuman inspection. "5/

Needham, 1963 [ 4 3 2 ] , p. 2.

yDoyle, 1963 [ 1 6 4 ] , p. 200.

2/

Doyle, 1961 [ 1 6 9 ] , p. 23.

8/

Garvin, 1961 [ 2 2 4 ] , p. 137.

9/

Edmundson, 1962 [ 1 7 8 ] , p. 4.

163

8.

OPERATIONAL CONSIDERATIONS

Whatever the verdict of evaluation of one or more automatic indexing techniques,
whether of the derivative, modified derivative, or assignment type, there are certain
operational considerations and problems that typically affect any attempt to apply suchtechniques in actual production operations. These considerations, which also affect l i n guistic data processing operations in general, include input considerations, availability ofmethods or devices for converting text to machine-usable form, programming considerations, questions of format and content of output, and problems of customer acceptance ofthe machine products.
8. 1

Questions of input

Input considerations include, first, questions of the extent and availability of material which can be handled directly by the machine. This may be limited to title only, totitle plus abstract, title plus other material, \J preselected text or automatically generated extracts; or it may in a few cases extend to full running text. Possible future r e quirements may extend to the processing not only of full text but of interspersed graphic
material (equations, charts, diagrams, drawings, photographs) as well.
We have considered typical arguments for and against the limitation of input to titles
only, to augmented titles, and to abstracts in other sections of this report. The points tobe emphasized here are requirements for pre-editing or post-editing, provisions for error
detection and error correction, the time and cost requirements of conversion equipment ifmaterial is not already available in machine-usable form, and the like. As Cornelius
suggests:"Present day computers, if used for machine indexing, will be generally input
limited and will require excessive data preparation. Causes of these limitations
are: time required for translation to machine language, verification of this m a chine language, and the capability or lack of capability of correction in the input
media. f f 2 /Examples of pre-editing requirements, even for the simple case of keyword-intitle indexing, include the spelling out of chemical symbols, the encoding or the omissionof subscripts and superscripts, insertions of hyphens to prevent indexing of a word, andsubstitutions of blanks for hyphens in compound words to assure indexing of each c o m ponent. 3/ For full text, a far more extensive and elaborate set of rules and conventions
must be developed and applied. 4/ Other editing may be required for format standardlyThis may specifically include cited titles, as suggested variously by Bohnert, 1962
[ 6 9 ] , p. 19; Giuliano and Jones, 1 9 6 2 ( 2 2 9 ] , p. 10; Swanson, 1963 [ 5 8 0 ] , p. 1;
Gallagher and Toomey, 1963 [ 2 0 5 ] , p. 53; and as used in the SADSACT method, see
pp. 98-99 of this report.

Z/

Cornelius, 1962 [140], p. 42.

3/See, for example, Kennedy, 1961 [311], p. 120.

4/See, for example the sophisticated proposals of Nugent, 1959 [ 4 4 1 ] , and Newman
et al, I960 [ 4 3 9 ] .

164