9.

CONCLUSION: APPRAISAL OF THE STATE OF THE ART IN AUTOMATIC INDEXING

Notwithstanding the difficulties of evaluation we have discussed, we shall herewith
attempt to evaluate the present state of the art in automatic indexing techniques, using such
available criteria as s e e m most appropriate. First, we suggest that all of out initialquestions except possibly the last, can today be answered affirmatively. "Is indexing bymachine possible at all? 11 To this we can answer an unequivocal "yes 11 in view of the many
examples of KWIC type indexes extant and in practical use. Secondly, "Is what can be doneby machine properly termed 'abstracting 1 , 'indexing 1 , or 'classifying'?" If, by definition,
word indexing of any kind is not "properly termed. . . indexing", then, as we have seen,
automatic derivative indexing, such as KWIC, or the selection of words to serve as index
tags based upon the frequencies of their occurrence in text, is not so either.
The fundamental Luhn concept for indexing based on word frequencies i s , as we haveseen, straightforward: namely that, after disregarding the most frequent "common words",
especially those that are syntactic-function words - - articles, conjunctions, prepositions,and the like, together with those words that occur infrequently in a given text, the remaining high frequency words should give a reasonable indication of what the author was writing"about". Critiques of the Luhn position have been made on several-fold grounds:
(1)
(2)
(3)
(4)
(5)

Information-theoretic - that, in fact, the most information is conveyed by
the least frequent words.
Absolute v s . relative frequencies of usage within specialized fields.
Modifications of semantic purport by contextual and syntactic associations.
Problems of synonymity and, conversely, of orthographic ally identicalwords. \JMulti-aspect points of interest, and future need of a c c e s s to material theauthor himself did not emphasize.

The last point raises again the criticisms that have been made against derivative,
extractive or "word" indexing of all types. To repeat, although such procedures may
index "as the author himself indexed best - - in his own language", the significant points
are (1) there may be peripheral, minor, or unrecognized aspects of his topic and incidental information disclosed, of future interest to others, which the author himself is in no
special position to recognize, and (2) notwithstanding the "author's own terminology" being
current usage rather than the "fossilized" vocabulary of any previously established c l a s s i fication or indexing scheme, this very "currency" changes from field to field and, quite
literally, from day to day. Nevertheless, it should be re-emphasized that the validity ofthese criticisms is not limited to automatic derivative indexing as such, but rather isapplicable against any indexing system whatsoever, manual or machine, which i s sostrictly limited to author-terminology, author-emphases, and the consideration of the
document at hand as a self-contained entity, without regard to other documents in a c o l lection, in a particular field, and without respect to specific user needs. By contrast to
this type of limitation, more promising approaches should s t r e s s both similarities anddifferences between a new document and previously received documents, between documents "belonging" to some definable category, or not, and even, as responsive to a particular u s e r ' s profile-of-inter est, or not.

\J

See Baxendale, 1962 [ 4 2 ] , pp. 67*68: " . . . resolution of orthographic ambiguities
is a non-trivial and over-riding prerequisite for the computer processing oft e x t . . . ", p. 67.

173

Derivative indexing, whether by man or machine, is thus subject to many disadvantages. First and foremost, it is constrained by a particular individual's personal mannerof expression of concepts in language. This limitation is controlled only by his presumptive desire to communicate with some particular (more or l e s s general, or more or l e s s
specialized) audience. His choices of natural language expressions, however, will be
conditioned by at least some of the following factors:
(1)

(2)

(3)
(4)

The range and precision of his personal mastery of both general andspecialized vocabularies for a given time, place, and specialized fieldof discourse.
His personal expectations as to the probable reactions (in the sense ofeffective communication) of his intended audience to the expressions that
he does choose, involving all of the problems of different usages of technical terminology from field to field, from formal to informal presentations, from scholarly reviews to progress reports heavy in current"technese" and "fashionable words 11 .
His habits of thought and his training in his field.
His awareness of more than one possible audience and of more than one
point or topic of potential interest to his readers.

Secondly, indexing by the author's own words is remarkably sensitive to a particularperiod of time, so that the terminology becomes rapidly outdated and often seriously m i s leading in its connotations. Thirdly, the user has no advance knowledge of the terminologythat has been used in all the varied texts of a collection and he must therefore be able topredict a wide variety of possible ways of expressing ideas in words, phrases, and evenby implication. Fourthly, for collections indexed on a word-derivative basis, there islittle or no possibility for generic searching. \J Finally, there is the more generalquestion, applicable to both derivative and assignment indexing, of how well, ever, can acondensed representation serve the purposes of specific subject content recapture? In thestrict sense, only by the elimination of truly redundant information. But even this is arelative matter. What i s redundant for an author may not be so for several different potential users of the reports or papers that this author writes. What is redundant for one
user i s not necessarily so for others.
The further problem for machine techniques is therefore: how selection rules canbe provided that will replicate a given human pattern of selectivity, or, alternatively, howselection rules can be established and defined that will produce an equivalent and comparable result - that i s , one which typical u s e r s would agree is as pertinent to their queryanswer relevance decisions as any available alternative.
Certainly the problem of appropriate selection is at the heart of the matter. This i sa crucial question, even if we sort out and can specify the different u s e s , for a particularcollection, a particular clientele, at a particular time, that automatically generated condensed document representations may have. Wyllys, in appraising automatic abstracting
efforts, considers that the goal should be to provide extracts which will serve a searchtool function - - that i s , they will furnish the searcher with enough information about the
document content so that he may decide whether it is probably pertinent to his then interests
or not and hence decide whether or not to read the document in full. By contrast, he saysof the "content-revelatory function" that an abstract should: "furnish the reader withenough information about the related document so that in most c a s e s he will not need toread it itself. " 2 /\J
2/See for example, Doyle, 1963 [ 162], with respect to lack of capacity for genericsearching as one of the major disadvantages of natural text search systems.
Wyllys, 1963 [653], p. 6.
174

Let us r e c a l l the objections to the use of the t e r m s "auto-encoding" (or "auto-indexing" or "auto-abstracting") because of the possible connotation of self-encoding, e t c . . \JThis is an objection based upon avoiding ambiguous or misleading terminology, but it also
points to an objection as to the principle involved--that is, of treating the document itself,
in its own right, as a self-sufficient, self-contained, universe of discourse, and of a s s u m ing that some type of summation-condensation over a number of different and individuallyderived r e p r e s e n t a t i o n s of the separate documents in a collection can provide an effective
selection-retrieval guidance s y s t e m to the contents of various specific documents in thatcollection. Even when the actual operations a r e to be abetted by synonym reduction andnormalization p r o c e d u r e s (whether at the indexing or s e a r c h negotiation stage, or both),t h e r e is a significant difference between this endogenous hypothesis and its exogenous
alternative: that the basis for automatic indexing be the consensus of the collection, or ofa sample of the collection, or of p r i o r indexing.
Assignment indexing, especially in the sense that concept-indexing is the goal, may
be subjectively preferable to derivative indexing not only because it involves exogenousemphases but because it tends to delimit, centralize, and standardize the a c c e s s points
available to the u s e r in his s e a r c h - r e t r i e v a l operations. However, in t e r m s of the humanindexing situation, it involves all the traditional difficulties of indexing - which in turninvoke the problems of evaluating indexing s y s t e m s :"Justification for any indexing technique m u s t ultimately be based on successfulr e t r i e v a l . Success can only be evaluated in t e r m s of a closed system; that is, asystem wherein sufficient knowledge is available of the entire contents of them a t e r i a l s , so that an evaluation can be made of various techniques as to theirr e t r i e v a l effectiveness. The various s y s t e m s . . . cannot really be weighed except
on the b a s i s of a test comparing one against the other. This has not been done inany place. " Zj
Nevertheless, there a r e a variety of r e a s o n s for accepting even tl^a relatively crude
derivative indexing products as p r a c t i c a l tools today, for seeking machine-usable rules
for the improvement of these products, and for continuing r e s e a r c h efforts in automatic
assignment indexing and automatic classification. There a r e , first and foremost, thec a s e s where conventional indexes a r e inadequate or non-existent. Thus Wyllys c l a i m s :"It is well-known that the c u r r e n t methods of producing, through human efforts,
condensed r e p r e s e n t a t i o n s of documents a r e already hopelessly inadequate to copewith the p r e s e n t volume of scientific and technical l i t e r a t u r e . Many papers a r enever indexed or a b s t r a c t e d at all, and even in the c a s e s of those that a r e indexedor abstracted, the indexes and abstracts do not become available until six months
to two years after the publication of the paper. " 3/Again, with r e s p e c t to automatic derivative indexing, especially KWIC indexes basedon titles alone, there can be no question as to the evaluation c r i t e r i o n of t i m e l i n e s s . Thes u c c e s s of this aspect is widely acknowledged by u s e r s , s y s t e m s planners, and interestedo b s e r v e r s . On the other hand, t h e r e is very little reported evidence available on which1/

See p . 3 of this r e p o r t .

2/Black, 1963 [ 6 4 ] , p. 16.

3/

Wyllys, 1961 [650], p. 6.

175any objective measure of comparative cost-benefit ratios may be obtained.
but without supporting data, that:

Black reports,"It has been estimated that the efficiency of KWIC indexing is about 76 per cent c o m pared with about 82 per cent for conventional indexing or classification. " \JWhite and Walsh report that:"From the limited experiment on methods of indexing the 1962 issues of the Abstractsof Computer Literature, the permuted title indexing retrieved only 52 percent of theinformation. This low percentage may be attributed to the changing and not yet
uniformly standardized terminology existing in computer technology. ,f 2 /KWIC indexes, because of their very currency, are fulfilling significant maintainingawareness needs today. Improved titling practice, enforced by editorial rigor or contractual requirements or both, can improve their usefulness. They fill gaps in the benchscientist's or engineer's ability to know about what might be of interest to him, either
because the material i s not otherwise covered in normal secondary publication ( e . g . , conferences and proceedings of symposia, internal technical reports not produced t>n Government contracts and therefore not announced and indexed by the cognizant agencies, and thelike) or because the sheer bulk of the product of indexing-abstracting services in his fi^ld
prevents his effective use of these services unless more specific a c c e s s points are provided. The claim that "something i s better than nothing" is not without merit, 3 / evenwith all the problems of non-resolution of synonymity, homography, topical scatter, longblocks of entries under the sorting term, the even more significant disadvantages of authorbias towards his principle topic, the author's choice both of emphasis and terminology,
and the like. Williams, considering word-with-context indexes, whether limited to title
only or to titles with readily available augmentation, makes the following comments:"Limitations and other troublesome features of the method have been obvious, but
perhaps over obvious, in the light of its growing acceptance and of the basic validityof permitting a document to speak for itself, even in a much abstracted recapitulation.
Wherever there are large and growing problems in maintaining publication schedules
for established subject indexes, or wherever pressing needs develop for more f r e quent indexes, for rapid, low-cost cumulation, or for indexes in areas where suitable indexing services are wanting, there no apology is needed for proposing that
this method be considered and tried, as a precursor to 'better' indexing, if not as a
substitute. Its use may be of interest also in l e s s troubled circumstances, in its
own right, and because of common elements involved in its production and the provision of other wanted products and functions (catalog records, current-awareness,
lists, etc). " 4 /Returning to the question of whether automatic indexing i s possible, it can be seen
that, at least in the derivative indexing sense, it is not only possible but can be practicallyuseful. To*dismiss the evidence of automatic derivative indexing operations that are in
production today by rigorous definition of what indexing is in effect anticipates both our\JBlack, 1962 [ 6 5 ] , p. 318.

2/

White and Walsh, 1963 [639], p. 346.

3/

See Veilleux, 1962 [ 6 2 4 ] , p. 81: "Accepting the premise that partial control of information satisfies more consumers than absence of control, perfection was tradedfor currency. "4/

T. M. Williams, private communication, dated January 4, 1962.
176

third and fourth questions: whether machine-generated indexes are as good or better thanthe products of human operations and of how we can m e a s u r e and a p p r a i s e the adequacy of
any indexing s y s t e m whatever. Here are encountered the " c o r e " problems of meaning incommunication, of information loss in any reductive transformation of actual m e s s a g e s or
documents, of relevance of particular m e s s a g e s to particular queries and to particular
human needs, of judgments of relevance.
Because of these underlying yet overriding questions, the state-of-the-art in theevaluation of indexing s y s t e m s is in fact far more primitive than that of automatic indexing
itself. An easy, and an early, solution is not likely. Therefore, today, in appraisingmachine potentials for assignment indexing we are faced with what is in effect a single
c r i t e r i o n : namely, will a given group of human evaluators, whatever their standards andrequirements, a g r e e as much with the products of an automatic indexing procedure, o t h e r wise competitive on a cost-benefit ratio with human indexing of the same material, as they
do amongst themselves?
Within the limits of small, specially selected samples of document or m e s s a g e c o l lections, it is possible to demonstrate that:
(1)
(2)

(3)

(4)

Replication of the products of at least some existing s y s t e m s , within theconsistency levels observed for these systems, can be achieved.
Retrieval effectiveness with respect to relevant items indexed by automatic assignment p r o c e d u r e s can be at least as good as, and may besuperior to, that obtained from run-of-the-mill manual indexing of thesame i t e m s .
Costs of indexing can be held at or below the costs of equivalent manualindexing, provided both that the input material r e q u i r e d is already inmachine-usable form, or can be held to an average of, say, 100 words orl e s s , and that the clue-word lists, association factors, or probabilistic
calculations can be accommodated within internal memory.
Significant gains in time r e q u i r e d to generate an index or to index or r e index a collection can be achieved.

Some degree of theoretical s u c c e s s in assignment indexing by machine can thus certainly
be claimed. Moreover, many of the t e s t r e s u l t s r e p o r t e d do clearly indicate a quality ofindexing, for a given collection at a given level of specificity of indexing, at least c o m parable to that which is typically and routinely achieved by people in a p r a c t i c a l indexing
situation. No more should be asked of the automatic techniques unless better human indexing can be specified as being equally feasible, timely, and practical. Further, no m o r eshould be asked of automatic techniques in t e r m s of the evaluation of their potentialities,
than is now asked of the manually-pre pared a l t e r n a t i v e s . \JData with respect to comparison of the r e s u l t s of automatic assignment indexing
techniques to either a priori or a posteriori human judgment have been mentioned previously in this r e p o r t in t e r m s of actual t e s t r e s u l t s reported, and the m o s t significant of theser e p o r t e d data are summarized in Table 2. Zj Typically, however, these data reflect, invarying degrees, so small a sample of test c a s e s , of u s e r preferences, a n d / o r of special
purpose and interest, that no g e n e r a l extropolation is r e a s o n a b l e . Moreover, the g e n e r a lquestions of the "core" problems of evaluation in g e n e r a l again r e a r their own ugly heads.
\J

Compare, for example, Kennedy, 1962 [311] and Needham, 1963 [ 4 3 3 ] .

2/

See pp. 101-103 of this r e p o r t .
177

Thus, Borko and Bernick point out:n

Up to this point we have used human classification as our criterion for the accuracyof automatic document classification* Against this criterion we have been able topredict with approximately 55% accuracy, and no more. Is this because out techniques of automatic classification are not very good, or is it because our criterionof human classification is not very reliable? There is some evidence to indicate thatthe reliability of human indexers is not very high. The reliability of classifying
technical reports needs investigating and, perhaps even more basically, the reasonsfor using human classification as a criterion at all. " 1/In general, the results of automatic index-term assignment procedures appear to run
in the area of 45-75 percent agreement with prior human indexing, ljand this in turn is wellwithin range of, and often superior to, estimates of human inter-indexer consistency basedon actual observations and t e s t s . There can be little or no doubt that the results of automatic assignment indexing experiments to date, (if extrapolation from the small and often
highly specialized samples so far used in actual tests is in fact warranted 3/) do suggestthat an indexing quality generally comparable to that achievable by run-of-the-mill manual
operations, at comparable costs and with increased timeliness, can be achieved by machine.
The question which remains is simply that of practicality, today. Extrapolationfrom small samples is highly dangerous, as is well noted even by enthusiastis for machinetechniques. The fact that for at least some systems, the limitations on number of cluewords that can be handled (due in part to computational requirements, matrix manipulations,and the like) are such that, even in an experimental situation, certain "tests" are excluded
from the result statistics, because the items contained an insufficient number of clues, isa serious indictment of reasonable extrapolations for these techniques today. Most tests
so far reported have involved not only a highly specialized "sample" library or collection,
but a severe limitation on the total number of "descriptors", subject headings, or c l a s s i fication categories to be assigned. Maron used 32, Borko 21, Williams 20, SADSACT 70,
Swanson 24. How would any of these approaches fare, given several hundred, much l e s s

1/

Borko and Bernick, 1963 [ 7 8 ] , pp. 31-32.

2/

See Table 2.

3/

This i s an important, perhaps crucial* caveat. See, for example, Goldwyn, 1963
[ 2 3 3 ] , p. 321: "In the micro-experiments of many of those who would apply statistical techniques . • . The document collection consists of 0-100 units. Results basedon the manipulation, real or imagined, of such a collection can be valid for it, yetbecome shaky or even nonapplicable to larger collections"; Perry 1958 [471], p. 415:
"A degree of selectivity quite acceptable for files of moderate size may prove quite
inadequate in dealing with large files. This fact often makes it necessary to exert
unusual care and considerable reserve in evaluating the results of small-scale tests
and demonstrations which may tend to cause the mass effects of large files to beunderestimated or overlooked completely"; Swanson, 1962 [ 5 8 6 ] , p. 288: "Theextent to which semantic characteristics of natural language are susceptible to being
generalized from small sample data is deceptive. "178several thousand, possible indexing or class ificatory labels? \J
The use of very brief short articles, or of abstracts, as the members of experimental corpora for investigations of automatic assignment indexing techniques presuming theprocessing of full text, either for indexing purposes or for subsequent "indexing-at-time of search", is seriously misleading. First, it is not truly representative of discursive
text, either in vocabulary-syntax, or stylistic variations involving synonymity, tropes,
elisions, dangling referents, and inumerable other meaning-implications, not explicitly
stated*Secondly, as any author of a technical paper, for which he must provide an abstract,knows all too well, he must concentrate in the abstract on a telegraphic emphasis toward
his principal topic and the points he wishes to make. He must omit most qualifying, specifying, and suggestive-of-other-leads-or-applications words and phrases, which he will infact develop in the text itself* For this reason, even supposing that the author himself isunusually well-aware of the multiple points of a c c e s s that many different potential u s e r s
might desire, the required brevity of the abstract form almost necessarily demands terse,
shorthand-type statements that can only increase the problems of "technese", of homography, and of single-subject representation.
Granted, in either manual or machine-serviceable systems today, the currentawareness scanning need is largely met by indexing based solely or primarily on title only,or title-plus-abstract. But is this good enough for search and retrieval? If and only if it
i s , then automatic indexing potentialities available today should be considered for both
purposes.
Our final question as to whether automatic indexing can be accomplished by statistical means alone or must involve syntactic, semantic and pragmatic considerations is notentirely answerable. In terms of achieving comparable quality with many manually p r e pared indexes available today, statistical means alone do appear promising. But is theachievement of just this level (even if accompanied by significant gains in timeliness,coverage, and economy) really good enough? There are a number of serious investigators

For example, Black predicts (1963) [64 ] , P* 19) that for most systems an adequate
vocabulary or thesaurus will comprise some twenty thousand terms. See also
Arthur D. Little, Inc., 1963 [ 2 3 ] , p. 65: "The enormous number of computations
required increases very rapidly with the number of indexing terms. Existing c o m puters, operating serially, do not appear to be capable of handling the problem
economically for collections with 9000 or more terms even if the simplest associativetechniques are employed"; Williams, 1963 |_642], p. 162: "One of the practicalproblems. . . is in the inversion of large matrices. In certain methods the order of thematrix will equal the number of different word types in the population, which isusually in the thousands. "179

convinced that it is not, \J and for this reason, research efforts are being directed toward
these other considerations.
On-going research and development work - whether in modified derivative indexing
approaching a "concept-indexing 11 level; in automatic assignment indexing techniques assuch; in automatic classification or categorization procedures, or in potentially related
efforts directed toward automatic abstracting, automatic content analysis, and otheraspects of linguistic data processing - is both reasonably extensive and quite promising.
Most of the investigators who are seriously active in the field report their current objectives and recent accomplishments regularly to the National Science Foundation for publication in the s e r i e s "Current Research and Development Efforts in Scientific Documentation. " In the most recent issue, unfortunately current only as of November, 1962, there
are not l e s s than 25 reports of KWIC and similar title-permuted derivative indexing
methods generated or proposed-to-be-generated by machine, there are several instancesof investigations into various possibilities of modified derivative indexing to be a c c o m plished by machine, and there are five to ten reports of active experimentation with various
automatic assignment indexing schemes. These efforts and even more recently organizedprojects point in the hopeful direction that "KWIC indexes should be merely a sample ofthings to come". Zj
Assignment indexing techniques so far investigated can be, as we have seen, of two
types which are quite distinct in terms of the principles involved. The first, which can be
the more readily mechanized, involves the use of thesaurus-type lookup procedures c o v e r ing the definable rules of "scope notes", "authority lists", or "see also" reference practice. The second type of assignment indexing, however, depends upon decision-making as
to the propriety of assigning a particular indexing term to a particular document withreference to assignments to the collection as a whole (or a sample thereof). This latter
type of assignment may be in terms of a priori categorizations of separable subsets of thecollection.
Alternatively, the bases for the latter type assignment-indexing procedures may bederived from a posteriori determinations of the suitable subsets as in the factor analysis
experiments of Borko, the latent class analysis approach of Baker, and the clusteringclumping approaches to automatic classification of Needham and others. It is to be notedin particular that Needham thinks an automatically generated categorization is preferable
precisely because of lack of knowledge as to the exact attributes defining a class in
1/

See, for example, Climenson et al, 1962 [133], p. 178: "The statistical approach
attempts to use no more than the occurrences of word spellings and their relative
distances in the document environment . . . [and] cannot provide the discriminationnecessary for most indexing and abstracting applications"; Doyle, 1963 [ 162], p. 3:"Automatic indexing and abstracting, as currently conceived, do not require any sortof dictionary or other semantic reference, but only counting, comparing, and sortingoperations well known in numerical data processing. But success in applying suchrules on a purely automatic basis can't help but be limited"; Borko, 1962 [ 7 5 ] , p. 5:"Although difficult, identification [of different meanings carried by the same word,of the same meaning carried by different words] must be accomplished before theautomatic categorization of document content can be truly effective. For the mostpart statistical methods, and even syntactic analysis, are inadequate for the job. A
technique of textual analysis based upon the semantic properties of language is needed"; Grosch, 1959 [244], p. 20: "We need semantic methods . . . that will look forthe intersection of redundant descriptors, each of which is at least slightly erroneous. "2/

Doyle, 1962 [163], p. 381.
180

existing classification schemes. However, in the related field of pattern recognition Uhr
and Vossler have shown promising results both for criterial feature analysis (a priori
assumption as to attributes or properties governing membership in specified classes) andfor randomly generated discrimination operators which, applied in a recursive manner,are increasingly adaptive to the detection of class-mambership (Uhr and Vossler, 1961[615]).
One particular way of looking at the problems of automatic indexing results, in
effect, in placing these problems within the broader field of pattern perception and patternrecognition. We suggest that this is in fact a particularly fruitful approach. Certainly
there is a wide area of potential commonality, and many promising leads for further r e search in automatic categorization can be found in the general pattern recognition literature, especially in work on randomly generated operators and on the problems of determination of membership in c l a s s e s . \J Conversely, automatic classification techniques
originally conceived as applicable to the handling of documentary information have in fact
been applied quite successfully to at least one case of groupings of physical objects on thebases of machine-detectable common properties.
The question of determination of membership-in-classes is basic to the problems ofautomatic classification and categorization. Thus the techniques for discriminating thestatistically significant associations between "properties" of objects or items that are to
be grouped into c l a s s e s or categories, even when such "properties 11 are not known inadvance and have no a priori identification, point to an increasing and promising convergence of research in pattern recognition, propaganda analysis and psycholinguisties, mathematics and statistics, studies of linear threshold devices, and the like, as well as in the
linguistic data processing field as such.
It is true that such synthesized "classes 1 1 may have no convenient "names 11 or
linguistic interpretations which make much sense to the individual human searcher or user.
Nevertheless, what is suggested is that a radical departure from conventional habits ofliterature search and retrieval may be desirable from the standpoint of effective use ofmachine potentialities. This might mean that, ab initio, the customer would pose to thesystem a search query request not couched in his notion of words or terms actually usedin the system, but either (a) an outline or statement of his own research proposal andplan of attack or (b) an indication of one or several items that he has already decided are
pertinent to his interests, with a request for "more like these".
An equally radical departure from conventional present habits and thinking is already
implicit in Needham's suggestion of an automatically derived classification system andmanual assignments thereto. Zj It would attack present-day machine capacity and p r o c e s sing time limitations such that property and class or category associations must be held tosomething l e s s than 1, 000 x 1, 000, unless prohibitive processing costs are to be incurred.
This approach would assume a one-time large-scale building of vocabulary and term or
category associations and derivation of assignment algorithms, and the printing out of theresults in multiple copies for use by low-level clerical personnel carrying out, indeed,1'machine -like'' inde xing.
A final promising approach to the future prospects for fully automatic indexing andcategorization is the perseverance in research and development efforts in advance of the\J

See, for example, Sebesyten, 1961 [ 5 3 9 ] , 1962 [ 5 3 8 ] .

2/Needham, 1963 [432], p. 1.

181

advent of versatile character readers and inexpensive, very large capacity, rapid directa c c e s s m e m o r i e s . These efforts will include not only further systematic exploration ofsyntactic, semantic and pragmatic considerations in linguistic data processing, but also
further attacks on the problems of language and meaning themselves. Thus, we may conclude with Maron thate "automatic indexing represents the opening wedge in a general attackat not only the problems of identification search and retrieval, but also the problem of
automatically transforming information on the basis of its content. " \JIf we are to attempt to solve this problem, as indeed we should, must we not lookforward to the possibilities of rapid up-dating, thesaurus growth and revision, and quick
and economical re-indexings of entire collections that only machine-processing capabilities
can promise today?

ACKNOWLEDGEMENTS
The contributions of Miss Josephine L. Walkowicz and her staff in the preparationand checking of items for the bibliography, and of Mrs. Betty J. Anderson, Mrs. Helen B.
Grantham, and Mrs. Anna K. Smilow in the typing and editing of the manuscript are
gratefully acknowledged. The courtesy of Miss ThyllisWilliams, Mr. Joseph Becker,
Mr. Herbert Ohlman, and the late Hans Peter Luhn in making available unpublishedmaterials is also gratefully acknowledged.

1/
Maron, 1961, [395 ], p. 240.
Bernick, 1962 [ 77 \ p. 3

See also Salton, 1962 [518], p. 234 and Borko and182