The proposed "Multilindex" system is also based on micro-thesauri or small
vocabularies designed, by human analysis, for clue-indications to a relatively narrow
subject field, together with potential syntactic-semantic role indications built into thedictionary, again by extensive human analysis, following the approaches previously taken
by A. L. (Lukjanow) Loewenthal in her suggestions for solutions to problems of mechanized translation. An unpublished proposal-type brochure describing the system wasavailable as of December 1 9 6 3 . U As of that date, also, demonstration printouts wereavailable from an IBM 1401 Fortran program, illustrating an index compiled fromabstract-text input and a 1, 200-word dictionary for documents in the field of space antenna tracking radar. ?J A repetoire of 350 ''concepts 11 or indexing terms was involved,
with an average of 10 assigned to 22 test documents, many of these assigned terms being
identical to words occurring in either the title or the text of the abstract of the item.
Slamecka and Zunde have investigated the extent to which the "notations-of-content 11
in the system developed by Documentation, Inc. for NASA's STAR might be derived bymachine techniques from the text of the abstracts with enough normalization-standardization via inclusion dictionary lookup to qualify as an assignment indexing technique.
These workers claim:"This preliminary investigation indicates the possibility of using the computer
to index documents adequately for machine retrieval by matching their abstracts
against an authoritative subject-heading authority . . . The inconsistency inherent in
human indexing can be eliminated as the number of terms derived from any one
abstract will always be the same. The abstract and its automatically derived set ofindex terms will always be equivalent. . .
ti3/

A final example of other approaches to automatic assignment indexing research, notyet reported in the open literature, is an NIH sponsored project at Goodyear Aerospace, in
cooperation with the Universities of Minnesota and Rochester and Western Reserve
University, looking toward an automatic classification procedure based on word coocurrences for a set consiting of 100 four-to-five page documents in the field of diabetes
literature. Programs for statistical analyses of the full text of these documents, all ofwhich have previously been processed for the manual W. R. U. "telegraphic" abstracting
system, are being developed. 1 /5.

AUTOMATIC CLASSIFICATION AND CATEGORIZATIONIn all the experimental work, to date, that has been directed toward the use ofcomputers and other machine-like techniques for the automatic indexing of documents, ay"Description of MULTILINDEX. A mechanized system for indexing documents,
storing information, retrieving information", P. S. Shane, Dec. 4, 1963, Information Systems, Inc. , 7720 Wisconsin Avenue, Bethesda, Maryland.

yPrivate communications, A. L. Loewenthal and P. S. Shane, Dec. 11, 1963.

ySlamecka and Zunde, 1963, [ 5 6 l ] , pp. 139-140.

±1
E. Tuttle, private communication, Oct. 30, 1963.

106

dichotomy can be observed. There i s , on the one hand, a spate of examples of automatic
derivative indexing where words used by the author himself or by human analysis are
sorted and arranged, by machine, to provide index listings, announcement bulletins, andcurrent awareness distribution notiij&fr. There are also, on the other hand, at least a
few instances of investigations where the machine assigns category labels, indexing
terms, or "heads11 and "headings 1 ' from a classification schedule, to new items.
In general, as Needham — points out, proposed automatic assignment indexing procedures can be investigated with reference to a previously existing index term vocabulary,an existing classification system or schedule, or to specially designed vocabularies andsubject heading l i s t s . On the other hand, if it is not known how well existing systems doin fact characterize documents and if it is not known whether all pertinent properties ofthe documents have been consistently identified, then it may be preferable to develop
methods for assigning documents to the appropriate class in a classification system which
is itself set up automatically. 2/ Needham also suggests still a third possibility: that ofsetting up automatically a classification within which the subsequent classifying of documents is done by hand.
The principal experimental results, to date, of attempts to achieve automaticclassification of documentary items, especially in the sense of machine-generated
groupings or categorizations of such items, have been those of applying techniques of"clumping", zJ factor analysis, and "latent class analysis". ZJ We shall briefly consider
below some typical investigations into automatic classification or categorization procedures that have already had, or may have, applicability in automatic index ing techniques.
In the late 1950's, Tanimoto undertook theoretical studies of mathematical
approaches to problems of classification and prediction with special reference to matrix
manipulations of sets of attributes of items to be classified. 2 / He also investigatedU
Needham, 1963, [ 4 3 2 ] , p. 1.

I)uIbid, p. 1-2: "If we are to assign a document to a class automatically, we musthave a) a list of facts about the c l a s s e s which will make ascription possible:b) an algorithm, usually some sort of matching algorithm, to tell us which classbest suits a document. Given a classification like the U. D. C. , it is not at all
obvious that a) and b) exist, or even, if they can be found, a) and b) imply a degreeof uniformity about the classification which may just not be there. "That i s , the clustering of objects that are in some sense similar because theyshare certain attributes or properties, even if* and especially when, the identityof cluster-producing common properties is not known in advance.

4/

Compare Doyle, 1963 [ l 6 2 ] , p. 13; "There are other statistical techniques besides
factor analysis whose output is document clusters, such as latent class analysis
and clump theory, and there is a surprising increase in research in this kind ofanalysis just within the last two years. "5/

Tanimoto,
1958 [ 5 9 3 ] , 1961 [ 5 9 4 ] . See also Borko, 1963 [ 7 6 ] , pp. 4-5: "In
1958, Tanimoto published a theoretical paper on the applications of mathematics tothe problems of classification and prediction. Specifically, he pointed out how the
problems of classification can be formulated in terms of sets of attributes andmanipulated as matrix functions. "107theoretical aspects of automatic indexing and sentence extraction involving co-occurrencesof words. While Tanimoto's studies with respect to linguistic information processing forclassification purposes have apparently been limited to the theoretical considerations,similar concepts of probabilistic, computational, and matrix manipulative operations toderive and use coefficients of correlation of associations between such attributes as words
occurring in text or the index terms assigned to documents are involved in the factor
analysis and theory of clumps techniques as applied in actual experiments in documentary
classification.
5. 1

Factor Analysis

The factor analysis technique which seeks to derive from word associations in
representative documents an automatically generated classification schedule for use inactual indexing experiments has previously been mentioned, i / Reasons suggested for its
use in research at SDC have been reported as follows:"The development of automatic procedures for purposes of classification and abstracting requires the identification and specification of attributes of words or
passages so that the relevancy of topics or content can be determined. Automatic procedures to detect such attributes may be based on a number ofcharacteristics of the text: word frequencies, syntactical information, semantic
information and pragmatic contextual clues. Currently, word frequency information can be generated and manipulated by automatic procedures, whereas theother attributes are not as readily handled this way. However, a correlation
matrix of content words becomes very unwieldy because of its size and the complexity of relationships* For this reason, factor analysis is used to identify
clusters of relationships. Current work concentrates primarily on determining
the usefulness of factors identified in this way as classification and indexing
schemes. .. 2/
As noted above, Borko and Bernick (1961 [ 7 3 ] , 1962 [ 7 7 ] , 1963 [ 7 8 ] ) have applied
this technique to abstracts drawn from psychological literature and to the same computer
literature abstracts as had been used by Maron, (1961 [ 3 9 5 ] ) . This technique had also
been investigated in the studies looking toward information retrieval classification andgrouping undertaken at the Cambridge Language Research Unit from about 1957 onward.
However, certain apparent limitations of the factor analysis approach led Parker-Rhodes
and Needham to the alternative of the "theory of clumps" (I960 [ 4 6 5 ] , 19611435, 464]).
Parker-Rhodes gives the rationale, and some of the distinctions between the two techniques, as follows:"It has been assumed that statistical methods could be applied to the data in sucha way as to reveal any objectively existing c l a s s e s which may be there. The generalyPp. 94-97 of this report.
System Development Corporation, 1962 [ 5 9 0 ] , p. 15.

108

name for the techniques evolved in this way is factor analysis. Insofar as itis practically applicable this technique has worked well enough; but. . . it has two
limitations (a) that some classification problems are outside its scope, and(b) that it is not susceptible (at least as hitherto conceived) of adaptation computationally to the study of really large u n i v e r s e s * . . I f 1/". . . The procedure of factor analysis first finds certain clumps, but then, as
output, it gives us vectors relating the descriptors of the universe to theclumps found. . .
"In most c a s e s , factor analysis is used (especially in psychology) to debug thedescriptor space; more conventionally put, to eliminate those tests (descriptors)
which have an equivocal membership in several factors (Clumps) in favor of
those which, having more definite allegiances, convey more information of thekind which the analysis suggests as valuable. It is thus only related to theclassification of the universe at one remove; the classification it suggests is a
simple categorical classification defined by the descriptors suggested as themost valuable. . .
nThe descriptive array of a universe is a table giving the applicability or
inapplicability of each descriptor to each element. To classify the elementsof the universe, we calculate for every pair of elements a similarity as afunction of the corresponding rows of the descriptive array, and then regard
the similarity matrix as a sufficient description of the universe.
In factor
analysis, on the contrary, we start with the matrix of correlations between
the descriptors, each being a function of a pair of columns of the descriptivearray. . . If Jj
Other investigators who have considered factor analysis techniques for possible
applications to automatic indexing, automatic categorization of items in a collection of
items, or search prescription renegotiation in a mechanized selection and retrieval
system include Stiles (1962 [ 5 7 3 ] ) , Doyle (1963 [ 1 6 2 ] ) , and Hammond (1962 [ 2 5 l ] ) .
Stiles, whose principal experimental results relate rather to the use pf statisticalassociations between terms manually assigned to documents for search prescription
formulation and renegotiation than to automatic indexing procedures as such, $J has also
considered both automatic indexing and automatic classification approaches. Specifically, he has made at least preliminary investigations of the factor analysis technique
independently developed for similar purposes by Borko. For a large collection of
105,000 items, the statistics of co-occurrence of indexing terms were in some cases notas precise as desired because the same terms were used in different senses for different
items in the collection.
1/

Note that Borko himself confirms this limitation as recently as November 1963,
in stating, of the CLRU work on clumps: "However, even now these techniques
have been applied to a 346x346 matrix which is beyong the capabilities of presently
available factor analysis programs. " (1963 [76] , p. 8).
Parker-Rhodes, 1961, [ 4 6 4 ] , pp. 3-6.
3/This principal concern is discussed below with reference to potentiallyrelated research, pp. 119-122 of this report.
109

The possibilities of using factor analysis to sort out the different meanings were
therefore explored. 1 / Using an IBM 704 program, the centroid method of factor analysis
was applied to a matrix of correlation coefficients of terms that had co-occurred significantly with the term n exposure". Three factors were derived, one generally relating to
the corrosive effects of exposure, another to "exposure" in the sense of photographic
exposure, and the third dealing with both exposure-to-weather and exposure-to-radiation.
Although the results were considered quite satisfactory, more extensive experimentation
and use did not appear feasible because of computer matrix manipulation limitations.
Doyle notes, in particular, that factor analysis might be used to give well-definedclusters separated one from another by clear boundaries rather than the l e s s precise
clusters found by most document grouping techniques. He emphasizes, however, that"its success in doing so of course, depends on the well-defined clusters actually being
present in the data". zJ He suggests that a combination of factor analysis and humanediting to select items most typical of statistically derived categories could be valuable
in such applications as the sorting of Congressional mail or the identification of trends
in political or military intelligence materials free from the personal biases of an analyst.
Hammond and his Datatrol associates who have worked on an application of the
Stiles association factor technique for search question negotiation to legal literature havealso considered the potentialities of factor analysis. Thus they report:" . . . The present association factor gives the relationship of one term to another.
A factor analysis study would allow us to determine the relationship of a single
term to a group of terms. From this we could learn how terms cluster whenrelated to the same concept. " ±!
5. 2

The Theory of Clumps

It is assumed, in the work on the theory of clumps, that we have a population of
objects or items among which at least some classes or groupings do objectively exist,
but that we do not have any bases for precisely determining class membership requirements. There may, therefore, be many possible ways of grouping and many possible
definitions of clumps. On the other hand, such diverse definitions must conform to theextent of some similarities of membership in the clumps that they define if in fact they
do define any of the existing c l a s s e s . Assuming further that we are given information
about properties ascribable to various members of the population, it is theorized that
useful clumps can be discovered by investigating similarity connections between pairsof items, such as the number of co-occurrences of specific properties. Thereafter, only
these similarity connections are considered, and the connection matrix is used as theb a s i s for trial partitions of the population into various possible subsets.

U
Stiles, 1962 [ 5 7 3 ] , pp. 10-12.
2/
Doyle, 1963 [ 1 6 2 ] , p. 12.

II
Hammond, et al, 1962 [ 2 5 l ] , p. 17.

110

In early work on clump definition, Kuhns of Ramo-Wooldridge }J proposed the useof a threshold value such that if a subset is a clump every pair of members in it has aconnection strength equal to or greater than the threshold value and no member of thesubset's complement has connections of more than threshold value to the members of thesubset. In the more extensive investigations carried out by Parker-Rhodes and Needham(I960 [ 4 6 5 ] , 1961 [434, 435, 464]), other clump definitions have been explored andspecifically that of the "GR-Clump". This is defined as a subset of the universe suchthat all its members have a positive (or zero) bias to the subset and all non-members
have a negative bias to it, where bias is defined as the excess (positive or negative) of thetotal connections of a member of the population to the members of the subset over its
total connections to the members of the subset's complement, following the convention
that the connection of the element to itself is taken as zero.
An iterative procedure for discovering GR-clumps can now be followed. This isbased on an arbitrary initial partition of the given universe of elements into a subset andits complement. Then, since each element has a bias toward both the subset and its
complement, differing only in sign, the biases of each element are computed. If the biasof a particular element is positive with respect to the subset, it is transferred to the subset if it is not already a member of it, and conversely if its bias is negative, it is transferred to the subset's complement if it is not already there. Each time a transfer ismade, the biases are recomputed and the process is repeated until for a complete scan of
all elements no further transfers can be made. The result is a GR-clump even though it
may have no members or may contain all the elements of the universe. In such case, afurther partition is made and the procedures are re-applied.
These GR-clump finding procedures have been applied to such diverse collectionsof items to be classified as archaeological artefacts and patients' symptoms as related
to specific disease diagnosis. In the latter case, groupings were obtained that corresponded satisfactorily to certain specific disease syndromes, but no group was foundcorresponding to Hodgkin's disease where a great variety of symptoms typically occur.
Needham comments: "I can scarcely conceive of a clump definition that would be likely
to group these patients; I am unsure whether this is a reflection on clump theory or on
Hodgkin's d i s e a s e . " 2/In applications more directly related to documentation, some investigations havebeen made of the use of co-occurrence coefficients of index terms assigned to documents
in order to form a connection matrix from which clumps were then derived (Needham,
1963 [ 4 3 l ] ) . These experiments covered 342 terms occurring more than once in theindex-term sets assigned to several hundred documents in the general subject field ofmachine translation. Computation of the matrix required 20 minutes of computer time
and the 40 clumps found took 6-8 minutes each to find. Needham reports on the results
as follows:

See Kuhns, 1959 [ 3 3 6 ] , and Needham, 1961 [ 4 3 5 ] , pp. 20-21.

I!
Needham, 1961 [ 4 3 5 ] , p. 46.

Ill

"Evaluation of the results was unexpectedly difficult. The acid test is presumably
the efficiency of the retrieval system embodying the grouping given by the program;but the efficiency of retrieval systems cannot be easily measured. An apparently
simpler test would be to see if the clumps were intuitively satisfactory, i. e. , weregroupings that a classifier in his right mind could have made. This also was unsatisfactory because the groups are mostly rather large, larger in fact thanclassifiers ordinarily make, and were thus very difficult to judge. The test
eventually adopted was to group the terms not distinguished by the clump classification, and look at these. Accordingly, for each term, a list of the clumps towhich it belongs was prepared, and groups of terms were found which had alltheir clumps in common. These groups were quite small (2-6 terms) and could
be studied easily. It turned out that some groups were ones of which a humanclassifier could have thought ( e . g . , words concerning suffix removal for machinetranslation came together) while others were quite justified by the documents concerned, but would never have been thought of a priori. For example, the group:"phrase marker, phoneme, Markov process, terminal language" was entirely
justified by the. . . contents of the library. It is groups of the latter kind that
represent a success for clump theory, for they function usefully in retrieval but
in no way form part of the structure of thought. . .which the human classifier's work
is likely to reflect. " 2 /Still another application of the theory of clumps may be of use in the construction of
thesauri (Sparck-Jones, 1962 [ 5 6 4 ] . Here the assumption is that rows of a correlation
matrix can be formed for words giving other words which are synonymous with respect tomeaning. The overlaps of the same word's occurrence in two or more rows can then be
used to find clumps which are presumed to represent conceptual groupings.
Applications of clump theory to problems of mechanized documentation are also
being investigated by Dale and Dale of the Linguistics Research Center, the University ofTexas. £' They have begun experimentation to derive clumps for the 90 clue words usedby Borko and the 260 source-item computer abstracts used by both Maron and Borko.
Preliminary results reported so far are principally limited to considerations of the a s s o ciative networks between terms as derived from the structure of the clumps discoveredby several clump definitions. Mention should also be made of the work of Meetham andVaswani at the National Physical Laboratory, Teddington, England, looking toward theuse of similar techniques for machine-generated index vocabularies, with preliminaryemphasis on testing them against a "library" consisting of the propositions of Euclid's
geometry. ±'yNeedham, 1963 [ 4 3 1 ] , p. 285-286.

I!
Dale and Dale, an unpublished report dated February 1964, [ 1 4 7 ] .
National Science Foundation's CR&D report No. 11, [ 4 3 0 ] , p. 137; and Meetham,1963[413].

112

5. 3

Latent Class Analysis

Like the e a r l i e r work of Tanimoto, the latent class analysis approach of Baker (1962[27])to problems of automatic information classification and r e t r i e v a l is at l e a s t to datetheoretical r a t h e r than experimental in nature, and so will be considered only briefly h e r e .
Baker claims that the latent class model developed v ln tliev field of the sociological sciences
for the determination of latent c l a s s e s among individuals responding l l yes , , or "no" toitems in a questionnaire would have attractive features for application to information
categorization and s e a r c h , because the model is based upon response p a t t e r n s that are
analogous to the presence or absence of clue words or phrases in documents and because
the analysis yields an ordering ratio that could serve a function s i m i l a r to the relevanceweightings suggested by Maron and Kuhns.
This ordering ratio is the probability that a given pattern of clue words will occur
in a document properly belonging to a particular latent c l a s s . The probabilities of thesame pattern being generated by a document properly belonging to other c l a s s e s a r e also
provided, giving an uncertainty which Baker thinks justifiable because a "document could
generate a given pattern of key words, yet not belong to the same area of interest as the
majority of documents possessing the same pattern of keywords". U It should be noted,
however, that the question of how to select appropriate clue words is begged __' and thatno computer programs are as yet available for carrying out latent class analyses. zJ
5. 4

Examples of Other Proposed Classificatory Techniques

There a r e certain other document classificatory techniques that have been proposed
and to some extent investigated experimentally. Trials of document clusterings basedon co-citingness, co-citedness, or bibliographic coupling as compared with subject content groupings have, as noted above, been conducted both by Kessler at the M.I. T.
Libraries and by Salton's group at Harvard. JU Consideration of Doyle's work on word
co-occurrence statistics has been deliberately deferred to a later section which covers
his general "association map" approach. Similarly, several other investigations will be
discussed in terms of potentially related research such as linguistic data processing.
Two particular examples of other suggested classificatory techniques for documentgrouping or classification a r e somewhat unusual, however. These are the methods proposed by Te Nuyl and by Lefkovitz (1963 [ 353]). Cleverdon and Mills comment on Te
Nuyl's method as follows:

Tl
Baker, 1962 [ 2 7 ] , p. 518.

yIbid, p. 517. Note also that the footnote states: "A referee of this paper has properly cautioned that the effectiveness of an information r e t r i e v a l system may be due
m o r e to the a p p r o p r i a t e n e s s of the key words than the s u b s e q u e n t p r o c e s s i n g . " Seealso Hillman, 1963 [ 2 7 2 ] , p. 323: "Baker's theory, however, is based on i n t e r relationships of key w o r d s , and thus constitutes an approach which is regarded withsome suspicion by F a r r a d a n e , who thinks that the r e a l problem concerns the interrelationships of the concepts which k e y w o r d s denote. "Baker, 1962 [ 2 7 ] , p. 516.

±1
See Kessler, 1963 [ 320]; Lesk, 1963 [ 3 5 6 , 357], and p . 3 0 of this report.
113

"Te Nuyl.. . u s e s , as quasi-descriptors, word-sets chosen from the Oxford English
Dictionary (e. g. , any word falling between A-Ah) and relies on the subsequentcorrelation of terms to make sense of his seemingly bizarre choice. M U
Lefkovitz is concerned with the so-called "automatic stratification" of a file inwhich both generic or associative relationships and exclusive partitioning is used tofacilitate search. He claims:n. . . The exclusive partitioning implies a separation of descriptors into groups
such that no two descriptors in a group co-occur in any given document descriptionof the file. This arrangement presents the dissociative properties of the file, or
forbidden combinations. When coupled with a superimposed display of the'inclusive 1 or associative properties of the file a unique classification of thedescriptors of this file results, which is based solely upon the association of thedescriptors themselves within the document descriptions and not upon an arbitrary
set of c l a s s e s constructed by professional indexers." 2/
The purpose is to assist the searcher by warning him that if he chooses more than
one descriptor from any one group as terms in his search request, there will be a null
response from this particular file. However, the particular application considered
involves a limited number of highly quantifiable or scalable "attribute-value" pairs, (forso the descriptors involved are defined), such as "Age-23", and "Hair-red". It is by
no means obvious that comparable exclusive partitionings could be achieved for literature
items or that the recomputations necessary as new items enter the file can be achieved
on a practical basis.
6.

OTHER POTENTIALLY RELATED RESEARCH

In this section we shall consider certain areas of potentially related research thatmay prove applicable to the improvement of automatic indexing techniques. First is thearea of thesaurus construction and use, which in turn is somewhat related to the development of statistical association techniques, especially for "indexing-at-time-of-search"and search renegotiations. Natural language text searching will also be briefly
considered, together with related research in the general area of linguistic dataprocessing.
6. 1

Thesaurus Construction, Use, and Up-Dating

The first area of potentially related research which promises improvements in
automatic indexing procedures i s that of thesaurus lookups by machine. There areseveral different possible definitions of the word "thesaurus" in the context of information storage, selection and retrieval systems. The first is that it is a prescriptive
indexing aid, or authority list, serving the function of normalizing the indiexing language,
primarily by the use of a single word form for words occurring in various inflections, by
the reduction of synonyms, and by the introduction of appropriate syndetic devices. The
second definition relates to the intended function for the provocation and suggestion to
the indexer or the searcher of additional terms and clues, and it follows the idea of word
groupings related to concepts as in a traditional thesaurus like Roget f s. The third

T7
Cleverdon and Mills, 1963 [ l 3 l ] , p. 8.
Lefkovitz, 1963 [ 3 5 3 ] , Preface, pp. VHI-IX.
114