"Te Nuyl.. . u s e s , as quasi-descriptors, word-sets chosen from the Oxford English
Dictionary (e. g. , any word falling between A-Ah) and relies on the subsequentcorrelation of terms to make sense of his seemingly bizarre choice. M U
Lefkovitz is concerned with the so-called "automatic stratification" of a file inwhich both generic or associative relationships and exclusive partitioning is used tofacilitate search. He claims:n. . . The exclusive partitioning implies a separation of descriptors into groups
such that no two descriptors in a group co-occur in any given document descriptionof the file. This arrangement presents the dissociative properties of the file, or
forbidden combinations. When coupled with a superimposed display of the'inclusive 1 or associative properties of the file a unique classification of thedescriptors of this file results, which is based solely upon the association of thedescriptors themselves within the document descriptions and not upon an arbitrary
set of c l a s s e s constructed by professional indexers." 2/
The purpose is to assist the searcher by warning him that if he chooses more than
one descriptor from any one group as terms in his search request, there will be a null
response from this particular file. However, the particular application considered
involves a limited number of highly quantifiable or scalable "attribute-value" pairs, (forso the descriptors involved are defined), such as "Age-23", and "Hair-red". It is by
no means obvious that comparable exclusive partitionings could be achieved for literature
items or that the recomputations necessary as new items enter the file can be achieved
on a practical basis.
6.

OTHER POTENTIALLY RELATED RESEARCH

In this section we shall consider certain areas of potentially related research thatmay prove applicable to the improvement of automatic indexing techniques. First is thearea of thesaurus construction and use, which in turn is somewhat related to the development of statistical association techniques, especially for "indexing-at-time-of-search"and search renegotiations. Natural language text searching will also be briefly
considered, together with related research in the general area of linguistic dataprocessing.
6. 1

Thesaurus Construction, Use, and Up-Dating

The first area of potentially related research which promises improvements in
automatic indexing procedures i s that of thesaurus lookups by machine. There areseveral different possible definitions of the word "thesaurus" in the context of information storage, selection and retrieval systems. The first is that it is a prescriptive
indexing aid, or authority list, serving the function of normalizing the indiexing language,
primarily by the use of a single word form for words occurring in various inflections, by
the reduction of synonyms, and by the introduction of appropriate syndetic devices. The
second definition relates to the intended function for the provocation and suggestion to
the indexer or the searcher of additional terms and clues, and it follows the idea of word
groupings related to concepts as in a traditional thesaurus like Roget f s. The third

T7
Cleverdon and Mills, 1963 [ l 3 l ] , p. 8.
Lefkovitz, 1963 [ 3 5 3 ] , Preface, pp. VHI-IX.
114

possible definition involves the special case of devices or techniques which display or use
prior associations and co-occurrences or words, indexing terms, and related documents
to provide a guide or suggestive indexing and search-prescription-formulation or
renegotiation aid.
The idea of a mechanized authority list, following the restrictive first definition,
has been proposed by a number of investigators ]J and has actually been used in computer
programs as discussed for example bySchultz and Shepherd (I960 [ 532]),Shepherd (1963[ 545]) and Artandi (1963 [ 2 0 ] ) . It is the second definition of thesaurus with which weshall be principally concerned. It i s , as we have said, close to the conventional idea ofsuch a thesaurus as Roget's. It is based on the hypothesis that patterns of co-occurrencesof words in a new item or in a search request can be compared with patterns of prior cooccurrences, as given by a thesaurus "head", in order to expand, clarify, or pin-point"meaning" and thus provide a more effective indication of the true subject content. The
third definition will be considered as falling within the more general scope of statisticalassociation techniques, although as Giuliano points out, "a retrieval system embodying
an automatic thesaurus thus qualifies as being Associative 1 ."The application of a thesaurus-like approach to indexing and searching problems isagain an area in which Luhn is one of the earliest proponents. In January 1953, heproposed a new method of recording and searching information in which a special dictionary would be compiled for use in broadening the terms of a search request and innormalizing word usage as between various indexers (recorders) and searchers. Although he did not then use the term "Thesaurus" as such, he said in part:"The process of broadening the concept involves the compilation of a dictionary
wherein key terms of desired broadness may be found to replace unduly specificterms, the latter being treated as synonyms of a higher order than ordinarilyuSee, for example, "Summary of discussions, Area 5," ICSI, 1959 [ 5 7 8 ] ,p. 1263: "Two further complications arise from a mechanical index.
Some articles might deserve as an indexing term a word not containedin the article. By an authority list, the product of the mechanized indexing
procedure might have such additional words added to it. Again, an article
might use a particular word but the vocabulary of the system might preferanother one. This also can be handled by a mechanized authority list."*Giuliano and Jones, 1962 [ 2 2 9 ] , p. 4.

115

considered. Translating criteria into these key terms is a process of normalization
which will eliminate many disagreements in the choice of specific terms amongst
recorders, amongst inquirers, and amongst the two groups, by merging the terms
at issue into a single key term. However, the dictionary does not classify or index
but maintains the idea of being fields. • . A specific term may appear under theheading of several key terms and if according to its application an overlapping ofconcepts exists then the term i s represented by the several key terms
i n v o l v e d . . . , f 1/
In subsequent papers, Luhn has developed related ideas of a "family of notions 11 anddictionaries of notional families 11 . 2 / In particular, he emphasizes that for automatic
indexing, by contrast with automatic abstracting, consideration should be given to thenormalization of variations in author-chosen terminology: "It will be necessary for amachine to resolve variation of word usage with the aid of a device the functions of which
resemble a dictionary at one level and of a thesaurus at another level of requirements. "2/
11

The first issue of the National Science Foundation's compendium of project statements, "Current Research and Development in Scientific Documentation", which appearedin July 1957 [ 4 3 0 ] reported several projects of interest in terms of thesaurus construction and use, ^/namely: (1) work by Luhn at IBM involving the establishment of athesaurus to facilitate encoding of items whose texts would be available in machine-us able
form, (2) work by Bernier and Heumann at Chemical Abstracts Service looking toward thedevelopment of a technical thesaurus, (1957 [ 57]), and (3) an approach to mechanizedtranslation proposing to use a mechanized thesaurus at the Cambridge Language Research
Unit. This latter project incorporated the ideas of Masterman and her associates fromabout 1956 on (Halliday 1956 [ 2 4 9 ] , Masterman, 1956 [ 4 0 3 ] ; Joyce and Needham, 1958
[ 305]), to apply the principle of checking co-occurrences of text words against thesaurus"heads" to which they belonged, in order to resolve homograph!c ambiguities and thusachieve more idiomatic translation by machine.
For the ICSI Conference in 1958, Masterman, Needham and Sparck-Jones prepareda paper discussing analogies between machine translation and information retrieval, andrecapitulated the arguments of Needham and Joyce for the use of a thesaurus in theformulation of search requests, as follows:"If a large number of terms are used to describe a document, the existence ofsynonyms is likely: in a system such as Uniterm no attempt is made to bracket
the synonyms, which means that a request will produce only the document describedyLuhn, 1953 [ 3 8 3 ] , p. 15.

I)
Luhn, 1 9 5 9 [ 3 7 l ] , p . 5 1 , 1959 [ 384] ; 1957 [ 385] , p.316.
Luhn, 1959 [ 3 8 4 ] , p. 12.

±1
National Science Foundation's CR&D Report No. 1, [ 4 3 0 ] , pp. 21, 6,4.

116

in identical terms and not in synonymous ones. If the existence of synonyms
is avoided, by using a small number of exclusive descriptors, the descriptionof a document in terms useful for retrieval is more difficult, also it is equallydifficult to relate a request to the description of documents. A further difficultyis that descriptions only list the main terms, and take no account of their relations
to one another. The C. L..R. TJ. experiments being carried out make use of athesaurus* a procedure through which it is hoped that these difficulties will be
avoided and that a request for a document although not using the same terms as
those in the document will produce that document and others dealing with thesame problem, but described in different, though synonymous, t e r m s . " A/In general, the use of a thesaurus to constrain variations in word or term usage(as in our first definition, a mechanized authority list), to reduce synonymity, to resolve
homographic ambiguity, to provoke and suggest additional terms or ideas to indexer andto searcher alike, is related to the improvement of automatic indexing procedures inprecisely the same sense that its use would be effective in any indexing system whatsoever. In another sense, however, the construction and use of the thesaurus is related
to linguistic data processing by machine in another way. Garvin suggests:
11.. .One may reasonably expect to arrive at a semantic classification of the contentbearing elements of a language which is inductively inferred from the study oftext, rather than superimposed from some viewpoint external to the structure of thelanguage. Such a classification can be expected to yield more reliable answers to
the problems of synonymy and content representation than the existing thesauri
and synonym l i s t s , which are based mainly on intuitively perceived similarities
without adequate empirical controls."
This is with respect to the recognition that the machine itself can be used to compileand construct the thesaurus. While Luhn in some of his 1957-8 proposals still considered
the compilation and organization of a thesaurus to be primarily a matter of human effort,
he nevertheless pointed out that: n The statistical material that may be required in the
manual compilation of dictionaries and thesauri may be derived from the original texts
in any desired form and degree of detail. " If De Grolier makes the complementarystatement that the Luhn techniques should "considerably facilitate" the preparation of
thesauri. 2 /Even more importantly, the computer can be used for periodic up-datings andrevisions. The work on the FASEB index-term normaliNation procedures involved earlyrecognition of the need to "educate the thesaurus" by examining print-outs when no
matches occurred and providing a continuous process of amendment. zJ Computermaintained statistics of word and term usages are closely related to possibilities for
1/
Masterman, Needham, and Sparck-Jones, 1958 [ 4 0 5 ] , p. 934-935; Needham andJoyce 1958 [ 3 0 5 ] .

ZJ
II
±1

Garvin, 1961 [ 2 2 4 ] , p. 138.
Luhn, 1959 [ 3 5 4 ] , p. 12.
De Grolier, 1962 [ 1 5 2 ] , p. 132.

5/
Shepherd, 1963 [ 5 4 5 ] , p. 392.
117

construction and revision of a mechanized thesaurus, as again Luhn has suggested. —'Schultz suggests that machine records should be maintained of what thesaurus terms are
actually used for indexing and searching, the frequencies of term usage, the cooccurrences, the number of items described by particular combinations of terms and thelike.i/
The potential combinations of natural text processing, automatic indexing, andthesaurus construction and updating are stressed in many current programs. For
example, Eldridge and Dennis discuss:"Indexing by machine from natural text in a fully automatic system, in which
statistical analysis of the words i s employed as a device for (a) building automatically a 'concept 1 thesaurus, (b) indexing incoming documents with reference
to the thesaurus, and (c) continuously revising the thesaurus to reflect new word
usages in currently incoming documents. 11
Similarly, Giuliano and Jones suggest that given a term-term statistical association
matrix, a transformation can be arrived at with a unit vector assigning value only toindex term Z that ranks every other index term according to degree of association with Z,then by listing the higher ranked terms for each term Z, "a 'thesaurus 1 listing can beobtained completely automatically. M \1
6. 2

Statistical Association Techniques

A special definition of the word "thesaurus" might, as we have noted, include thedevelopment of devices and techniques which either automatically or by man-machine interaction serve to suggest the amplification of a set of index terms. We shall briefly consider here both devices that visually display associations between words, terms, anddocuments \i and techniques for machine use of coefficients of correlation for prior c o occurrences in a collection of word-word, word-term, term-term, term-document, anddocument-document associations, the statistical association factor technique as firstdeveloped by Stiles.
1/Luhn, 1957 C 385], p. 316: "Provision should be made to register the number oftimes each word is looked up in the index and the number of times each family
number has been used for encoding* Such a record would be an indispensable
part of the system for making periodic adjustments based on the usage of words
or notions as mechanically established. 11uSchultz, 1962 [ 5 2 9 ] , p. 104.

II

Eldridge and Dennis, 1962 [ 183], p. 6.

±1

Giuliano and Jones, 1962 [ 2 2 9 ] , p. 12.

5/It should be noted that Tabledex, the Scan-Column Index, and similar tools provide to some extent a display of prior associations between index terms. (Seepp. 25-27 of this report.) Thug Cheydleur (1963 [ 115], p. 58) remarks: "Ledley. .
has focussed on inter-item concepts in designing his economical TABLEDEX
arrangement for displaying the connectivity of index terms and related file i t e m s . "118

6. 2. 1 Devices to Display Associations: EDIAC
The i n t e r e s t aroused among some documentalists by the provocative idea of a "Memex"
to record and display associations between ideas as proposed by Bush in 1945 ([93]) led to
specific attempts at Documentation, Inc. in the 1950 f s to develop a device which would
incorporate at least the associations between indexing terms assigned to documents andbetween documents with respect to their sharing of common indexing terms (1954 [157],
1956 [155, 156]). The first approach to this objective, as reported by Taube, was the ideaof a manual dictionary of terms arranged in alphabetical o r d e r , with a "page" reserved foreach and every indexing term used for any document in the collection. On each page would
be listed all other terms that had co-occurred with that term in the indexing of one or more
documents. Another idea was to display associations of terms used in a collection throughthe "superimposition of dedicated positions in a set of cards or plates. . . " !_/Subsequently, an actual device to demonstrate a system for display of term-term,
term-document, and document-document associations, was built under an Office of Naval
Research contract. 2/ The demonstration model contained a vocabulary of 250 terms which
had been^used in various combinations to index 100 reports. Interconnections in an electrical network provided the associational linkages. A display panel was provided withsymbol-indicators which could be lighted up to identify particular terms and particular
report numbers.
This EDIAC device (for Electronic Display of_Indexing Association and Content) was
intended for use both in guiding an indexer to either the extension or refinement of his
initial choice of indexing terms and in assisting the searcher. It was claimed that theoperation of such a device would be extremely simple. Thus:"For the index question the searcher selects any term in which he is interested
and applies a voltage. He is told instantly the number of the repqrts dealing withthat subject. Putting voltage in at any term also lights all other terms associatedwith the first term. . . " ^3/
A later analog device, ACORN, will be discussed below in connection with the workof Giuliano and associates, at Arthur D. Little, Inc.
6. 2. 2 Statistical Association Factors - Stiles
The name of H. Edmund Stiles, like those of Luhn, Baxendale, Maron, Swanson,
Edmundson and Wyllys, is generally associated with pioneering innovations in those areasof mechanized documentation which are directly related to the use of high-speed computercapabilities. While Stiles' work has been directed p r i m a r i l y to problems of searchp r e s c r i p t i o n formulation and renegotiation based on the r e s u l t s of p r e l i m i n a r y search, he
has specifically recognized that the use of statistical word association techniques insearching operations can provide a logical corollary to automatic indexing procedures.
Thus:

U
Taube et al, 1954 [599], p. 102.

u
It is described and illustrated in Taube et al, 1956 [599], p. 63 ff.
Documentation, Inc. 1956 [156], p. 7.

119

"Automatic indexing, based on the relative frequency of words used in a document,
produces a partial vocabulary of the content words used to express its subject.
Retrieval can then be accomplished by expanding the request vocabulary. . . Thismethod tends to overcome the deficiencies and inconsistencies inherent in the useof terms derived automatically from a text. " If
Conversely, Stiles also points out the possibility that the results of automatic derivativeindexing procedures, extracting indexing words from the documents directly, might provea more realistic or reliable basis for the development of his word co-occurrence correlation data than do the Uniterms assigned by human indexers. 2/ The work of Stiles has alsostressed the importance of two factors that may well be critical for the improvement of
automatic indexing techniques. These are, namely, the consensus of prior human indexing
and the consensus of subject coverage of a particular collection. Zj
In his experimental investigations, Stiles began with an existing collection of approximately 100, 000 items which had previously been indexed, over a period of time, with a
Uniterm indexing vocabulary consisting of about 15, 000 terms. The objective of theexperiments was to determine how, given a specific search request, a more effective "netto catch documents" 4 / could be generated and how the responding items might be ranked
in order of their probable relevance to the request.
The statistics of co-occurrence of terms used to index the same documents were firstobtained. A modified chi-square formula was then applied to determine relative frequencies of use of co-occurring terms. 5/ Patterns of term co-occurrence could then be
derived in the sense of term-profiles which show, for each term, the more significant of
its associational values of pairing with other terms in the collection. The actual procedurefor using these term-profiles in search prescription formulation and in document selection
involves several steps, generally as follows: hj—
Stiles, 1962 [573], pp. 12-13.

D
Stiles, 1961 [572], p. 205.

II
Stiles, 1962 [573], p. 6 and 1961 [572], pp. 273, 277.

±1
Stiles, 1961 [572], p. 192.
In general, we shall not be concerned with the precise mathematical formulations.
It is to be noted that in a recent report Giuliano and his colleagues have reviewed
a number of the various mathematical formulas proposed in the literature for thecomputation of word, term, and document associations, including those of ParkerRhodes and Needham, Maron and Kuhns, Stiles, Sal ton, Osgood, Bennett andSpiegel (Giuliano et al, 1963 [230], Appendix I).

kl
Stiles, 1961 [571], pp. 273-275.

120

1.

For each t e r m in the initial formulation of a search request, theappropriate t e r m - p r o f i l e is obtained, which gives weighted values
for those other t e r m s that had significantly c o - o c c u r r e d with it.

2.

The profiles of each t e r m in a m u l t i - t e r m request a r e comparedand those additional t e r m s common to all or a specified number ofthe profiles are selected and added to the initial s e t . ! /3.

The "first generation" t e r m s resulting from step 2 a r e next treatedas though they also w e r e request t e r m s , and steps 1 and 2 a r e repeated
for them.

4.

A selection is made from some reasonable proportion of the profiles
associated with the first generation t e r m s to produce the "second
generation" t e r m s . £ '5.

The expanded list of search terms is then compared with the index
t e r m s assigned to each document in the collection, and whenever amatch is found the weight of the request t e r m is assigned to thematching document term. These weights a r e then summed to provide
a n u m e r i c measure of probable document relevance to the original
request.

6.

Documents responding to the expanded request a r e printed out in theorder of document relevance s c o r e s .

Some experiments have been made using a computer p r o g r a m which accepts up
to 300 weighted t e r m s in an expanded request vocabulary. Representative results havebeen reported, in part, as follows:
11.. . We asked a qualified engineer to examine these documents and specify whichw e r e related to 'Thin F i l m s 1 and which w e r e not. . . This engineer was notfamiliar with our project. . . yet. . .we found a remarkably high correlation between
his evaluation and the document relevance n u m b e r s . . . We then checked to see how
the documents containing information on 'Thin F i l m ' had been indexed. We foundthat the first five documents on our l i s t had been indexed by both , Thin l and 'Film 1 .
Three m o r e documents had been indexed by ' F i l m ' alone, and other related t e r m s .
Two documents had not been indexed by either 'Thin' or ' F i l m ' , but only by a groupof related t e r m s , yet they contained information on 'Thin F i l m s ' and had a highdocument relevance number. By using association factors and a s e r i e s of s t a t i s t i cal steps, easily p r o g r a m m e d for a computer, we w e r e thus able to locate

1/

These are called "first generation terms" and tend to reflect only statistical a s s o ciations without including synonyms and n e a r - s y n o n y m s which, over the course oftime, have o c c u r r e d in the indexing vocabulary.
2/

Stiles, 1961 [ 5 7 l ] , p. 274: "Among these we find words closely related in meaning
to the request t e r m s . " An example given in Ref. [572], pp. 200-201, is thederivation of 'weathering, f 'fungicidal 1 , 'deterioration', and 'preservatives' as
second generation t e r m s when the initial request included the terms 'plastics','fungus 1 , 'coating', and ' t e s t s ' .
121documents relevant to a request even though the documents had not been indexedby the terms used in the request." j /In another case, which was analyzed in detail, a request profile of 26 terms that
had been intuitively weighted by the customer resulted in the machine listing of 246
presumably responsive documents. Of these, 81 documents were of primary interest
to the customer, and an additional 78 were of secondary interest to him. 2 /The statistical association technique as proposed by Stiles has also been investigated at the Datatrol Corporation, with particular reference to the field of legal
literature (Hammond et al, 1962 [ 2 5 l ] ) . About 350 documents in the field of Federal
public law were indexed in cooperation with George Washington University, using a
vocabulary of 680 index terms. A computer program was written for the IBM 7090 that
can accommodate a 1200 x 1200 matrix to calculate the Stiles 1 association factors. Trials
were made of various thresholds to determine which other terms were sufficiently high inassociation strength to a particular term to be selected for that term's profile.
Given the generation of the term profiles, a l e s s sophisticated computer such as
the 1401 can be used for the expansion of request terms and the actual conduct of searches.
Such a program was demonstrated at the Annual Meeting of the American Bar Association,
August 1962, with running of "live11 requests suggested by jurists and with what areclaimed to be "highly gratifying results". A point of interest relates to the question of
updating of term-profiles and other statistical association factor data. Hammond, et alreport:"The term profiles were generated a total of three times in the course of thepilot study* making it possible, to some extent, to a s s e s s the effect of
vocabulary growth. Judging from this limited experience, it appears that a bimonthly, or perhaps even quarterly, recompilation of term profiles should be
sufficient for a mature collection. " \i6. 2. 3 The Association Map-

Doyle and Related Work at SDC

The name of Doyle is again that of an early and prolific investigator and innovatorin the field of mechanized documentation and linguistic data processing. One of his
provocative suggestions is generally known, in his own terminology, as that of "semantic
road maps for literature searchers" or an "association map" technique. As a matter ofconvenience, we have chosen to consider this suggestion and a variety of related work

1/
Stiles, 1961 [ 5 7 7 ] , pp. 198-199.

I!
Stiles, 1962 [ 5 7 3 ] , p. 9.

II
Hammond et al, 1962 [ 2 5 1 ] , p. 6.

122

under the general heading of the association map technique, — although passing reference
has been made to some of Doyle's suggestions and findings elsewhere in this report.
Beginning in 1958 (Doyle, 1959 [168]) information retrieval projects at the System
Development Corporation have had, among other objectives, that of developing ways touse computers in the processing and interpretation of natural language text. By Februaryof 1959, a computer program was already in operation that could search fragments ofabout 100 words of keypunched text, match input words against a pre-established clue word
selection list (i. e. , an inclusion dictionary) and substitute a short encoded form to be
used for subsequent search. Processing of keypunched abstracts using this program involved computer time at the rate of four abstracts per second.
Other features of this text compiler, and of subsequent text processing programs
developed at SDC, enable the making of frequency counts and other statistical measures.
Such features are then used for the investigation of, for example, word-word, worddocument, and word-subject associations, looking toward the determination of answers tosuch questions as: "Do subject words have distribution characteristics within a librarythat a computer program can detect? 11 ]J
Doyle's investigations of word co-occurrences have included hypotheses and testsof various probabilistic measures in terms of observed frequencies, in terms of l f boing! nwords (so-called because of the mental sound effect they elicit), — in terms of adjacent
word pairs and affinities between particular nouns and particular adjectives, 2 / and interms of distinctions between frequency (the total number of times a word appears in agiven library corpus) and prevalence (the total number of items in which a particular word
appears). zJ He has also stressed distinctions between adjacent words and high correlations for words that are not closely positioned together in text, as follows:1/
Compare Doyle himself, 1962, [163], p. 383: "Swanson and others have offeredthesauri of synonyms and related t e r m s . . • (to assist in indexing or searchp r o c e s s e s ) . . .An association map i s , in a sense, an extension of this solution; it isa gigantic, automatically derived thesaurus. Confronted by such a map, thesearcher has a much better 'association network' than the one existing in his mind,
because it corresponds to words actually found in the library, and, therefore, words
which are best suited to retrieve information from that library." See also Wyllys,
1962 [ 6 5 l ] , p. 16: "L. B.Doyle (1961) has invented a fascinating search tool which
seems to us to belong at a level intermediate between automatic indexes and automatic abstracts; i. e. , a possible search method might be to have the computer scan
automatic indexes and compare the index terms therein with the request, thenobtain the possibly pertinent documents and display their association map for theuser to e x a m i n e . . . "I!

Doyle, 1959 [ 1 6 8 ] , p. 6.
Doyle, 1959 [ 1 6 5 ] , p. 5.

±1

Doyle, 1961 [ 1 6 9 ] , p. 12; 1959 C l 6 5 ] , p. 16.

5/
Doyle, 1962 [ 1 6 3 ] , p. 380.

123

11

We have also perceived that two different cognitive p r o c e s s e s s e e m to be
responsible for each type of correlation, one (adjacent correlation) involving
the habitual use of word groups as semantic units, and the other (proximal
correlation) having to do with the pattern of reference to various aspects ofthat which is being discussed. We can call the statistical effects, respectively,
language redundancy 1 , and 'reality redundancy. Such a resolution of statisticaleffects is full of significance for information retrieval because it appears likelythat reality redundancy can vary greatly from one science to another, whereas
language redundancy, a universal property of talking and writing, is relatively
invariant. fl }JWith respect to the "semantic roadmap11 or "association map" technique itself,
Doyle's suggestion is that various measures of word and index term cross-associations
may be applied to the generation of graphic displays of both types of co-occurrence
relationships. Because of the variety of, in particular, the "proximal" correlations, it
is assumed that the literature searcher should be given a display in which the representation of the assemblage of the varied relationships is two-dimensional rather thanone. 2 / An example is given, based upon computer processing of 600 abstracts of SDC
internal reports to find intersections between 500 topical words, of associational connections for the word "output". This was generated by selecting the eight words moststrongly correlated in the data with "output", such as "manual" and "radar", and thenfinding three other words highly correlated with each of these and also correlated with
"output" itself. From the initial graph, it is further shown that item surrogates might
be generated by word selection rules applied to documents to pick up, for example,"New York Air Defense -» system -» data -• outputs -• D.C zJContinuing related work by Doyle and others at SDC has included various experimental studies of "pseudo-documents" consisting of lists of the twelve most frequently
occurring words in 100-item samples of abstracts in various subject fields (Doyle, 1961
[ l 6 l ] ) * Of special interest in terms of potential improvements and modifications tomachine indexing techniques are studies, based on similar l i s t s , looking to the separation of words that may have been used in several different s e n s e s , i. e . , the detection ofhomographs by statistical means (Doyle, 1963 [ l 7 l ] ) < More recent investigations by
Doyle involve considerations of differences between word-grouping and document-grouping techniques and of possibilities for use of hybrid methods.
6. 2. 4 Work of Giuliano and Associates, the ACORN Devices
A program directed toward the design of "an English command and control language
system" under an Air Force contract with Arthur D. Little, I n c . , involves several interrelated aspects of natural language text processing, use of statistical association factors
in search, man-machine interaction during search, and display of associational relationships by means of analog network devices. In this program and in related research,
Giuliano and his associates are convinced that:y
Doyle, 1961 C 169], p. 15.

I!
Doyle, 1962 [ 1 6 3 ] , p. 379.

I'
Doyle, 1961 [ 1 6 9 ] , pp. 24-25.

124

"Automatic index term association techniques are needed to improve the recallof relevant information, to enable indexers and requestors to use language in amore natural manner, and to enable retrieval of relevant m e s s a g e s which aredescribed by different index terms than those used in the inquiry. 11 }J
For the most part, the work to date has been directed to "associative retrieval 11 ofm e s s a g e s limited to single sentences of English text, and to the search phases of a proposed system.
In the case of a corpus consisting of 230 sentences from a single text, a partiallyautomatic indexing method was used. The text was first processed against a modified
version of the Harvard Multipath Syntactic Analysis computer program and the resulting
analyses were manually screened to select a unique, correct analysis for each sentence.
Next, approximately 500 words, those that had been marked "noun11 by the syntactic
analyzer, were listed out and these in turn were manually screened to provide an"inclusion list" of 273 words. Sentences were then "indexed" with respect to which ofthese selected words they contained. Word associations were computed both in terms ofco-occurrence within a sentence and of co-occurrence in syntactic structures.
Retrieval tests were then applied using both computer programs and the analog
device, and evaluations were made on the basis of examining sentences selected in orderof machine-ranked relevance and of comparisons of word lists associated with a given
search term against association lists for another term picked at random. It is noted that,"although quantitative conclusions cannot be drawn", the results support the conclusion
that: "Items retrieved due to automatically-generated associations tend to be more r e l e vant than is explainable on a chance b a s i s . " £'The "request reformulation" retrieval program has also been used to generate term
profiles from a collection of approximately 10, 000 documents (previously indexed with atleast 6 terms from a selective term vocabulary of 1, 000 terms) which have then been
compared against lists provided in the entries for corresponding terms in the Thesaurus of
ASTIA Descriptors, Second Edition. The machine-produced association l i s t s , at least
for those words occurring relatively frequently in the corpus, appear to give thesaurus
entries that are extensive, specific, and intuitively acceptable, and of high quality,
especially with respect to listings of synonyms as well as factually related words* _'The development of the ACORN (Associative Content Retrieval Network) devices
has provided additional tools for testing and display (1962 1T229L 1963 [227, 304]).
These devices are networks of passive resistance elements. Each word or index termand each sentence (240 by 230 in ACORN-IV) are represented by terminals interconnectedby resistors with conductance equal to the connection strength, and with "leak" resistorsyGiuliano 1962 [ 2 2 8 ] , p. 10.
Giuliano et al, 1963 [230], p. 47.

I'
Ibid, pp. 57-58.

125

providing for various normalizations that may be applied to compensate for word or
sentence frequency factors. These devices differ from the earlier EDIAC in thevariable weightings provided, in the normalizations that may be applied, and in multipath
interconnections.
When, for example, currents are applied at some of the word terminals, the voltages appearing on any of the other word terminals depend on the strengths of association
between these words and the input words via all direct and indirect paths. The responses
of sentence terminals to the input words of a query similarly depend upon how strongly asentence is connected to these words and how strongly it is connected to other words
which in turn are strongly connected to the query words. It is to be noted further that:"Pulling out or cutting a few randomly selected wires in an ACORN generally
has a surprisingly small e f f e c t . . . This insensitivity is of course, explainable
in terms of the multiplicity of indirect and redundant association paths which
remain intact when a direct path is s e v e r e d . . . I t . . . suggests that the retrieval
process can indeed be made insensitive to minor variations in indexing. " 2 /In addition, there are intriguing possibilities for imposing a "viewpoint" withrespect to a search by injecting bias currents. Thus if only non-"Air Force" jetplane items were desired, the "Air Force" items could in effect be grounded out. If there
were no jet items in the collection other than those which were also Air Force items,
these would be indicated as responsive, but largely they would appear only if this shouldbe the case. Some words used have some connection to almost all other words, but thesehave little effect in the system and the hardware thus tends to compensate for the highfrequencies of very general words.
6. 2. 5 Spiegel and Others at Mitre Corporation
Bennett and Spiegel, reporting at the Symposium on Optimum Routing in Large
Networks, IFIP Congress-1962, £ ' consider modifications to formulas for the calculationof statistical association factors which will normalize against such influences as frequencyof word occurrences, relative word position within a string of words, and string length.
This work has been carried forward at the Mitre Corporation in a program for developing
procedures to encode various statistical properties of m e s s a g e s or documents and to use
these codes for message routing and retrieval.
Differences between this approach and those of Maron and Kuhns, Stiles, and Doyle,relate primarily to the questions of how best to normalize. The objective is closely
similar: to use associational weighting so as to provide, in response to a query, output of
documents or m e s s a g e s ranked in order of probable relevance to the query.
1/
Giuliano and Jones, 1962 [ 2 2 9 ] , p. 22.
See Juncosa, 1962 [ 3 0 6 ] , especially paper 4, E. Bennett and J. Spiegel,"Document and m e s s a g e routing through communication content analysis 11 ,pp. 718-719.

126

Additional features include provision for the m a t r i x of coefficients of association
to change with time or with deliberate manipulation to improve performance. Thus:"Each normalized cell weight. . . r i s e s and falls with time as each specificassociation i n c r e a s e s or d e c r e a s e s in relative frequency. In this way, them a t r i x m e m o r y of associations changes with time, maintaining a cumulative
pattern of associations reflecting one statistical c h a r a c t e r i s t i c of m e s s a g e s
fed into it in the past. . .
"In addition to this adaptive characteristic of changing m e m o r y with time and withchanging inputs, the m a t r i x is also readily subject to formal education. Any
specific cell weight can be strengthened by repeatedly reading into the m a t r i x
m e m o r y the specific strings that contain the desired a s s o c i a t i o n s . F o r example,
by introducing the strings is am, is a r e , am i s , am a r e , and a r e am, we canc r e a s e the statistical tendency of the tokens is, am, and a r e to be associated. " —"N!

Experimental r e s u l t s have been obtained for a corpus of 500 bibliographic entries
contained in DDC's Title Announcement Bulletin. In the case of a t h r e e - t e r m query, 40
items w e r e selected and ranked in probable relevance o r d e r , with selection based on ap a r t i c u l a r relevance s c o r e value threshold. The investigators then reviewed the a b s t r a c t sof all 500 items and rated them as to relevance with r e s p e c t to the query. Seven
additional items were found, of which three would have been m a c h i n e - s e l e c t e d with al e s s stringent selection threshold. F o r the remaining four, it is reported that they "werepcorly indexed and could have been judged not relevant by a human who depended upon thed e s c r i p t o r string only, as the m a t r i x did, r a t h e r than upon review of the a b s t r a c t s . " 2/
6. 3

Clues to I n d e x - T e r m Selection from Automatic Syntactic Analysis

Several of the organizations and r e s e a r c h teams most active in the investigationof linguistic data processing techniques, especially for automatic indexing, extracting
and s e a r c h renegotiation applications, a r e actively considering the use of clues derived
from automatic syntactic analysis to improve c r i t e r i a for machine selection of"significant" words, p h r a s e s , and sentences from raw text. Such approaches, in general,
however, are subject to the limitations of non-availability of sufficient corpora of text
in machine-usable form, in the first place, and, even m o r e importantly by the nonavailability of satisfactory computer p r o g r a m s for complete syntactic analysis up to theuSpiegel et al, 1963 [ 5 6 6 ] , p. 17.
2/
Ibid, p. 34.

127

present time. — In terms of the state-of-the-art of automatic indexing, therefore, weshall not consider these approaches as more than indications for future research. A few
suggestive examples are discussed briefly below.
The multi-pronged attack on mechanized information selection and retrieval
problems headed by Salton and his associates includes the exploration of tree structures,to represent both the relationships between terms in a classification schedule or indexing
term vocabulary and the representation of the results of automatic syntactic analyses ofnatural language text. It is proposed, then, that computer programs can achieve transformations of the syntactic trees representing word strings in the original text into
simplified, condensed structures with normalized terms and can compare these trees
with the classificatory trees (Saitoh, 1961 [516]). Manipulation of such trees together
with appropriate dictionaries or thesauri can result, for a given proposed index term, inthe finding of a preferred term for a particular system, or a set of synonymous terms, ojf^ysets of all terms in which the given term is included, and the like.
Anger considers some of the problems involved in complete syntactic analysis oftexts with the objective of identifying the total network of relationships expressed andimplied, as proposed by Lecerf, Ruvinschii, and Leroy, among others, of the ResearchGroup on Automated Scientific Information (GRISA), EUR ATOM. Assuming that computer
programs for syntactic analysis are or will be available, he suggests that simplifications
may be obtained by determining only the basic relations that are indicated by direct
syntactic dependencies or by linking words, (Anger, 1961 [ l 5 l ).
A specific program for automatically extracting syntactic information from text has
been studied by Lemmon (1962 [ 3 5 4 ] ). The possibilities for combining dictionary lookups,
word suffixes as indicators of syntactic role, and predictive syntactic analysis for text
processing have also been further explored by Salton himself (1962 [ 5 1 8 ] , 1963 [ 519] ).
A variety of word and document association techniques and of synonymous word andphrase groupings which serve to "clue" the selection of a subject heading are also being
investigated by members of the Harvard group and guest investigators.

yMajor difficulties have to do with limitations both upon grammars and vocabularies
so far tested and with ambiguities and the number of alternative parsings generated.
See, for example, Bobrow, 1963 [ 6 8 ] , Kuno and Oettinger, 1963 [ 3 4 l ] andRobinson, 1964 [ 502]. Bobrow provides a survey of syntactic analysis programs
as of 1963, noting limitations or restrictions on each. He reports, for example,
that available programs to compute word classes are not always correct in theclass assignments made and that analysis systems are not complete unless they
provide means for distinguishing between "meaningless strings and grammaticalsentences whose meaning can be understood 1 '. He concludes: ''Until a method ofsyntactic analysis provides, for example a means of mechanizing translation ofnatural language, processing of a natural language input to answer questions, or ameans of generating some truly coherent discourse, the relative merit of eachgrammar will remain moot." ( [ 6 8 ] , p. 385) Robinson ( [ 5 0 2 ] , p. 12) says ofsentences which can be parsed correctly, that they are: "Usually short sentenceswith no complicated embeddings of relative clauses and few participial or
prepositional phrase modifiers. These include the basic sentences that most
grammars are equipped to handle and that adult writers seldom produce."128

Another p a r t i a l approach to applying syntactic analysis techniques to automatic
indexing is based upon syntactic word-class recognitions. Giuliano and his a s s o c i a t e s
at Arthur D. Little, Inc. , (1963 [230] ), have investigated on a s m a l l - s c a l e basis theuse of the Kuno-Oettinger p r o g r a m s developed at Harvard for this purpose (Kuno andOettinger, 1963 [ 3 4 0 ] ) . The broad prograrn of information and language data processingr e s e a r c h at System Development Corporation specifically includes investigations ofstructural patterns of sentences at the syntactic level and also of semantic factors
such as the studies of polysemy and homographic ambiguity by Doyle, W a s s e r , ando t h e r s . Borko r e p o r t s :n. • . We. . . a r e analyzing actual written text for multiple meanings. . . The data
for this study w e r e drawn from the corpus of 618 psychological a b s t r a c t s .
Tabulations of frequency of paired and single word listings were used. A
number of corpus-derived word frames have been p r e p a r e d . Although thisr e s e a r c h is still in its early phase, we feel that we have made a good s t a r t
on the problems of semantic analysis. " z!
In Czechoslovakia, at the Karlova Universita, both statistical and semantical methods for 1
automatic abstracting a r e reported as being under consideration. iL'
Other examples of proposals for the use of syntactic/ analysis techniques for the
improvement of automatic indexing products include those of Spangler, Levery, Plath,Thorne, and Climenson and his colleagues at RCA, as well as the suggestions of those
whose i n t e r e s t s in automatic syntactic analysis have been p r i m a r i l y directed to problems
of machine translation or m o r e general problems of linguistic analysis. Hays, forexample, although principally concerned with MT, indicates that the methods fordetermining p h r a s e s t r u c t u r e s have obvious applications to the automatic determinationof categories useful in the indexing of documents. 3JAn existing GE-225 computer p r o g r a m for KWIC-type indexing from both titles andabstracts at General E l e c t r i c ' s Phoenix L a b o r a t o r i e s is being extended to incorporate
word analysis features taking into account both syntactic and semantic aspects of a given
line or sentence of text. ZJ Levery provides an example of s i m i l a r directions being explored in European research, m o r e generally oriented toward linguistic considerations
as such than to machine-derivable c r i t e r i a (largely statistical to date), which seek tocombine the benefits of both human and machine p r o c e s s e s by way of automatic syntactic
analyses. He claims, for example, that:

J7
Borko, 1962 [ 7 5 ] , p. 6.

U
National Science Foundation's CR&O report, No, 11 [ 4 3 0 ] , p. 123.

I'

1/

See Hays, 1961, [ 2 5 8 ] , p. 13: n . . . Two broad problems on which work is justbeginning at RAND: g r a m m a t i c transformations and distributional s e m a n t i c s . The
latter problems a r e especially important for automatic indexing, abstracting, andtext searching.» See also de Grolier, 1962 [ 1 5 2 ] , p. 137.
National Science Foundation 1 scR&D report No. 11 [ 4 3 0 ] , p . 2 1 .

129

11

. . . The study of the position of keywords in the text and the syntactical
relationship which exists among them will show the way to automatic abstracting and the use of more sophisticated retrieval systems. n }JPlath suggests that, given a computer program to perform the parsing andsyntactic diagramming of a text sentence, the results can serve quite usefully to augment
the selection criteria based initially on statistical techniques, such as word-frequency
counting. He says, for example:''Another possible application of the outputs of the sentence diagramming programis their employment as an aid in language data processing for purposes of
information retrieval, particularly in systems for automatic literature abstractingof the sort proposed by Luhn (1958). The feature of the tree diagrams which ispertinent here is that the main components of a clause, including subject, verb
and object, always correspond to the 'main topics 1 in an outline, and are therefore
located at the upper levels of the tree. When the words on these upper levels are
considered apart from the lower-level structures which modify them, they often
summarize the content of the sentence in a sort of 'newspaper headline' or 'telegraphic style 1 ." zJ
The problems of multi-level selection, or screening, such that machine programsfor selection of the most probably significant words, phrases, or sentences can befocussed upon the most probably content-relevatory areas of text, are treated here, asalso by Salton, in the sense of a cutting-off at a given depth in the analyzed syntactic
structure. zJ A potentially important contribution to the future prospects for automatic
indexing, however, lies in the "discourse analysis" and "transformational linguistics"
approach of Harris (1959 [ 2 5 4 ] ) , where condensations and concentrations of similarities
and differences of topical interest may hopefully be achieved.
Harris himself suggested, at least as early as 1958, applications of his approach toboth automatic indexing and abstracting. A goal of the analyses he has proposed is to
identify 'kernels' of linguistic expression, having first, by various transformations suchas from passive to active voice, brought together different w^ys of saying the same thing.
He then suggests not only machine operations to normalize ^ a p p l i c a t i o n of his transformational rules but also to determine:" . . . Which kernels have the same centers in different relations (e. g. , withdifferent adjuncts), and other characterizing conditions. The results of thiscomparison would indicate whether a kernel is to be rejected or transformed
into a section.. . of an adjoining kernel, or stored, and whether it is to beindexed, and perhaps whether it is to be included in the abstract. " i /U
Levery, 1963 [ 3 5 9 ] , p. 236.

D

Plath, 1962 [ 4 7 4 ] , pp. 189-190.

3/
See also Thorne, 1962 [ 6 0 5 ] , p. v: "The approach followed requires that the computer itself syntactically analyse input text in order to convert it into special form
called FLEX, which preserves only that syntactic information which is useful fordata retrieval purposes. "4/Harris, 1958 [254],. p. 949.
130

Certain difficulties a r e self-evident. Consider, for example, the admittedly hypotheticaltext which might refer in various places to the "dissolute, disreputable, i l l i t e r a t e , elder
Lincoln 11 (underlining supplied) and which might be so p r o c e s s e d by machine as to imply
that Lincoln the son w a s , although also P r e s i d e n t of the United States, "dissolute, n" d i s r e p u t a b l e , " " i l l i t e r a t e , " and " e l d e r . " T h e s e , however, a r e difficulties that plaguealmost any machine p r o c e s s i n g of natural language text.
Climenson, Hardwick, and Jacobson have explored some of the possibilities of the
H a r r i s approach in experimental computer p r o g r a m s for the RCA 501 (1961 [ 133] ).
Specific features of these p r o g r a m s include:
1.

Establishment of the syntactic class or c l a s s e s to which a given word canbelong, by dictionary lookup.

2.

Investigations of sentence s t r u c t u r e and context in an attempt to resolve thenomographic ambiguities involved when the same word may function eithera s a noun or a v e r b .

3.

Isolation and marking of sentence segments, such as noun p h r a s e s , p r e positional p h r a s e s , adverbial p h r a s e s , and v e r b p h r a s e s .

4.

Identification and marking of segments - - clauses or degenerate c l a u s e s .

On a v e r y p r e l i m i n a r y b a s i s , a limited set of word and p h r a s e deletion rules w e r e
set up and s e v e r a l sample documents w e r e p r o c e s s e d against them, yielding reductions
to about 35 percent of the original text. These r e s u l t s suggest that "syntactical filteringc r i t e r i a " might be applied to the improvement of modified derivative indexing techniques,
such a s the word-frequency counting techniques, either by deleting syntactically insignificant p a r t s of selected sentences, or by counting identical p h r a s e s r a t h e r than w o r d s . The
investigators conclude, however, that:"A formal linguistic approach to the p r o b l e m s of n a t u r a l language processing
p r o m i s e s to yield r e s u l t s vital to the s u c c e s s of automatic indexing and data
extraction. But the work r e q u i r e d in s u c h an approach will be quite arduous;a long-range m a n - m a c h i n e effort will be required to formulate p r a c t i c a lmachine p r o g r a m s for indexing and a b s t r a c t i n g . " 1 /A final special case of linguistic data processing involving syntactic analysis isthat of Langevin and Owens. They claim:"A c r i t i c a l review of the analysis work done on the Nuclear Test Ban Treaty
by use of the Multiple Path Syntactic Analyzer d e m o n s t r a t e s that such a device
can, even at p r e s e n t , provide a powerful technique for the systematic discoveryof ambiguities in t r e a t i e s and other documents. Because the analyzer operates
without bias from the overall context of the document, it may sometimes bepossible for it to discover ambiguities that would easily escape a human reviewer
who knows what the document is 'supposed to s a y ' . " —'1/
Climenson et al, 1961 C 133], p . 182.
2/
Langevin and Owens, 1963 [346], p . 26.

131

6. 4

Probabilistic Indexing and Natural Language Text Searching

As in the case of automatic indexing proposals based upon automatic sentence
extraction techniques, machine searching of full natural language text has been suggested
as a basis for, at least, automatic derivative indexing. We have remarked previouslythat the machine use of complete text can only be considered to be ''indexing11 in a very
special sense, that it is subject either to the non-availability of suitable corpora alreadyin machine-usable form or to high costs of conversion to this form, and that too little
is yet known of linguistic analysis and searching-selection strategies effectively applicableto natural language materials. Various examples of corroborating opinion, other thanthose previously cited, are as follows:"Machine searching is superb if it is known exactly how to describe the object ofsearch, and if one could know how to choose from among many possible searching strategies. I doubt if any one is yet in this comfortable position with respect
to machine searching of text. f l zJ
"The most effective programs in automatic linguistic analysis have served only
to illustrate how really complex is the structure of the language, and how far
removed the present state of the art is from any system which might be usefulin practice* " 2 /"The recognition of words involves only the matching of digital codes, but
the recognition of an idea is a severe intellectual problem, the solution towhich will probably never be exact. Nevertheless, this is the problem whichmust be attacked if accuracy is ever to be attained, or even approached, inusing the text of information items as a basis for their recovery."
Nevertheless, some of the work both in natural language text searching and in"probabilistic indexing" (where weights representing judgments as to degree of relevanceof an indexing term to an item are used either in indexing or search), provide instructive
insights into some of the problems of automatic indexing.
In the period 1958-1960, work at Ramo-Wooldridge resulted in the release or
publication of provocative papers by Maron, Kuhns, and Ray on "probabilistic indexing"(1959 [ 3 9 8 ] , I960 [ 397] ) and by Swanson on natural language text searching by computer(I960 [ 587, 5 8 2 ] , 1963 [ 583] ). Subsequent work along these lines has included further
developments at Thompson Ramo-Wooldridge, the law statutes work at the Health J-»aw
Center at the University of Pittsburgh, and the experimental investigations of Eldridgeand Dennis in a project jointly sponsored by the American Bar Foundation, IBM, and theCouncil on Library Resources.

T7
Doyle, 1959 [ l 6 8 ] , p. 2.

ySalton, 1962 [ 5 2 0 ] , p. Ill,-1 through III-2.

I'
Doyle, 1959 [ l 6 5 ] , p. 12,132

6.4.1

Probabilistic Indexing - Maron, Kuhns, and Ray

The work in the area of "probabilistic indexing11 involves, as in the case of Stiles 1statistical association factors, an assumption that there should be machine means available for the automatic elaboration of search requests in order that relevant documents notindexed by the precise terms of these requests may be retrieved. Given that measures of"closenesses 1 1 and "distances 11 between similar documents can be obtained, probabilistic
weighting factors between index terms assigned to documents may be made explicit.
More generally, however, the notion of probabilistic indexing is based upon the assignment of weights that provide a numerical evaluation of the probable relevance of index
terms to a particular document, and of the relative importance of the various terms
used in a search request. Maron and Kuhns (1963 [ 397] ) thus consider the following
variables important in the formulation and following out of search strategies:
1.

Input- both the terms of the request and the weights assigned to them.

2.

A probabilistic matrix giving dissimilarity measures between documents,
significance measures for index terms, and closeness measures between
index terms.

3.

A priori probability distribution data.

4.

Output- a class of retrieved documents ranked in order of their "computed
relevance numbers" and an indication of the number of documents involvedin the c l a s s .

5.

Search parameter controls, such as the number of documents desired.

6.

Search prescription renegotiation involving amplification of the request byadding terms "close" to the ones in the original request and the selectionof additional documents following distance criteria for the collection. zJExperiments have been reported for 40 requests run against 110 articles taken from
Science News Letter. Without search renegotiation, the "answer" document was
retrieved in only 27 of the 40 tests. Three alternative methods of request elaboration
were then tried. First, additional terms most strongly implied, statistically, by theterms in the request were used. Secondly, those terms were added which most strongly
imply, again in a statistical sense, each of the given request terms. Thirdly, c o efficients of association between index terms were used. Results are reported as follows:"(1)

Using the method of request elaboration via forward conditional
probabilities between index tags, we retrieved the correct answer
document in 32 cases out of the 40.

(2)

Elaborating the requests via the inverse conditional probability heuristic,
we retrieved the correct document in 33 of Jhe 40 c a s e s .

(3)Using the coefficient of associationto obtain the elaborated request we
obtained success in 33 cases of the 40.

yMaron and Kuhns, I960 [397], pp. 230-231.

133

"Thus we see that the automatic elaboration of a request does, in fact, catch relevant documents that were not retrieved by the original request. f f U
6.4.2

Natural Language Text Searching- Swanson

The work in automatic indexing and related research directed by Swans on at Ramo
Wooldridge Corporation has included "indexing at the time of search 1 ' in natural language
text searching, (I960 [ 5 8 2 , 5 8 7 ] , 1963 [ 5 8 3 ] ) , the previously mentioned studies ofmachine-like indexing by people (Montgomery and Swanson, 1962 [ 4 2 l ] ) , and automatic
assignment indexing using pre-selected lists of clue words, (Swanson, 1963 [ 5 8 0 ] ) . The
last of these three major areas of investigation is the one of the greatest interest in thispresent study, but the earlier experiments in machine searching of natural language texts
warrant some discussion. In his reports on this text searching project, Swanson has
specifically claimed that the methods for transforming search questions can serve as
the basis for an automatic indexing method* Thus:n. . . A technique for automatic indexing can be derived immediately from a text
searching technique... it is necessary only to so organize the machine proceduresthat those operations of text reduction or reorganization common to all searches
are performed only once and prior to searching in order to create directly an
automatic indexing procedure. n
Swanson has also claimed that if automatic searching of full text is not feasible,
then automatic indexing is not feasible, the one being prerequisite to the other. For
example:"Clearly, if a computer technique for search and retrieval from the full textof a collection of documents cannot be developed, then it is unthinkable thatmatters could be improved by using the machine to operate on just part ofthe information (a 'condensed representation 1 ) — that i s , on an automatically
produced index. This line of argument demonstrates persuasively that thedevelopment of techniques for automatic full-text search and retrieval is aprerequisite to automatic indexing. It is equally clear that a technique for
automatic indexing can be derived immediately from a text-searching technique, and thus that the two processes involve conceptually equivalent
problems. n 2 /In the actual text searching experiments, a model "library" consisting of 100 shortarticles in the field of nuclear physics was set up in machine-us able form. These articles
were also studied by subject specialists who rated the relevance of each paper to each of
50 questions, and assigned weighting factors representing the degree of judged relevance.
A second group of people, who knew only that the papers were in the field of nucleary
Ibid, p. 240.

I!
Swanson, I960 [ 5 8 2 ] , p. 6.

1'
Swanson, I960 [587], p. 1100.

134

physics, then transformed the 50 questions into search prescriptions using threedifferent methods. The first method for the development of the search instructions wasto choose appropriate index entries from a subject heading list tailored to the contents ofthe sample library. Search was then made manually against a card catalog which
recorded the results of manual indexing of the same 100 articles to the entries of thislist.
The second method of search prescription tested involved the specification of
combinations of words and phrases likely to be found in any paper which would in fact berelevant to the search question. The third method involved modification of the second by
the use of a thesaurus-type glossary which suggested various alternative terms. Boththe latter two types of search instructions were fed to a computer program which carriedout searches against the natural language text consisting of 250, 000 words from theoriginal articles.
The results were then evaluated in terms of ratings of relevance made by thephysicists who had analyzed the papers. Retrieval effectiveness was not high: M. . . in nocase did the average amount of relevant material . . . retrieval (taken over 50 questions)exceed 42 per cent of that which was judged . . . to be present in the library. " ±1 However,
the results were indicative of the superiority of the machine methods to the manual catalog search.^/For this library in particular, in the case of "source documents 1 ' (thearticles from which the search questions were taken), only 38 percent of the relevant
papers were located by the manual search, whereas 68 percent of the relevant items
were retrieved by machine search of the text for specified words and phrases in various"and11 and "or" combinations. Machine search based on search instructions that had beendeveloped with the assistance of the thesaurus-glossary yielded 86 percent of the relevant
source item documents.
6. 4. 3 Full Text Searching - Legal Literature"The retriever of documents may be satisfied with a sample of descriptors that
represent the contents; the fact retriever or the question answerer must often haveaccess to every word in the text". _' The objective of fact retrieval is a major goal in
the experimentation that is being carried forward in the field of natural language text
searching of legal material, especially the texts of statutes of the State and Federal
Governments. The most extensive program to date is that of Horty and his colleagues
at the University of Pittsburgh Health Law Center (I960 [ 2 7 7 ] , 1961 [276, 309], 1962
[196, 2 7 8 ] , 1963 [ 2 4 , 280]).
Wilson at the Southwestern Legal Foundation is experimenting with a modified
version of the Horty-Pittsburgh System for legal cases dealing with arbitration in five ofySwanson, I960 [ 5 8 2 ] , p. 25.

D
Ibid, p. 1: "On the whole, retrieval effectiveness was rather poor, yet machine
search of the text of the model library was significantly better than was humansearching of the subject heading index. "1'
Simmons and McConlogue, 1962 [ 5 5 5 ] , p. 3.

135the southwestern states.— A joint American Bar Foundation—IBM research program has
been established to explore both text searching without prior indexing and automatic indexing techniques (Eldridge and Dennis, 1962 [ 1 8 3 ] , 196 3 [ 1 8 2 ] ) .
In the Horty-Pittsburgh System, approximately 6, 000,000 words of text have been
converted via Flexowriter to magnetic tape. An exclusion dictionary of 100 words is used
to eliminate the most common words and a word-concordance is prepared, resulting inword-occurrence location indicia by position in sentence, paragraph and section of thestatute. In searching, the user has available to him the alphabetized list of approximately
17, 000 different words and it is up to him to think of the words and synonyms most likely
to occur in statute sections likely to be the ones he seeks. Several search logics areavailable. One provides that at least one of a group of alternate words must appear;
another requires that at least one from two or more groups must appear in the same
Sentence. Intra-sentence distance criteria are also utilized: "If the phrase 'born out of
wedlock 1 i s sought, the operator. • • requires that the word 'wedlock 1 appear in the samesentence, no more than three words after f born f . n **Obviously, for the same question the searcher would also have to specify synonymous words and phrases--"illegitimate children", "illegitimate births", "unwed mothers","unmarried mothers", "illegitimacy", "bastardy", and so on. The reported success ofthe system is apparently due in large part to the ingenuity of the searchers in specifying
the expressions and synonyms most likely to be used. Hughes comments as follows:"It should be noted that this system will be most efficient only when the users
are thoroughly familiar with the linguistic style of the source material andsearch is made on words Known to occur in the appropriate statutes".
6. 5

Other Examples of Related Research in Linguistic Data Processing

Since, as Garvin has emphasized, "All areas of linguistic information processing
are concerned with the treatment of the content, rather than merely the form, of documents composed in a natural language," •*' much of the research in linguistic dataprocessing is potentially applicable to both the development and the improvement of
automatic indexing techniques. Thus developments in automatic content analysis, in
psycholinguistics, in question-answering systems, may eventually find application tomechanized indexing systems.

uEldridge and Dennis, 1964 [ 182], p. 90; Wilson, 1962 [ 6 4 5 ] .
Horty, 1962 [ 2 7 8 ] , pp. 59-60.
Hughes, 1962 [ 2 8 4 ] , p. IV-6 to IV-8.

136

In t e r m s of our present concern, however, we shall select only a few examples.
By automatic content analysis is meant the use of computer programs to detect or selectcontent themes in a sentence-by-sentence scanning of text or verbal protocols". JV Thei n t e r e s t of psychologists in machine techniques to a s s i s t in the analysis of linguisticallygiven materials, as in propaganda analysis, probably precedes at least in sophistication
if not by date, that of documentalists or of machine specialists interested in library andinformation problems. zJ
11

3/
The "General Inquirer11 program developed by Stone et a l , — is an example ofquestion-answering techniques based upon selective extractions from natural language
text. It involves the use of a m a s t e r vocabulary consisting of words previously selectedby an investigator as being likely to be content-indicative in a body of m a t e r i a l to beprocessed, together with his pre-established indications of the categories he expects
their occurrence should predict. It is to be noted that this is a c u s t o m - t a i l o r e d set of
categories and of clue-word l i s t s associated with each, manually pre-established. Text
is now processed in such way that each word is looked up and, if it appears in the m a s t e r
vocabulary, it is tagged with identifiers of the categories for which it is presumably
predictive. A subsequent "Tag Tally11 routine then counts the tag frequencies to determine for which categories the input material has high or low s c o r e s , and these in turncan be compared with expected n o r m s .
This type of program has been applied to such varied materials as suicide notes,
folk tales from different cultures, reports of field workers, recordings of group d i s cussions as in supervisory-leader ship training s e s s i o n s , and protocols for various
psychological tests. 2/ Interesting variations developed by Jaffe and others 2 / involvethe use of non-verbal as well as verbal clues as content-indicators, specifically, timesequence p a t t e r n s recorded along with the words spoken in client-the rapist s e s s i o n s . At
the meeting of the Association for Computational Linguistics and Machine Translation
held in Denver, August, 1963, Jaffe reported findings indicative of positive correlation
between the s t r u c t u r e of temporal and lexical p a t t e r n s in dialogue and suggested applications to automatic abstracting or indexing by the use of the time-sequence patterns asclues to high information-value areas.

yFord. J r . , 1963 [498], p. 3.
2/
See, for example, Jaffe 1952 [297], Hart and Bach, 1959 [256], Pool, 1959 [475 ] ,the latter covering the proceedings of a conference held in 1955.

2/
Stone and Hunt, 1963 [576]; Stone et a l , 1962 [575].

±1
See Ford, 1963 [498], p. 8.

y
See for example, Cassotta, et al, 1964 [ 1 0 4 ] ; Jaffe, [294 J to L^V (].

137

Hughes provides, as of September, 1962 ([284] ), a critical review of several
experimental and proposed question-answering systems using natural language statements
and natural language queries, including /'BASEBALL 1 ', ±J "SAD SAM1' U and the l ! Proto
Synthex11 investigations of System Development Corporation. —' Later developments on
the Synthex (synthesis of complex verbal material) project at SDC have included avariation on a natural language text searching program where ordinary text input is run
against an exclusion list and a table is set up to tally the substantive words remaining.
Words with the same roots or previously having been identified as synonymous are c r o s s referenced. A complete index results, with document location identifier tags for theword occurrences down to the single sentence level. This index can be used subsequently
to locate regions of text (volume, chapter, paragraph, and sentence) where answers
responsive to input questions are likely to be found.
It i s proposed that the Synthex system eventually should incorporate analyses ofsyntactic and semantic relationships in the linguistic expressions of both queries andtext. Of future interest in the extension of such considerations to automatic indexing andabstracting are the following comments:"The results of several early experiments within the project, coupled with thefindings of other language researchers, led to the following conclusions about
meaning and grammatical structure in English text:
1.

The degree of synonymity in meaning between any two English
words can be measured quantitatively with a synonym dictionary
and relatively simple scoring procedures.

2.

The difference in meaning between two sentences of identicalsyntactic structure can be expressed quantitatively as a functionof synonymity of their words. . . f l —'It is also of interest to note that although the "indexer" program of the Synthex
system provides cross-referencing between, for example, "whales 11 and "whaling1' or"England" and "Great Britain", the investigators admit that: "naturally it falls short ofsuch complicated cross-referencing as 'mouse-animal' 'Jones person' and otherconcept recognitions. " zJ However, concept recognitions based upon both a priori andu
See also Green et al, 1961 T238].

£/
See also Lindsay, I960 [ 3 6 3 } .
3/See also Klein and Simmons, 1961 [325 j ; Simmons et al [552]to [ 5 5 5 ] .

±'System Development Corporation, 1962 [ 590] .

I'
Simmons and McConlogue, 1962 [ 555 1 , p. 70.

138

a posteriori associations are at least foreshadowed in a s m a l l - s c a l e model of attributewords and proper names, together with prespecified relationships between them; U in
Olney's recent work at SDC exploring the possibilities for use of cognitive concepts as
b a s e s for establishing association between documents,
by Kochen f s work on machine 1
inference and concept p r o c e s s i n g . zJ
A final example of potentially related r e s e a r c h in the a r e a of content analysis istherefore the work of Kochen, Abraham, Wong and others at IBM's Thomas J. Watson
L a b o r a t o r i e s (1962 [ 3293). While concerned principally with adaptive organization andp r o c e s s i n g of stored factual statements and the possibilities for machine formulationof "hypotheses" about these and additional facts, some consideration has been given to
sampling p r o c e d u r e s applicable to determination of s i m i l a r i t y which might be used for
document clustering and to the possibilities for dynamic clustering for r e t r i e v a l based
upon a specific individual query. —' In the proposed AMNIP (Adaptive Man-machine NonA r i t h m e t i c a l JDnformation P r o c e s s i n g ) system,, t h e r e is no attempt at either automatic
indexing or automatic abstracting, j j / Instead, formal statements a r e made about named"things" and their a t t r i b u t e s . The sharing of common attributes then s e r v e s as a b a s i sfor relating i t e m s which a r e s i m i l a r and for grouping them together in the s y s t e m
m e m o r y . It is assumed that the organization of the stored statements changes dynamically with new data inputs and u s e r feedback in question-answering routines.
Where the named items a r e n a m e s of documents or of index t e r m s , a number of
documentation applications can be considered. Where the i t e m s a r e document n a m e s
and the formal p r e d i c a t e is " c i t e s " , the s y s t e m provides a p r o c e d u r e for production andu s e of citation indexes. zJ Where the i t e m s a r e index t e r m s or subject headings andthe p r e d i c a t e s a r e " i s used synonymously with" or " i s subsumed under", machine
construction of a growing thesaurus based on u s e is suggested. ±J The common attributeu

See Stevens, I960 [ 568]; see also H e r n e r , 1962 [ 2 6 6 ] , p . 5.

2/

See Borko, 1962 [ 7 5 ] , p . 5: "Instead of defining meaning in t e r m s of synonyms,
it is defined in t e r m s of the entities r e f e r r e d to by the word in context. A chairis thus described as belonging to a class defined by a given l i s t of p r o p e r t i e s . . .
Analysis yields an interpretation of the sentence as an a s s e r t i o n that certain
relationships hold between the specified referent c l a s s e s . The cognitive content
of the sentence is a function of this assertion plus the information about thesereferent c l a s s e s which has previously been stored in m e m o r y . "1'

Kochen et al, 1962 [ 329] .
Ibid, Appendix by C. T. Abraham, pp. 20-65.
Kochen et al, 1962 [ 3 2 8 ] , p. 45.

£/

Ibid, p . 37.

7/
Ibid, p. 37.

139

matching program, applied to logical similarities of texts related as by having various
assigned descriptors or citations in common, might provide a basis for generating
document surrogates by representing each text in a related group of texts with the words
or sentences these texts have in common. JL'
In the case of man-machine interaction during search, it is suggested that the user
should indicate the names of selected documentary items which are of particular interest,
then:"The machine forms an 'hypothesis 1 about the subset of articles likely to be ofinterest. It does this by examining all recorded statements common to the ones
selected but not to the rejected ones. The weight of different attributes anddegree of interest is taken into account. The machine may display this hypothesis or another random sample of titles consistent with it, or both." zJ
6. 6

Machine Assistance in Translations of Subject Content Indications to Special Searchand Retrieval Language

There are, also, in the areas of directly and indirectly related research, certain
programs of research, development, and experimentation which include investigationsof possibilities for using machines to assist in the "translation 11 of textual languages into
special intermediate or "documentary11 languages. Doyle's use of the inclusion list
principle to extract specified content-indicative words and to encode them in his "bigram"index was an early but relatively trivial example.
The work of Williams and her
associates, at Itek and elsewhere, z! has involved the objectives of determining which of
the subject-revealing implications of titles, abstracts and, if necessary, full text, aresusceptible to machine detection and manipulation such that the implied as well as the
explicit assertions made in a document may be incorporated in a formalized language for
retrieval.
While Williams, Barnes, Cardin and Levy, and others, have so far approachedsuch tasks primarily from the standpoint of human analytic judgments, Coyaud (1963[ 1 4 3 ] ) has dis cussed at least preliminary work looking toward the automation of theanalysis of natural language texts for purposes of encoding and organization of the terms
and relationships to be used in the "documentation language" known as "SYNTOL"(Syntagmatic Organization of Language), this work has used a corpus based on bibliographic abstracts from the Bulletin signaletique of the Centre National de la Recherche
Scientifique, Psychophysiology Section, for the period' 1958-1960. Notwithstanding suchdifficulties a s determining rules for proper subdivisions of text, reduction of synonyms,
resolution of lexical and syntactic ambiguities, and the fact that some words are always,y. Kochen et al, 1962 [ 3 2 9 ] , p. 2.

1)
Kochen et al, 1962 £ 2 8 ] , p. 7.

1'
Doyle, 1959 [ l 6 8 ] .

See also p. 123 of this report.

±1
See, for example, T. M. Williams, R. F . Barnes, J r . , J. W. Kuipers,
various references.

140

but some never, used in SYNTOL itself, he reports that both substantives and textual
expressions indicative of certain specific SYNTOL relations can be unambiguously identified. Contextual clues are used: for example, if the word "homme" occurs it is translated as n s e x e masculin" if "femme" also occurs, as "etre humain" if "animal11 i s alsomentioned, and as "suj et experimental" otherwise.
Melton and her associates at the Center for Documentation and Communication
Research, Western Reserve, have also been investigating machine processing of input
text with a view to the automatic selection and manipulation of clue words and relationships between them for information retrieval purposes. Their material consists ofabstracts from the metallurgical section of Chemical Abstracts. From sample abstracts,
a lexicon is developed which involves classification of words into those that are significant from a metallurgical point of view;those that name materials, compounds, environments; those denoting processes; those denoting characteristics of materials; prepositions; those which will not operate in the analysis of the text, and the like.
On the basis of analysis of a number of sentences from the sample text, rules for
combination and selection of specified words in specified relationships can be set up.
These rules are designed to identify sentence types which:(1)

Describe performance of a process on a material.

(2)

Discuss a material in terms of properties, components, form, or
environment.

(3)

Describe a process without reference to specific materials.

(4)

Discuss metallurgical properties without reference to specificmaterials.

(5)

Discuss two or more materials, properties or p r o c e s s e s .

(6)

Describe a causal relationship between two properties.

(7)

Give a comparison of materials.

(8)

Contain no words of interest in the system.

Computer programs to explore the possibilities for automatic analyses of thekind developed manually for the sample abstracts will be written with the objective offinding an effective compromise between mere word identification and total linguistic
analysis. Melton says:"If one considers this method of analysis from the point of view of the linguist,
he can immediately describe many grammatical constructions, which will
prevent the meaningful reduction of these sentences. It is not known at thistime how often such sentences will appear in the corpus of this investigation.
Nor is it known how adversely such failure would affect the retrieval of theinformation in these sentences. The answers to these questions will beavailable only after a large sample has been analyzed and put to an extensive
retrieval test. At its most successful the project will achieve an automatic
processing of metallurgical text which will permit retrieval of the type of
information which can be stated in its own terms with a tolerable amount of
inappropriate selections. Should this goal be unattainable, the project will
have generated a file of abstracts automatically searchable on the word level141

or somewhat beyond. For the benefit of other research, it will also haveproduced tapes of the true text of a large sample of natural-language abstracts and a lexicon containing all the words of a corpus of current
scientific literature. " }J •6. 7

Example of a Proposed Indexing-System Utilizing Related Research Techniques

In addition to the automatic assignment indexing and automatic classification
techniques for which experimental results have been reported, several other techniques
and programs have been proposed. One is the joint American Bar Association-IBM
research program (Eldridge and Dennis, 1963 [ 182]), for which discussion has beendeferred because of its proposed use of several of the research techniques covered
previously in this section. The experimental corpus will consist of the full text ofapproximately 5, 000 legal case reports taken chronologically from the Northeastern
Reporter. Approximately half of this material will be processed to obtain word frequency
counts. The frequencies will then be used to prepare for each different word an estimateof the skewness of its distribution in the collection. The investigators will then personallyinspect the word list as ordered by skewness to divide it into "non-informing" (Type I
words, or an exclusion list) and "informing11- (Type II words, or an inclusion list) at someappropriate cutting point. Then, for each document, a list will be prepared of its"informing11 (Type II) words, maintaining order within the document. For each pair ofsuch words, statistical association factors will be computed. Eldridge and Dennis
describe other aspects of their proposed technique, in part, as follows:"For each document in the body of 2, 500 c a s e s , a list will be prepared of its
Type II words, maintaining their original order within the document . . . For eachType II word an 'association factor 1 will be calculated for every other Type II wordwith which it appears in any one document by compiling the probability that Word A
would appear this close to Word B this number of tries over the entire file, if
the Type II words were distributed at random. (This amounts to borrowing Stiles 1
idea of the association factor, but implementing it with a numerical method whichtakes into account nearness of the words within the document as well as the factthat they both occur in the same document.) Since the factors are probabilities,
they will be numbers between zero and one . . . These numbers will be used toestimate the distances between words in index-word space.
"The next step is to construct from the information about distances between pairs ofwords an index-word space in which every word is at the correct (or approximately
correct) distance from every other word in the system with which it exhibits
association. The result of this operation can be visualized schematically as a sortof grid in which every word can be placed in its appropriate position by assigning
it a set of coordinates. "1/

Melton, et al 1963 [ 4 1 4 ] , [>p. 14-15.

142

'•Indexing of the remaining cases in the experiment will be performed by machinefrom full text, using the Type I list of discard words and the Type II list to prepare an analysis of the frequencies related to index-word space. Instead of
selecting specific words as indexing terms, concepts will be selected (statistically)
as volumes in index-word space* A rough physical analogy to this process would
be to toss pennies at the previously mentioned grid so that, for every Type II
word in the source document, a penny lands at its proper slot on the grid. Where
the pennies heap up in a pile, you have a concept. If"Searching will be carried out essentially by indexing a question presented
narratively, determining the concept volumes that represent the question, andsearching those volumes in document space for the relevant document numbers.
Since the 'edges' of the concept volumes are determined statistically, output canbe listed in order of probable relevance; as an option the question could beaccompanied by a request that 'at least 100 references be supplied', in which casethe concept boundaries would be adjusted to provide that number. " }J
It will thus be noted that the proposed indexing and search program begins on aderivative basis to establish for one-half the experimental material the significant words,
next combines word frequency with significant word distance data to derive probabilistic
association factors between words, then develops clusters, and finally indexes the items
in terms of the clusters rather than words so as to provide assignment rather thanextraction of index terms.
7.

PROBLEMS OF EVALUATION

We have noted, in the introduction to this report, that several fundamental andhighly controversial questions can be raised with respect to the feasibility and evaluationof any automatic indexing scheme and with respect to the evaluation of any indexing
systems whatsoever. Yet if automatic indexing procedures are to be based upon previous
human indexing or if their results are to be compared with human results, then thequestions of the quality, the reliability and the consistency of human indexing are crucial
ones indeed. Thus, Solomonoff warns:"The finding of exact languages for retrieval is also made l e s s likely, in viewof the fact that the categorizations of documents that are presented to the machineas a training sequence will not be performed altogether consistently by the humancataloger. " Zj
Montgomery and Swanson ask whether human indexers are in fact self-consistent andconsistent with each other, and they suggest:yEldridge and Dennis, 1963 [ 182], pp. 97-99.
Solomonoff,

1959 [ 5 6 2 ] , pp. 9-10.

143