Modifications to derivative indexing techniques that tend toward normalizations ofterminology and word usage, and increasingly sophisticated proposals for machine useof syntactic, semantic, and contextual clues hold out the promise of transition to moretruly fl subject fl indexing and to automatic assignment indexing systems.
4.

AUTOMATIC A6SlGN\tENT INDEXING TECHNIQUES

Answers to the question of whether indexing by machine is possible are actuallydependent in part on how the question of whether what can be achieved by machine is or
is not properly termed "indexing1' is answered. If "indexing11 is defined as being morethan the mere extraction of words from titles, abstracts, or text, then automatic
derivative indexing, even when augmented by various modifications, normalizations, andeditings, does not provide affirmative evidence. In the case of concept-oriented
definitions of indexing, the question becomes one of whether or not automatic assignment
indexing is possible. Experimental evidence suggesting that it is will be presented in thissection.
We should note first, however, that just as there are differences of opinion as towhat "indexing" means so there are similar differences, with respect to whether or not
it represents concepts rather than extracted words. There are also a number of conflicting definitions of what is meant by "indexing" in contradistinction to "classifying". For
some, the latter difference is related to questions of the number of labels or surrogatesassigned to a single item to represent its subject contents, ranging from the assignmentof a single subject category in a classification scheme involving mutually exclusiveclasses to the assignment of a number of terms or descriptor each standing for one of anumber of aspects of the subject. For our purposes, however, we shall regard both thec a s e of indexing with a number of descriptors and that of classifying to a single categoryor subject heading as being within the province of automatic assignment indexing, r e serving the term "automatic classification" for the case where the machine is used toestablish the classification or categorization scheme itself.
Actual experiments in automatic assignment indexing by Borko, Borko and Bernick,Maron, Sal ton, Stevens and Urban, Swanson, and Williams will be discussed briefly
below. These discussions are generally in chronological order with respect to firstreporting of results, except that the Salton-Lesk-Storm work reflects a somewhat different principle of assignment from the methods using clue word approaches and it istherefore described after these others have been discussed. Some of the similarities anddifferences between the various methods are then indicated. A brief final subsection
covers related assignment indexing proposals for which experimental data is not available
or has not as yet been reported in the literature.
4 . | Swanson and Later Work at Thompson Ramo-Wooldridge
Research on fully automatic indexing as well as on full text searching and retrieval
at the Ramo-Wooldridge Corporation has been reported as being under way at least as
early as the spring of 1958. }J As described elsewhere in this report, experiments insearch and retrieval based upon full natural language text had used as test items shortarticles in the field of nuclear physics. In additional experiments representing apreliminary "clue word" approach to possibilities for automatic indexing procedures,some of this same material was used.

yNational Science Foundation's CR&D rept. no. 2, [ 4 3 0 ] , p. 32.
91

In these additional experiments, 27 articles in the nuclear physics subject area were
included in a corpus of 100 articles, the remainder covering a variety of topics. F r e quency counts of word occurrences for the physics material were obtained and the 12 mostfrequent words that were judged to be discriminatory for the subject were selected. The
hypothesis was then tested, that if any document pertained to nuclear physics it would
contain at least two of these words. Retrieval was achieved for 25 of the 27 documents
and the two "irrelevant 11 documents also retrieved did include information at least peripherally related to the subject. It was thus evident that the retrieval effectiveness of
automatic recognition of nuclear physics subject material in the general collection wasconsiderably greater than the average effectiveness of retrieving responses to the highly
specific search questions in nuclear physics that had been used in the full text searchingexperiments (Swanson, 1961 [ 5 8 6 ] ) .
This second set of experiments provided a transition from the full text searching
work, which if it can be considered indexing at all is obviously derivative indexing, towork in the application of an automatic assignment indexing method to 1, 200 newspaper
clippings (Swanson, 1962 [ 584] , 1963 [ 580]). These were brief news items for whichmachine-readable texts in the form of punched paper tape were available. Thesaurusgroups of words likely to be associated with each of 20 to 24 subject headings were firstcompiled on the basis of human analysis of 1,000 or more representative items. Theseword groups were further screened so that no word appeared in more than one group andso that each word retained should be uniquely indicative of the particular subject
category. In the machine assignment procedure, subsequently, if a word occurs thatbelongs to a particular thesaurus group, the corresponding subject heading is assigned
to the item in which that word occurs.
Results achieved with this technique appear to be highly promising, at least for thistype of material. Swanson reports as follows:"Approximately 1,200 brief news items were classified into 20 nonhierarchicalsubject categories, both by a human and a machine procedure. Each item was
assigned on the average to about four categories. The results of the twoprocesses were compared. With the human process as a standard, the machinemissed only seven percent of the correct subject assignments and made a numberof irrelevant assignments equal to about 17 percent of the total. Nearly 40 percent of the automatic subject assignments judged finally to be correct were
missed by the human catalogers. "liWhile this accomplishment is actually due to the extensive human effort to compiling,organizing, and pruning of the uniquely indivative word l i s t s , it is pointed out that thisintellectual effort and the programming tasks need to be done only "once and for all". \L
It is further pointed out that garbles or misspellings in the input text do not appear toaffect the procedure, there being enough redundancy in the messages so that even if one or
two clue words are missed, others will be present.?.'"*Swanson, 1962 [ 5 8 4 ] , p. 468.

Z

J

Ibid, p. 469.

3/-

Swanson, 1963 [ 5 8 0 ] , p. 5.

92

Swanson and his TRW associates have further proposed extensions of the prespecified
unique clue-word technique. For example, it is suggested that machine processes ofcomparing words of titles, subtitles and chapter headings to lists of possible subject
heading can be extended in sophistication by machine lookups of synonym groups and ofcharacteristic subject-word associations. A/ Frequency weightings may be taken into
account, and similar measures of association and subject-indicativeness may be
developed for phrases as well as for individual words. — In general, however, theapparent success of this clue-word technique in tests to date should be considered in thelight of the special character of the items, their extreme brevity, and the high probability
that the fact-word incidence involved in news reporting is not typical of l e s s popular andl e s s factually oriented materials. ~JContinuing work along similar lines has been carried forward at Ramo-Wooldridge inthe "Word Correlation and Automatic Indexing Program" sponsored by the Council on
Library Resources (1959 C 490] and [ 4 9 l ] ) . Here, the objectives are to develop and apply
clue-word techniques to material that is much more representative of the scientific andtechnical literature. The thesaurus-groups, now called "indexonym" groups, are made up
of words and phrases selected by extensive human analysis as being significantly "usefulf or-retrieval-purposes".
New items would be processed in a word and phrase lookup operation, with each wordor phrase being initially assigned the identifier number codes of all groups to which it
belongs. However, unless a particular group-'s number is repeated several times within
the space of a few paragraphs, it is not used as the basis for the actual assignment of anindex tag. Provision would be made for calling human attention to items having a numberof words that are not deleted by processing against a "useless-for-retrieval purposes"
list, but that are not found in any of "accepted" groups. It is suggested that in this way it
should be possible to "ascribe measures of automatically recognizable 'newness 1 totechnical articles". ^7
4.2 Mar*bn's Automatic Indexing Experiments
By April of 1959, the reports of work at Thompson Ramo-Wooldridge on automatic
indexing and related problems submitted for the Current Research and Development in
Scientific Documentation series included reference to Maron and a "probabilistic model forthe assignment of index tags", as well as to Swanson's continuing projects. ±>ySwanson, 1962 [ 5 8 4 ] , p. 469.
Swanson, 1963 [ 580] , pp. 1-2.
See also Mooers, 1963 [ 4 2 4 ] .

±1
Thompson Ramo Wooldridge, 1959 [491],. p. 2A.

1'
National Science Foundation's CR&D report No. 5 [ 4 3 0 ] , p. 34.

93

In addition to his work on probabilistic indexing with emphasis on relevanceweightings for index tags manually assigned, Maron has actively explored automatic
assignment indexing chniques. The approach i s also probabilistic, with emphasis on
the statistics of association between content-indicative clue words %and subject headings
manually assigned to sample documents. The experimental corpus consisted of a groupof abstracts in the field of computer technology indexed to 32 subject categories designed
for the purposes of these investigations.
Common words such as articles and prepositions were first excluded. Next, words
occurring l e s s than three times were purged and words such as "data" and "computer"were also rejected because they occur so frequently in this literature. Approximately
1,000 words remained after these purging operations. After sorting the source documents to their most appropriate subject categories, statistical frequencies wereobtained for the co-occurrences of the candidate clue-words with the categories and theresulting listings were manually examined to determine which words peaked in aparticular category. Eventually, 90 such words were selected.
The occurrence of one or more of the 90 clue-words in the text of new documents wasthen used to predict the subject category to which the new item should belong. U
Tests
were run with two groups of documents, one consisting of the source items from which
the statistical frequency and word list data had been obtained, and the second groupconsisting of 145 genuinely new items. For the latter group, twenty documents contained
no clue words whatever and forty items had only one. For the remaining 85 items having
two or more clue words, the results of the computer assignment program were predictions of the correct category in 44, or 51.8 percent, of the cases.zJ Results using thesource documents were significantly better, as expected, with 84. 6 percent accuracy of
category prediction for 247 items. Results were also related to the number of clue words
that occurred in the test items, with a prediction accuracy of only 48. 7 percent for itemswith a single clue word rising to 100 percent probability of correct assignment if six ormore clue words occurred.
Trachtenberg (1963 [608]) has also considered a probabilistic approach to automatic
indexing and categorization of documents, similar to that of Maron. He suggests theinvestigation of two information theoretic measures with reference to determination ofwhich of various possible clue words are significantly discriminating with respect to thedifferent categories. He further suggests experiments using 90 clue words and the
corpus used by both Maron and Borko, but no actual results have as yet been reported.
4. 3 Automatic Indexing Investigations of Borko and Bernick
At the System Development Corporation, the work of Borko (I960 [73]), and of
Borko and Bernick (1962 [ 7 7 ] , 1963 [ 7 8 ] , 1964 [ 7 9 ] ) in the area of automatic indexing
has involved both automatic assignment indexing and automatic classification techniques.
They have not only reported actual indexing results but have provided data for the intercomparison of their techniques with the experiments of Maron for the same source
material.

1/
Note that the word itself is not necessarily used as an index tag or label, as is thecase for derivative indexing using an inclusion list approach. This is an importantdistinction.
Maron, 1961 [ 3 9 5 ] , p. 257.
94

The original Borko approach was based on the principles of factor analysis as these
had been developed for the analysis of multivariate date, especially in the field of
psychology^ Borko 1 s first experiments were directed to a corpus consisting of 618
abstracts in the field of psychology, amounting to approximately 50, 000 words of total
text and 6, 800 different words. These words were sorted by computer program into anorder reflecting their respective frequencies of occurrence. For the approximately 200
words that occurred twenty or more times in this corpus, the investigator himself
selected 90 words to serve as index (or, better, index-clue) terms. A matrix was thendeveloped for the frequencies of co-occurrence of these words and the documents in which
they appeared. From this, a 90 x 90 correlation matrix was computed as follows:M

To compute the correlation coefficient . . . we used the following formula
.

r

**

NSxy - (Sx) (Sy)/ [ N S x 2 - (Ex) 2 ] [ N S y 2 - (Sy) 2 ]Where N is equal to the number of documents (618) and x and y are the terms being
correlated. "-1J
The term-correlation matrix was then factor analyzed and the first ten eigenvectors
were selected as factors to be rotated and interpreted. Borko emphasizes that:"The interpretation must be made by the investigator and is based upon his knowledgeof the analytic procedures and the subject matter. There i s , therefore, a degree ofsubjectivity in the names selected for each factor. These names may be regarded
as hypotheses about the factor meaning. M 2 /Following the derivation of these "classification categories" by means of the factor
analysis technique, new items may be assigned to the categories on the basis of words
occurring in their texts (abstracts) in accordance with the following procedural steps:"1. Each document, in machine readable form, is analyzed by the computer.
A list of the index terms and their frequencies of occurrence in each document
is recorded.
"2. The category or categories containing the index term is assigned a value equal
to the product of the number of occurrences of the word in the abstract and thenormalized factor loading of the word in the category. If more than one index term
appears in a category, the products are summed.
"3. After each index term has been considered, the category having the highest
numerical value is selected."yBorko, 1961 [ 7 3 ] , p. 283.
Ibid, pp. 285-286.

1'
Borko and Be m i ck, 1962 [ 7 7 ] , pp. 7-8.

95

The choice of 90 clue words in Borko's work with abstracts in the field of psychological literature was apparently dictated by a matrix size which would be convenient
for computer manipulation. U However, it happened to coincide with the number of cluewords used by Maron in his experiments. Advantage was taken of this coincidence toobtain comparative data on the performance of the two assignment-indexing techniques
as applied to the same material. The 260 computer literature abstracts used by Maronf
as source documents were processed to derive a correlation matrix for Maron f s 90
manually selected words, which was then factor analyzed. Several sets of factors wereextracted, rotated, and the results studied, with a final selection of 21 categories .
Since these automatically derived categories did not coincide with Maron's original
32, it was necessary to analyze manually the total group of 405 abstracts (260 "source 11
and 145 "test" items) and assign them to the new categories, then to study the documents
falling into each factor-analytically derived category to determine which of Maron 1 s 90
clue words were category-indicative, and finally to substitute these words in the Bayesian
equation used by Maron so as to predict which of these classification categories his
probabilistic method should obtain.
The same two sets of 260 "source" and 145 "new" abstracts used by Maron were thensubmitted to the computer assignment program which compares the clue words of a newitem with the numeric values of the predictor words for each factor category, then computes the score for each item in all categories, and assigns the category with the highest
score to the item. For the source items, Borko and Bernick's results showed 63.4
percent correctly classified, by comparison with the 84. 6 percent correctness scoreoriginally obtained for them in Maron f s experiments. For the new items the factor
analysis method scored 48.9 percent correct assignment by comparison with Maron's
original 51.8 percent. zJ The later investigators therefore concede that the performanceof Maron's technique was somewhat superior for the same items using the clue words
originally selected by Maron.
Further experimentation was then carried out (Borko and Bernick, 1963 [ 78]) using
word frequency data for the selection of a new set of 90 clue words and a classification
scheme for 21 categories was again automatically derived. The 405 abstracts were again
manually classified to these machine-de rived categories by five subject-matter
specialists and the two investigators. Comparative data were then obtained for both theMaron assignment formula and the modified classification system assignments in termsof agreement with the manual assignments.
For the source items, the percentage of machine assignments agreeing with thosemade by people was 62. 7 when the Bayesian probability formula used by Maron wasapplied and 61)2 for the factojc^analysis score system. For the new items, thecorresponding correct perce^tafllPW^jji^S?. 9 and 55.9. Additional data compared theeffects of using the original ikaron Worci^ and the frequency-based word set (Borko's
words) for the same probability formula assignment method. While there was an overlap
of approximately 50 percent between Maron's words and Borko's words, the findings
indicated that:u
Now increased to 150 x 150.
Borko and Bernick, 1962 [ 7 2 ] , pp. 9-10.

96

" . . . The index words selected by Mar on are decidedly specific to the documentsfrom which they were derived and are of l e s s generality than the frequency based
terms. The Bayesian formula coupled with the Maron words c o r r e c t l y predicted
the classification of 79. 6% of the documents i n G r o u p l [ 'source i t e m s ' ] but only
45. 5% of the documents in Group II [ 'test i t e m s ' ] . The coupling of the Bayesian
formula with the Borko words resulted in a slight d e c r e a s e in the percentage ofGroup I documents whose classification was c o r r e c t l y predicted (62. 7%) but in.
creased the percentage of correct prediction for Group II documents to 58. 0%.!l —Other findings from the later experiments indicated that despite the differences inthe two w o r d - s e t s , the factor categories derived from them w e r e very s i m i l a r . It wasalso found that, at least for the source items (Group I), the two machine techniques andthe manual p r o c e s s classified 56. 1 percent of the items into the same c a t e g o r i e s . Itshould be noted, however, that in the case of the automatic assignment methods: "Elevendocuments contained no clue words and could not be automatically classified by either
system. f l
4. 4 Williams' Discriminant Analysis Method
The work of Williams in automatic assignment indexing, reported in the fall of
1963 [ 6 4 2 ] , has also involved tests on abstracts of the computer literature, directlycomparable to but not n e c e s s a r i l y identical with those used by Maron and by Borko andBernick. This work at IBM's F e d e r a l Systems Division, Bethesda is based in part onearlier work by Meadow which involved computer studies of matching functions for
document word lists and category word lists for test items drawn from such fields as
psychology, law, computer a b s t r a c t s , and news i t e m s . 2/ What has subsequently beendeveloped is t e r m e d a " d i s c r i m i n a n t " method which begins with h i e r a r c h i c a l classification structure of p r e - e s t a b l i s h e d subject categories and with a small set of sample
documents previously indexed by people into these categories. Frequency counts of words
in each of the sample documents lead to computations, for each category, of the theoretically probable frequencies of its most statistically significant words. For new items,
observed word frequencies a r e compared with the theoretical word-category associations
and a relevance value is computed for the item in terms of each category.
The corpus selected for experimentation consisted of 400 items from "Computer
Abstracts on Cards". 2/ These had previously been indexed using a classification
structure of 15 major c a t e g o r i e s , each of which is divided in turn into 10 subcategories.
The experimental sample, however, was so selected as to provide exactly 15 "source"
items and 5 "new" items for each of 5 subdivisions of 4 of these major categories.

yBorko and Bernick, 1963 [ 7 8 ] , p. 23.

U
Ibid, p . 11.
3/
Williams, 1963 [ 6 4 2 ] , cites H. R. Meadow, "Statistical Analysis and Classificationof Documents", IRAD 'Ssk No. 0353, FSD IBM, Rockville, Maryland, 1962, but
this is apparently a company-confidential document, containing p r o p r i e t a r y information. Meadow gave an informal r e p o r t on her work at the Computing Center
seminars, University of Maryland, in March of 1963.
Available on a subscription basis from Cambridge Communications Corporation,
Cambridge, Mass.
97

Discriminant coefficients were.then computed at both the major and minor levels for
all words occurring in the sample items falling into one of the 20 groups in accordancewith the formula:"The discriminant coefficient i s :n( P - - P .)2
= S
iJ
ij;
K

\{

J

iJ

Where:The relative frequency of the ith wordin the jth category.
P.. = f.. / S

and

P.. =

-

ij

n

r

The mean relative frequency per
category of the ith word, *J

P..

These coefficients are used both to set up threshold values to determine which words
should be used in the assignment formulas and to assign weighting factors to the words
themselves.
The results of the experiments to date are based on 83 items from the "reference
set" which were not used as source items. For 63 items, 78;percent were correctlyclassified at the level of a single major category (e.g.,^Pr<igxarnniing",'Hardware
Design") and also correctly classified at a single subcategory level, ( e . g . , "Programming Languages", "Semiconductor Devices"). The 20 remaining items were classified
to one major category with an accuracy of 95 percent and to two minor level subdivisions
with accuracies of 60 percent and 75 percent. Additional investigations were made on
the effects of using a discrimination threshold to eliminate insignificant words fromconsideration and on the use of weighting factors in the assignment calculations.
4 . 5 SADSACT
Stevens and Urban at the National Bureau of Standards (1963 [ 569, 570]) have also
explored an automatic indexing technique that u s e s , as in the experiments of Williams,
a teaching sample or reference set of previously indexed items to form patterns of word
and index-term assignment associations. However, there are much l e s s formal requirements for computing correlation coefficients and no consideration is required of either
1/

Williams 1963 [ 6 4 2 ] , p. 163.

98

the theoretical probabilities of word occurrence by category or of discrimination coefficients and t h r e s h o l d s . Instead, the technique involves ad hoc s t a t i s t i c a l associations
between the words occurring in the title and in the a b s t r a c t of a sample item and thed e s c r i p t o r s previously assigned to that item. A m a s t e r selection-word vocabulary isthus built up where each word is listed in t e r m s of the frequencies of its c o - o c c u r r e n c e
with each of the d e s c r i p t o r s with which it has c o - o c c u r r e d , r e g a r d l e s s of whether or not
such p r i o r associations a r e either revelant or significant. No attempt has as yet been
made to "purge" the resulting association l i s t s . Instead, reliance is placed on thepatterns of multiple word usage and of redundancy of words used in titles and cited titlesof new items to minimize the effects of i r r e l e v a n t or accidental p r i o r w o r d - d e s c r i p t o r
associations and to enhance the significant ones.
The SADSACT method (for "Self Assigned D e s c r i p t o r s from Self and Cited Titles")proceeds with the assumption, which it s h a r e s with the arguments for citation indexing
previously discussed, that the l i t e r a t u r e references cited by an author a r e indicative of
the subject content or contents of his paper. L' F o r the automatic indexing of new i t e m s ,their titles and the titles of up to ten bibliographic references cited a r e keystroked, converted to punched c a r d s , and fed to the computer. This input m a t e r i a l is run against them a s t e r vocabulary to obtain for each input word which matches a vocabulary word a" d e s c r i p t o r - s e l e c t i o n s c o r e " for each of the d e s c r i p t o r s previously associated with thatword. These s c o r e s a r e summed up for all words and at an appropriate cutting level
those d e s c r i p t o r s having the highest s c o r e s a r e assigned to the new item.
P r e l i m i n a r y r e s u l t s based on the titles and cited titles of items that w e r e " s o u r c e
i t e m s " in the sense that their titles and a b s t r a c t s had been used in the teaching sample
w e r e r e p o r t e d at the NATO Advanced Study Institute on Automatic Document Analysis
held in Venice in July, 1963. F o r 30 items drawn from such subject fields as computer
technology, information selection and r e t r i e v a l , mathematical logic, pattern recognition,and operations r e s e a r c h , all of which had previously been indexed by ASTIA personnel in
I960, the machine assigned 64. 8 percent of the d e s c r i p t o r s previously assigned. Subsequent t e s t s on genuinely new i t e m s , however, resulted in a drop to only 48. 2 percent"hit" a c c u r a c y .
These "new" item r e s u l t s w e r e also evaluated byhavings aevexal representative
u s e r s of the collection analyze the test items and assign d e s c r i p t o r s to them from a listof the d e s c r i p t o r s available to the machine. The extent to which the d e s c r i p t o r s assigned
by machine w e r e also independently chosen by one or m o r e of these indexers was thenchecked. In general, the fewer d e s c r i p t o r s assigned by the machine, the better was thehuman agreement, ranging from 4 7 . 4 percent overall in the case where the machine had
assigned twelve d e s c r i p t o r s to each item to 76% agreement where the machine assigned
only one. In p a r t i c u l a r , for ten items which were analyzed by five different indexers,
the chances that one or m o r e would also select the m a c h i n e ' s first choice (highest scoring)d e s c r i p t o r averaged 90 percent.
4. 6 Assignment Indexing from Citation Data
Certain phases in the p r o g r a m of investigation of information selection and r e t r i e v a l
problems at the H a r v a r d Computation Laboratory have been mentioned previously. The
work of Storm and of Lesk and Storm on the use of f i r s t - n o u n - o c c u r r e n c e s as selection
clues for both automatic indexing and abstracting was discussed in connection with techniques for improved derivative indexing. The studies on citation indexing have included,
as noted, experiments to assign indexing t e r m s to a new document by finding the indexingU

If n e c e s s a r y or d e s i r a b l e , however, a b s t r a c t s or portions of text can be used inaddition to or in lieu of the cited t i t l e s .
99terms previously assigned to the five most "related" documents, where "relatedness" isa function of the similarity in citation patterns as between the new document and items already in the collection. The results of such index term assignments are reported as
identical to those made by human judgment approximately 50 percent of the time. 1 /More specifically, in an experiment using documents drawn from a small collectionin the fields of mathematical linguistics and machine translation, a new item was pompared in terms of its citation data with the citation similarity data previously determinedfor earlier documents, and the set of five related documents was selected using themagnitude of the row similarity coefficients obtained from links of length one and two.
All index terms occurring at least twice in the set of terms assigned to these related
items were then assigned to tne new items. For the ten "typical" new item c a s e s , forwhich comparative data are shown, the citation data assignment method correctly.
assigned, on average, 47. 6 percent of the terms assigned manually to the same items. —A slightly more sophisticated indexing term assignment formula, described by Lesk,
was applied to additional test c a s e s , but "failed to raise accuracy above fifty percent"..!'For five typical new c a s e s , the improved method correctly assigned 11 of the 20 terms
manually assigned to these items, or an average accuracy of 55. 5 percent ..£'
4. 7 Similarities and Distinctions among Assignment Indexing Experiments.
In Table 2 some of the key points of the various automatic assignment indexing
experiments we have discussed above are summarized. Certain similarities, distinctions,and differences are to be noted. Borko and Bernick use the same corpus as did Maronand also re-apply Maron's formula to a different clue-word set for the same material.
Williams u s e s material similar to the Maron-Borko computer corpus. The SADSACT
tests also use some items that might be included in the Maron-Borko and Williams
corpora. The Swanson experiments with newspaper clippings represent a quite different
class of material consisting of brief, terse, factual m e s s a g e s .
1/Lesk, 1963 [ 3 5 7 ] , p. V-8.
Salton, 1962 [ 5 2 0 ] , p. Ill-41, Table 9.
Lesk 1963 [ 3 5 7 ] , p . V - 7 .

1/
Ibid, p. V-8, Table 3.

100

.3
2 • !

s s 0)
*8

M

s; S8

O
CO

J!
4->

^
^

o

$ 0) cr)
tfP-

0

p

<D

M
y

CJ

I Ii 8*

°

0«'H
GO

«

II *

!
o *»

o
u
*» a) _.s s O h S

(0

o

3g

M

a .8
P«fM

* • " • * • • CO

I*aj

(1)

43

^ u
O

o

CO

" • •

M g
o

o

©

in

2.2 J? ?

*» g n

it*
© H

CO
CO

*H

CO

4->

C
fl)

ft

AH

U

U

*> t g
ft

0)

0

n

1-3

oo
o „s

£>

3-2 to3 ^2 •O

tJ +25 U
n
.8 to

<D

CO N O» d CM0•l ~O II*3IS

A 3 '•

U oo
H
O -H

M
H

•o

O r3

a; a)

u
O

o

s
a) oo O

o g ?

O

U
0

a)

•SI

O
fi

h22

CO <0

o
o
o
o"
CM

5
O

1 01

.alii

<D
CO

O
>

2(J
u 5 3

o
o« o

5

T3
h

*d 2 *
o

(N3

• o

* ,§*

CO
•H

Si m 00coC

in ^* >o|

rd

tO

•

CO s O^«"I

i g 6 •U* «» ^?a
Co o
O *
o ^
<D £

Si

g 8

o <»
o d

10

4

•2
o ^

CD

CM

«H^ J
rr\

^

0) 4$ (t)

u^

cd CO

5O i °

co

"gTB

"^ d

co X

„»

2a
g
rt

CO

Sg

aJ

.2

**>

3s« " n • s *
CD

»

Q)

W

CO

a) p
> o
**}
<u

O

,8 4)

a _? e« s* *s « S"J nd A d $ S -M

Jijg "G 0) flJ O °5P8 •2 ^

ccj
co

"Sfl

73 a

iitt

d

n) ^

flj

,d
» •3
a) co

Ol g H

X

is

1

1

So a

Mi CO

g « ° Ctf M

o
CO

cu

CO

102

w

d

? «2j .S 0 Si I*- 6

vD

ml

^ r-l

g1 S*$1 8 *Z*a•H TO O

TO
CD

CD
h

0
.2°
TO S hTO O(ti TO

H

« { d
+3

«$

S

O

o d © ow «J u v
i

**S
S
i

o

A

A
I
X

U
O

U O
a) -

U

I

h

% o *dIs M 8
O -^

TO

« 5

• a 3 js .s .ga
C5

K o 5 S S 5 o3 o

O TO *

0

fir

-M ^

.H

3 ,1 * 3 |d 'H wQ< O a) o 'D

U U
04**
o M

U TO £

*tf

pj 0
i
p

(D

TOtf>

eg *rt1•uI> s TO

TO Q<D - HM

CD« .2 H

3

O* 1
TO

«*

§

S"2

O
a •CDM 4->

O

rt

0

P CO .5
a)
TO

>

n

CO

£

1
103

* «

2 g z • a g*g «3 TO 4->

l-s'S

TO

8
o-2
°
CO g
0) * 2 « o a)
3 d S S 0) o

-<

o «J

#

n! -H

t.

<D

O
O

^

•h2 1

CO

<u

o

<D

U
CD

3

£

O

»2

&a fl.suUO^

O . H J S

TOTO

None of the experiments has so far encompassed testing of anything but very small
test item samples and the dangers of extrapolating from so small and so specialized
bodies of data should be clearly recognized. Mooers identifies these dangers in terms of"The Silent Postulate:That(real people)(real documents)(real jobs to do)can somehow

be eliminated from the experimental study, and that (role-playing people)(substitute documents)(imaginery jobs)
can be substituted and still give valid experimental results. 1 1 —In most of the experiments in automatic indexing conducted to date, indexing andclassification schedules have been especially designed, or evaluations made, specificallyfor the purposes of these tests. Williams, however, s t r e s s e s the point that the material
used in his experiments had been "classified by professional indexers for the purposes ofactual retrieval. -• 2 / A similar claim can be made for SADSACT, as noted by Mooers. 3/
Swans on f s news item work also obviously relates to real items and implies a real job
to be done, but is directed, as noted, to a class of material not generally comparable tothat found in documentation operations on scientific and technical literature.
In contrast with the treatment of each document as a self-contained entity without
reference to any other documents, as is the case for derivative indexing, all of theautomatic assignment indexing experiments, by virtue of the fact that they are assignment techniques, do to some extent embody the effects of a consensus of a particularcollection, or a consensus of prior indexing, or a consensus of human subject content
analysis applied to sample documents, or some combination of these effects. The SADSACT method, in addition, wherever cited titles are available for new items, takes
advantage of terminology other than the author's own as a source of clue words. Other
proposed methods of assignment indexing, such as the use by Salton, Lesk, and Storm ofcitation-pattern similarity data, would carry the latter principle even further.
1/
Mooers, 1963 [424], p. 5.

I!
Williams, 1963 [ 6 4 2 ] , p. 162.

yIbid, p. 5.

104

4. 8

Other Assignment Indexing Proposals

A few additional automatic assignment indexing proposals are under development.
Examples for which experimental data i s not as yet generally available include, forexample, work at EURATOM, some preliminary experiments at Chemical Abstracts
Service, work at General Electric, Bethesda, the proposed "Multilindex" system ofInformation Systems, Inc. , investigations by Slamecka and Zunde, and a special purpose
development project at Goodyear Aerospace.
Meyer-Uhlenried and Lustig report for the EURATOM developments as follows:
11. . . Procedures are being developed which allow based upon given keyword
lists first for abstracts: (a) to assign significant keywords and (b) based
upon hierarchically organized keyword lists, to assign the documents inquestion to specific subject fields.
"Experiments were made at first on narrow fields with so-called m i c r o thesauri, they showed encouraging results when automatic and manual assignment were compared. Positive results depend of course on the quality of theabstracts and the significance of the words employed in them. It remains tosee how far this favorable prognosis is confirmed by keyword collections ofmore complex contents." 1/Friedman and Dyson (1961 [203]) have reported on manual experiments designed torelate words occurring in a sample of abstracts from a particular section of Chemical
Abstracts to the title or heading for that section. Significant words in these abstracts
were counted and the number of occurrences as well as the number of different abstracts
in which they appeared were determined, with a rank order listing as a result. It
appeared, from inspection, that it should be feasible to develop, for each CA section, a
relatively small vocabulary of words that would be descriptive, and indicative of, thesubject matter contained in it. They conclude: n In our opinion, the results were significant, the small vocabulary of words did select a large percentage of the abstracts in thesection it was based on. "A project at Information Systems Operations, General Electric, on possibilities
for automatic indexing and abstracting of text has been reported in the November 1962
issue of Current Research and Development.j/The META project (Methods of Extracting
Text Automatically) is said to be concerned with the use of statistical, linguistic, andsemantic criteria for analysis and selection of significant words and significant sentencesfrom text. Computer programs are being developed in modular fashion for the GE-225
computer.

yMeyer-Uhlenried and Lustig, 1963 [ 4 1 7 ] , p.229.

I!
Friedman and Dyson, 1961 [ 2 0 3 ] , p. 10.

II
National Science Foundation's CR&D report, No. 11 [ 4 3 0 ] , p.. 97.

105

The proposed "Multilindex" system is also based on micro-thesauri or small
vocabularies designed, by human analysis, for clue-indications to a relatively narrow
subject field, together with potential syntactic-semantic role indications built into thedictionary, again by extensive human analysis, following the approaches previously taken
by A. L. (Lukjanow) Loewenthal in her suggestions for solutions to problems of mechanized translation. An unpublished proposal-type brochure describing the system wasavailable as of December 1 9 6 3 . U As of that date, also, demonstration printouts wereavailable from an IBM 1401 Fortran program, illustrating an index compiled fromabstract-text input and a 1, 200-word dictionary for documents in the field of space antenna tracking radar. ?J A repetoire of 350 ''concepts 11 or indexing terms was involved,
with an average of 10 assigned to 22 test documents, many of these assigned terms being
identical to words occurring in either the title or the text of the abstract of the item.
Slamecka and Zunde have investigated the extent to which the "notations-of-content 11
in the system developed by Documentation, Inc. for NASA's STAR might be derived bymachine techniques from the text of the abstracts with enough normalization-standardization via inclusion dictionary lookup to qualify as an assignment indexing technique.
These workers claim:"This preliminary investigation indicates the possibility of using the computer
to index documents adequately for machine retrieval by matching their abstracts
against an authoritative subject-heading authority . . . The inconsistency inherent in
human indexing can be eliminated as the number of terms derived from any one
abstract will always be the same. The abstract and its automatically derived set ofindex terms will always be equivalent. . .
ti3/

A final example of other approaches to automatic assignment indexing research, notyet reported in the open literature, is an NIH sponsored project at Goodyear Aerospace, in
cooperation with the Universities of Minnesota and Rochester and Western Reserve
University, looking toward an automatic classification procedure based on word coocurrences for a set consiting of 100 four-to-five page documents in the field of diabetes
literature. Programs for statistical analyses of the full text of these documents, all ofwhich have previously been processed for the manual W. R. U. "telegraphic" abstracting
system, are being developed. 1 /5.

AUTOMATIC CLASSIFICATION AND CATEGORIZATIONIn all the experimental work, to date, that has been directed toward the use ofcomputers and other machine-like techniques for the automatic indexing of documents, ay"Description of MULTILINDEX. A mechanized system for indexing documents,
storing information, retrieving information", P. S. Shane, Dec. 4, 1963, Information Systems, Inc. , 7720 Wisconsin Avenue, Bethesda, Maryland.

yPrivate communications, A. L. Loewenthal and P. S. Shane, Dec. 11, 1963.

ySlamecka and Zunde, 1963, [ 5 6 l ] , pp. 139-140.

±1
E. Tuttle, private communication, Oct. 30, 1963.

106