ISR-9
August, 1965

J-l

I. A PROGRESS REPORT ON SMART
G. Salton

1.

Introduction
The SMART project was initiated in the fall of 1962 with the ob-jective of designing, and implementing on a computer, a fully automatic
document retrieval system, capable of processing documents and searchrequests available in English, and of retrieving those documents most nearly
similar to the corresponding search requests. The system was to be used
not only for retrieval purposes, but also as a means for evaluating theeffectiveness of the search and analysis techniques*The emphasis on fully-automatic procedures, resulting in the design of a system in which natural language texts are processed without
any prior manual analysis, was motivated primarily not by certain crisis
situations in the information field, or by making appeal to the so-called"information explosion."

Rather, it was recognized that, in the long run,the manual indexing of every information item to be processed could not be
considered a practical alternative to automatic operations, in part because
the increasingly large numbers of required subject experts could not be
expected to become available when needed.
The other main emphasis, on systems evaluation, stems from thefact that very little is known about the effectiveness of automatic retrieval systems. It therefore becomes essential to investigate theappropriateness of a variety of evaluation measures, and to apply these1-2
measures in experimental situations before operational systems wouldactually be designed and implemented0
The SMART system which eventually emerged can be operated on theI EM 709J4 computer, both as an experimental automatic retrieval system, andas a testing device for determining performance characteristics of a large
variety of automatic analysis and search techniques. The system ischaracterized by the following properties:(a)

it is designed for eventual operation in a time-sharing
environment, in such a way that a multiplicity of users
are given simultaneous access to the files;(b) the search function can be undertaken iteratively underuser control, by performing several partial searches toapproach the desired subject area, rather than a single
one-shot process;(c) several different analysis and search procedures are incorporated in the system, including in particular anumber of stored intelectual aids for vocabulary normalization;(d) the computation of sophisticated correlation measures
between documents and search requests makes it possible
to present ranked document output in answer to thesearch requests in decreasing order of the correlations
between documents and requests;(e) evaluation procedures are available which are specificallydesigned to be used with the type of automatic retrieval
system that furnishes ranked document output.
An initial version of the SMART system became operational on an
IBM 709U during the summer of 1961ie

In the next few paragraphs the research1-3
undertaken with this system is summarized briefly, and future plans are
outlined in Part 3*

2.

The SMART Project (1961-1965)
The general characteristics of the SMART system have been described
1 2

in some detail in a number of previous publications, 5

A set of diction-aries was initially prepared for the computer literature, and the firstexperiments were performed with two collections of document abstracts eachcomprising about 1|00 abstracts of documents in the computer area*

Tests

were conducted to determine the effectiveness of a variety of manually constructed intellectual aids,, such as synonym dictionaries, hierarchicalsubject arrangements, dictionaries of concept groups ("phrases'1), syntactic
analysis procedures, and so on„
The following design features of the SMART system appear to be noveland may be of particular interest:(a) the synonym dictionary (thesaurus) is stored in tree
form, and an efficient look-up process allows very
rapid dictionary consuLtation (of the order of 2 or 3
seconds for texts of several hundred words)y } '(b) a sophisticated suffix cut-off process, incorporatinga large number of English morphological rules operates
in conjunction with the dictionary process to produce
the correct word stems for all original text entries;(c) a complete set of list-processing methods has been
progi^ammed (largely in machine language for reasonsof program efficiency) to permit a. variety of treef)

tracing options using the concept hierarchy^ 97

i-a(d)word grouping procedures permit the construction ofso-called " s t a t i s t i c a l phrases/ 1 consisting of p a i r s89
or triples of concepts which appear to be related; 3(e) a tree matching procedure, coupled with an automaticsyntactic analysis process, permits the recognitionof a large variety of semantically equivalent, but
syntactically quite different constructionsj'9(f) the system operations are organized in such a waythat a programmed supervisory system is used to callupon the proper subroutines as required at any given
instant; this makes it possible to update the system
at any time by deleting obsolete routines and adding
new oneSo•

The programmed system is thus

based on the manipulation of avariety of different data structures, including linear word strings, tree
and graph structures, and reasonably complex vector and matrix operations0
The programs themselves are largely written in machine language (FAP),
since no existing programming system could accomodate the variety of required
data structureSo

9

Work in. systems evaluation has proceeded along several lines of re~searchu

New recall-like and precision-like evaluation measures have been

derived, which are especially adapted to the evaluation of systems producing
ranked document output as a result of the retrieval operations.

Thesemeasures have then been applied to the evaluation of automatic search andanalysis techniques, using the abstract collections previously referred to*Results obtained with these relatively small collections indicate that synonymrecognition is useful in improving retrieval performance; that hierarchical1-5
expansions are often effective in broadening or narrowing the coverage ofsearch r e q u e s t s ; t h a t phrase procedures using combinations of concepts are
far more powerful than procedures based on individual concepts; that weighted
concepts are more discriminating than concepts r e s t r i c t e d to weights of 0 or
1; t h a t analysis methods which use complete document a b s t r a c t s are more ef19 20
fective than those based on document t i t l e s only, and so on0" 'Procedures have also been implemented t o simulate the type ofi t e r a t i v e search process, using several different search procedures, which
may become common in a real-time system where customers obtain direct accessto the equipment*,

213 22Preliminary r e s u l t s indicate t h a t in most cases, ani t e r a t i v e search process based on user feedback i s much more useful than, .
.
19,21,23,2k
one-shot procedures« 9 * 'The f i r s t operating version of the SMART system suffers from avariety of size r e s t r i c t i o n s , notably concerning the t o t a l number of documents which can be processed (less than £00), the length of each document( l e s s than 1000 words), the number of concepts included in the synonym
dictionary (less than 1000), and so on.

These r e s t r i c t i o n s are a r e s u l t ofthe o r i g i n a l program organization which i s based on i n t e r n a l operations(with d i c t i o n a r i e s and f u l l documents stored e n t i r e l y in core storage) togain a considerable speed advantage.
In order to permit the SMART system to serve also for the manipul a t i o n of document c o l l e c t i o n s of more r e a l i s t i c s i z e , an "extended" SMART
system has been designed during the f i r s t half of 1965, as described in
Sections I I to XVII of Report ISR-9*

This new system should permit operations

under more l i f e - l i k e conditions, and should make i t possible also to apply

1-6
the e v a l u a t i o n p r o c e d u r e s t o c o l l e c t i o n s of p r a c t i c a l i n t e r e s t such a s theNASA, DDC, and MEDLARS systems„
1 ft

The new system can accomodate up to 2(about 262,000) documents;

the number of c o n c e p t s i s e f f e c t i v e l y u n r e s t r i c t e d (up t o 2of the h i e r a r c h y i s u n r e s t r i c t e di ft) ; the s i z e(except t h a t the branching r a t i o of eachi n d i v i d u a l node i s assumed n o t l a r g e r than 2 0 ) ; the l e n g t h of each document i s a l s o u n r e s t r i c t e d for a l l p r a c t i c a l purposes (a l i m i t e x i s t s only
because of the l i m i t a t i o n on t h e t o t a l number of concepts which may begenerated)oIt, may be expected t h a t t h i s extended SMART system w i l l proveu s e f u l immediately i n extending t h e r e t r i e v a l experiments t o new s e a r c hmethods and t o more i n t e r e s t i n g o p e r a t i n g environment s .

3«

FutureProjects

The extended system makes i t convenient for t h e f i r s t time t o usea v a r i e t y of s t a t i s t i c a l word a s s o c i a t i o n methods f o r t h e a n a l y s i s andr e t r i e v a l of i n f o r m a t i o n c

These s t a t i s t i c a l p r o c e d u r e s a r e based on t e r m -term, and document-document c o r r e l a t i o n m a t r i c e s which could n o t , becauseof t h e i r l a r g e s i z e , be processed in the o r i g i n a l syfetem©
The following p r o j e c t s a r e under c o n s i d e r a t i o n i n connection w i t h
the extended SMART system:(a)experiments using l a r g e r document c o l l e c t i o n s and a l s odifferent

subject f i e l d s(arrangements have been madei n t h i s connection w i t h the A s l i b - C r a n f i e l d p r o j e c tforthe t r a n s f e r of a c o l l e c t i o n of 1200 hand-indexed document a b s t r a c t s in a e r o n a u t i c a l e n g i n e e r i n g t o g e t h e r w i t h
the n e c e s s a r y s u b j e c t d i c t i o n a r i e s ; experiments w i t h

1-7
this collection will make it possible to compare
the efficacy in retrieval of the manually assigned
index terms with that of the automatic text manipulations incorporated into SMART);(b) experiments using new processing methods, including
statistical term and document associations, bibliographic coupling procedures, and author and Journalimages;(c) experiments in a real-time environment with actualuser populations, assuming the availability of console input and a time-sharing computer organization;(d) experiments using combinations of analysis methods,
and the iterative search techniques previously developed, to test the effectiveness of multiple
searches compared to the single search operations
now used with all large, operating systems;(e) experiments using documents in other languages, suchas French, together with translated versions of thedictionaries, to determine whether multilingual collections can be processed;(f) experiments using different sections of each document for retrieval purposes, such as section titles,
figure captions, summaries, reviews, and so onQ
In addition, the efforts on systems evaluation, up to now largely
confined to the generation and computation of various recall and precisionmeasurements, may be expected to be continued, and to be extended also tocost and other parameters, if more realistic operating conditions can beachieved 0

REFERENCES

1.

G. Salton, "A Document Retrieval System for Man-machine Interaction,"
Proceedings of the ACM 19th National Conference, Philadelphia (1961|).

2.

G. Salt on and M0 E. Lesk, "The SMART Automatic Retrieval System - An
Illustration," Communications of the ACM Vol. 8, No, 6 (June 1965).

3.

E e H. Sussenguth, Jr,9 "The Use of Tree Structures for Processing
Files," Communications of the ACM, Vol. 6, No, 5 (May 1963)•

km

M. Cane, "Dictionary Look-up and Updating Procedures," Report ISR-7,Section IV, Harvard Computation Laboratory (June 196U)•

5a

M, Cane, "Dictionary Look-up and Set-up Procedures," Report ISR-9
Sections V and VI, Harvard Computation Laboratory.

6.

G0 Shapiro, "Processing of the Concept Hierarchy," Report ISR-7,Section V, Harvard Computation Laboratory (June 1961|).

7.

G0 Shapiro and M. Razar, "Hierarchy Set-up and Expansion Procedures,"Report ISR-9, Section XV, Harvard Computation Laboratory,

8.

M. Lesk and Ta Evslin, "Statistical Phrase Processing," Report ISR-7,Section IX, Harvard Computation Laboratory (June 1961;).

9*G« Shapiro, "The Statistical Phrase Program," Report ISR-9, Section VII,Harvard Computation Laboratory,

10.

E. H. Sussenguth, Jr., "The Sentence Matching Program - GRAPH,"
Report ISR-7, Section VII, Harvard Computation Laboratory (June I96I4)o11.

G. Salton, "Automatic Phrase Matching," Report ISR-8, Section V,
Harvard Computation Laboratory (December 196ij)j also presented at theInternational Conference on Computational Linguistics, New York (May1965).

12.

G. Salton and E. ff. Sussenguth, Jr., "Some Flexible Information Retrieval Systems Using Structure Matching Procedures," Proceedings ofthe AFIPS Spring Joint Computer Conference, Washington (April 196U).

1-9
13.

M. Lesk, "The SMART System - General Program Description/Report

ISR-7, Section II, Harvard Computation Laboratory (June I96I4.) •111. M. .Lesk, "The SMART Automatic Text Processing and Document Retrieval
System," Report ISR-8, Section II, Harvard Computation Laboratory(December I96I4.).
15.

Tc Evslin, "A General Discussion of the SMART System/1 Report ISR-9,Section II, Harvard Computation Laboratory*16.

G# Salton, "The Representation of Structured Information," Proceedingsof the IFIP Congress - 65, Spartan Books, New York (1965),

17•

G. Salton, "Data Manipulation and Programming Problems in AutomaticInformation Retrieval," ACM Working Conference on Pragmatics andProgramming Languages, San Dimas, August 1965o18.

J. J. Rocchio, Jrc, "Performance Indices for Document Retrieval Systems,"
Report Noo ISR-8, Section III, Harvard Computation Laboratory (December196U) 019.

G. Salton, "The Evaluation of Automatic Retrieval Procedures - Selected
Test Results Using the SMART System," Report ISR-8, Section IV (December
19610$ also to be published in American Documentation (1965)020.

C. Harris, "Analysis of Student Requests," Report ISR-9, Section XX,Harvard Computation Laboratory021.

J0 J. Rocchio, Jr0 and Go Salton, "Automatic Search Optimization andIterative Retrieval Techniques," Report ISR-9, Section XXIV, Harvard
Computation Laboratory.

22«

J. J. Rocchio, "Relevance Feedback in Information Retrieval," Report
ISR-9, Section XXIII, Harvard Computation Laboratory©

23.

M. Lesk, "A Program to Evaluate the Iterative Search," Report ISR-9,
Section XVIII, Harvard Computation Laboratory02k.

J. Jo Rocchio, "The Merged Output Results," Report ISR-9, Section XIX,
Harvard Computation Laboratory0