ISR-8
December196)4

VIII-1VIII.

THE EXTENDED SMART SYSTEMTom E v s l i n1.

Introduction
A new 'version of SMART has been designed as an extension andan improvement of the old system.

Various limitations inherent in theold system have been removed, provisions have been made for additional
options which are of interest in information retrieval, and several
logical changes were made to increase the capacity of the system and tospeed up processing.

2.

Limitations Removed
The SMART system described in Information Storage and Retrieval,

Report No. ISR-7? is designed to process not more than £00 documents ofshort abstract length.

The number of documents which can be processed

during a single run decreases as the number of words in the average
document approaches the maximum text length of 1,000 words. In the newsystem, there is no limit to either the number of documents which can beprocessed in a single run, or the number of words included in a single
document.
In the old system, the maximum number of word stems which canbe included in the thesaurus is 1,000.

Up to 30*000 noncomrnon stems

may be included in the extended system.

This number approximates moreclosely a useful technical vocabulary.

In addition, 2,?67 concept

VIII-2
numbers are reserved for common words, 32,000 numbers for mechanicallygenerated word clusters, and 32,000 concept numbers each for authors,
journals, and bibliographic entries,

3.

New Options
A new program has been written to provide inexpensive semi-syntactic phrase searching.

This program weights statistically foundphrases according to the probability that the word elements actuallyconstitute a phrase in the sentence where they are found to co-occur.
Intervening punctuation, position in sentence, and syntactic values aretaken into account by this program.
A new option is available which permits the user to specify thata looked-up document collection is on tape AG. Using this option, it ispossible to run by submitting only new requests or by submitting only a
few texts not previously looked up, etc.
As described above, provisions are made for concept numbers
designating authors, journals, and citations. These concept numbers areadded to the document vectors and correlated in the same way as conceDt
fibers representing word stems or word clusters.

The document vector

format is described more fully in Part k of this section on LogicalChanges.
In the old system, concepts which appear in the titles ofdocuments can be weighted differently from those appearing in the bodyof the document. The weight is specified by the user at run time. Inthe extended system, the user may also specify separate weights forVIII-3

statistical phrases, syntactic phrases, semi-syntactic phrases, wordclusters, authors, journals, and citations• In addition, all ofthese classes may be weighted differently for occurrence in the title
or body of a document.

4, • Logical Changes
Experience has shown that syntactic phrase searching is extremely time consuming. In order to avoid unnecessary syntactic
processing, statistical phrase searching is always executed prior tosyntactic or semi-syntactic analysis. In this way it becomes possible
to avoid the syntactic processing of any sentence which does not containall the terms of at least one phrase from the criterion tree file.
However, statistical phrase finding continues to be available as aseparate option.
The generation of word clusters by means of term-sentence
correlation is accomplished by a new program run prior to production
runs of the main SMART system.

This program, which is described more

fully in Part 5 on Support Programs, produces a file of word clusters
with generated concept numbers. These clusters are detected in documents and requests by the statistical phrase finder during the lookup,
and the corresponding concept numbers become part of the document
vectors.
In order to remove the limitations on the number of documents
to be processed, in a single run and the number of words which can be

VIII-4
included in a single document, it becomes necessary to store thedocument vectors on tape instead of keeping them all in corestorage•Requests, however, will remain in internal memory as

before*Significant changes have been made in the format of document
vectors*

Each, vector begins with a 12-word record of which the firstword is either *TEXT or *FIND and the second and third words are thedocument identifier•

The remaining nine words are not specified•

The rest of the vector consists of an unlimited, number of 260 word
records, the last of which need not be exactly 260 words, and is
terminated by a word of zeros• The following list summarizes thepossible word formats.
Prefix0:

either word stem, syntactic,
semi-syntactic or statisticalphraseU:cluster5:author6;

j ournal

7:bibliographic entry

1,2,3:
Decrement

Tag

not assigned
contains a concept ;(with a prefix of zero, 1-30,000
are reserved for noncommon wordstems, 30,001-32,767 for commonwords)0:

word stem from oody1;semi-syntactic phrase, body

VIIIstatistical phrase, bodysyntactic phrase, bodyword stem, title
semi-syntactic phrase, title
statistical phrase, title
syntactic phrase, title
Address

contains a weight for theconcept stored in thedecrement

These items appear in logically sorted order on the tape.
Although it may seem that the changeover to tape would increase
processing time, such an increase is not anticipated now. Previously,both the lookup and syntactic processing were done on a document-bydocument basis with different chain links looping through core foreach document. In the new system, these processes will be carried out
serially for the entire collection with each chain link remaining incore until it is no longer needed. This alteration is expected, tocompensate for the increased tape time previously referred to.

5. New Support Programs
A new null thesaurus generator has been written to produce thelarger thesaurus acceptable to the new system. This new program also
accepts an unlimited number of documents, each of unlimited length.
The null thesaurus program accepts input parameters specifying themaximum number of noncommon concepts desired, and/or the minimum numberVIII-6of times a word must appear in the collection in order to be includedin the null thesaurus.
A program has been w r i t t e n to generate word c l u s t e r s from adocument collection and to produce a dictionary of these c l u s t e r s .
Documents are looked up i n the thesaurus, and the term-sentencec o r r e l a t i o n i s performed.

Cluster coefficients are calculated forterms which co-occur in sentences.
each c l u s t e r .

A concept number i s generated forThe user supplies a cut-off coefficient and/or amaximum number of c l u s t e r s to be included in the dictionary.

This

dictionary i s used at lookup time to find phrases (word c l u s t e r s ) int e x t s and in search requests.

The programming of a l l changes describedin t h i s section i s presently under way.

VIII-7
LINK 1START

A2

Al?

Read Input

Parameters

Legend:and Set Flags>

Read Card U-"-LI ST

Logical Block in Kain
Flow of Control> A<zzz>*FIND — * EDecision*LIKE —> C
*TEXT — > D( _ )Logical Block to be
Executed if Needed orRequested*ONLY — > E*TAPE — ¥ F-x-STOP —=r F*E0F—> F

4 Set Flag for Punch

Set Flag for Punch
on Al;•^ Set ttLIKE Invalid

J Set -LIST, -FIND
Invalid
TA2

j*L'A3-^Ai;

Segment Text-&<( Is Buffer Full?
YesNo

VIII-Lookup Text

B5

(Write Words \not FoundJ(Find ClustersJ-

M B6

Find StatisticalPhrases->(B?)

Put Out Sentences forSemi or Full SyntacticProcessing-MA8Form Vectors</*)B6

Last Text Complete?

NosYesBack Space
Record</hi

Go To Next |rLinki

VIII-9

LINKS 2-3I

P r i n t Words
Not Found

A3

Link 2( Semi-syntacticProcessing

M B6

Link 3- * •Syntactic
Processing

A8

uO toNext
Link

I<Were Documents
Looked Up?

lesi

H Head Record-LIST->CtfTND
s-LI KE-*• E

fTEKT•> F&0F-> B-*B

Links U-8

VIII-10

Set Flag for Punch-*A> Backspace Record^B>,sA<B6^ o + ft1! ^ <- f n T *Ot?t/ r J_d g X OXF1T»^W-+x O-A. u

J/Merge Vectors

1(Punch)IEZ
Weight Concepts

B^

Stack i n Core"ZZIIIZZ<^ Text F l a g O n ? ^No-*AYes
Put on TapeT

1.
Documents on A6? >—•—Yes-»-G

VIII-11

A6

4 Read Record k-*TEXT^ II*EEND>• I

E0F — » - H
si Backspace Record•»GI
A6Weights Concepts
Stack i n CoreG1A2

Read Card-x-FIND*J*TEXT — > K*TAPE — * K-x-STOP>KE0F>K

T4 Backspace Record
J©-"Weights Concepts
Stack i n Core-*L

VIII-12

Expand Requests by Hierarchy
and Add Expanded Requests t oOriginals

NoWere Documents Looked Up?

>MYetNoAre Documents on EI?

->N0©-

E0F

Read-in and WeightVector^NC o r r e l a t e with Requests

B2

Expand Document byHierarchy & Recorrelate

B2

Put Document Vector out
if There is to be DocumentDocument or Term-Term Correlation

Read Record

E0y\s—v\-4j^l )/ ^—- /->M

J Merge Vectors(timch)Weight ConceptsiBit

B2

Correlate with Requests

Expand Document by Hierarchy \& Recorrelate
JJ/.
Put Out V e c t o r for D o c u m e n t Document & Terra-Term Correlation

MJL.

Read Card

A2>P-*TEXT*STOP — > Q*TAPE — > Q
E0F

R-*R

BSR

Read-in Vector &Weights Concepts

A2

Correlate with Requests(Expand <k Recorrolate Vci

Put out V e c t o r

TMVVHI-lli

Q<Are Documents on A6?

j>—

>Sles
A6

Read-in Vector & WeightConcepts

E0F

Correlate with Requests

f Expand & Recorrelate J

B2

Put out Vector

A7>

Go to Next LinkLINKS 10-12AM

Docume n t -Do curne nt
Correlation

Link 1.0;KTerm-Term Correlation& Document Expansion

Link 1 1

3:

Recorrelate withRequests

W r i t e Report

Link 9A3

Link 12