VI-1

VI.

Suffix Dictionaries
E. M. Keen

1.

Introduction
The use of suffix removal procedures as a simple method of vocabularycontrol is investigated with two types of suffix dictionaries.

The need forvocabulary control and the desirability of synonym and partial synonymrecognition are discussed in Section I.

A suffix removal procedure has been

incorporated into the SMART system from its inception, which has been known
as the Mnull thesaurus", but is here described as the stem dictionary.

Asecond type of dictionary recently tested is the "suffix f s f dictionary",
since this offers the most basic language analysis method involving virtuallyno vocabulary control; as such, the suffix f s ? method provides a convenient"base-line" from which dictionaries exerting greater control can be evaluated.
A brief description of the two dictionaries will be given, together withretrieval performance comparisons and an analysis of the results.

2.

Description of Suffix Dictionaries.
Both the suffix f s' and stem dictionaries are automatically generated,

and the suffix removal procedure and collection look-up operations havebeen described elsewhere [1,2,3,^,5,6].

Briefly, the full suffix removal

process (stem dictionary) is carried out in two stages:

first, the constructionof a dictionary of word stems, formed by applying a hand-made list of suffixes
to a body of text; and second, by a look-up process which uses the dictionaryof word stems plus certain spelling rules to reduce the documents texts to

VI-2
be stored in the system to word stems only.

The suffix f s ! dictionary isapplied in the same manner, but in this case the only 'suffix' removed is theterminal 's', with the object of conflating singular and plural word forms.
Many of the considerations relating to the methods Oi constructionof the stem dictionary have been discussed by Salton and Lesk [7]. The comments
made here relate to the extent to which the present dictionaries correctly
conflate English word forms and so use the correct stems.
The conflation of singular and plural words is not perfectly achievedby terminal "s" removal, although over JCfjo success is obtained in the case of
the Cran-1 aerodynamics terminology.

The failures are due to well-known

singular and plural forms such as "body" and "bodies", "axis" and "axes","bureau" and "bureaux", "appendix" and "appendices", etc. Also, the terminal "s" does not always denote a plural form, and words like "bluntness"and "aerodynamics" have the "s" removed.

This latter occurrence rarely affects

retrieval, however, since a request and document both containing the word
"bluntness" will match on the word without its terminal "s". It is possible
to imagine a case of incorrect conflation, for example, the word "axe" could
be incorrectly conflated with "axes", but such occurrences are extremely rare
within the narrow subject fields under test.
The full suffix removal procedure incorporates spelling rules whichcorrectly identify "bod" as the stem of both "body" and "bodies", and correctly
conflate "hope", "hoped" and "hoping", as well as "hop", "hopped" and "hopping".
There are some cases, however, where the correct stem is not recognized.
For example, the words "computation", "computations" and "computational" arecorrectly conflated and given the same concept number as the look-up procedure,
but a second group of similar words is given a second concept number including
such words as "compute", "computed", "computers", "computer", and "computing".

vi-3
A second example is the term "compressible", used in the aerodynamics
literature, which is kept separately from "compressibility".
It appears that amendments to the automatic procedures used could
solve at least some of these problems, and it is certain that for every suchproblem there are at least ten cases of correct conflation.

Examinationof the groups of words that are related by this conflating procedure suggeststhat the majority are helpful for document retrieval.

A distinction between"computer" and "computing" is not believed to be useful, and preservationof the two forms is unlikely to be helpful to a requester.

An exception tothis situation may be furnished by the inclusion of a noun with the adjectival and verbal forms.

Although the practice of using a "computer" isrelated to the "computer" itself, a request for documents

describing one

named computer may not perform well if documents describing computational
procedures are highly matched with the request.
The performance results presented suggest that this type of unwelcome conflation is a contributing factor to the poor performance of the stem
dictionary on the Cran-1 aerodynamics collection.

The words "compressor"and "compressors", for example, are unhelpfully grouped with "compressible"and "compression", when notions such as "jet engine compressor", "compressible
flow", and "compression buckling" are quite unrelated.

Naturally any hand-produced dictionary, such as the thesaurus dictionaries described in section
VII, can easily handle such conflation problems, but the claim for automatically generated dictionaries is that cases of failure are few enough tojustify the large saving in effort of construction.

This general claim seems

to be potentially far better justified by the automatically generated thesaurustype dictionaries produced by statistical association (see section VIII andappendix C ) , since hand construction of a stem dictionary would require little
effort if an exhaustive concordance of the collection were available.

VI-U
3.

Retrieval Performance Results
Comparisons of the suffix f s f and stem dictionaries are presented

for the three document collections, using the normalized measures, precision
versus recall graphs and data from individual requests.

Figure 1 gives ten

results using the normalized recall and precision measures.

The ADI results

include text, abstract and title results, and some results are displayed
both for the ADI and IRE-3 collections with overlap correlation and logical
vectors.

All ERE-3 results and four of the six ADI results show the stem

dictionary to have higher normalized values, although by quite small amounts.
The single Cranfield result and the ADI text cosine and overlap logical runs
show suffix f s ! to be the superior dictionary.
Four results are given using precision versus recall graphs:

IRE-3

Figure 2(a), Cran-1 Figure 2(b), ADI Abstracts Figure 3(a) and ADI Text
Figure 3(b). These results confirm those in Figure 1, and the Cran-1
result is seen to favor suffix f s f over the whole range of the curve.

To

complete all the runs given in Figure 1 in terms of precision and recall,a table is given in Figure h that summarizes six more precision/recall plots
not presented in detail, by recording the precision merit at three levelsof recall.

Some disagreement between these results and the normalized measures

may be noted, and the reasons for this axe discussed in section II.

Thecases of disagreement all consist of very small differences in merit between
suffix f s f and stem, and all the more valuable comparisons which use thecosine correlation and numeric vectors display consistent results.

The aver-age performance measures show, therefore, that stem is superior to suffix 's1on the IRE-3 a ^d ADI collections, and that suffix f s ! is the better dictionary on the Cran-1 collection.

VI

COLLECTION

IRE-3
31+
Requests

CRAN-1
1+2
Requests

ADI

INPUT AND
MATCHING FUNCTIONEVALUATIONMEASURESTEM
DICTIONARY

SUFFIX 'S'
DICTIONARY

Abstract, Cosine
Numeric

Normed. Recall
Normed. Precision.8951+.67I+6.8817.61+81+

Abstract, Cosine
LogicalNormed. Recall
Normed. Precision

.8777
.6167

.8707
.6131+

Abstract, Overlap
LogicalNormed. Recall
Normed. Precision

.8725
.5829

.81+08.5611

Abstract, Cosine
Numeric

Normed. Recall
Normed. Precision.861+4
.670U

.8717
.7018

Text, Cosine
Numeric

Normed. Recall
Normed. Precision• 7779• 5573• 7520.5308

Text, Cosine
LogicalNormed. Recall
Normed. Precision.7695.521+8.7768.51+62

Text, Overlap
LogicalNormed. Recall
Normed. Precision.7I+3I+.1*978.751+6• 5097

Abstract, Cosine
Numeric

Normed. Recall
Normed. Precision.7601• 5326• 7253.1+997

Abstract, Cosine
LogicalNormed. Recall
Normed. Precision.751+6.5221.7296.501+1+

Title, Cosine
Numeric

Normed. Recall
Normed. Precision.6722.1+537.61+35.1+20935
Requests

Performance Results Comparing Stem and Suffix f s f Dictionaries
for Ten Options on Three Collections, using Normalized Recall and Precision.

Figure 1.

vi-6

X
•H
CM
CM

en

DO

2

o
•H
-P

a

O

CD

-P

o
o

•H

3

w
o

o
oa

o
o

CO

OJ
•H
M

0)

c

w
w

-p

o

o
CM

w

O
cd
M

-p

w

3
on
i

CVJ

O
•H
-P
O

•H

vi-7

o
o

m
m

1?I I

EH

~

CO
CD
•H

h
cd

«

ext

03

ion

0.4

<o
o

CO

•p

ques

«?
o

-P

H

o

H

•H

a
0
CO
-p

o

h

a

p.

Com

O

I
CO

t
I
t

<
%

•H
CO

3

a
o

•H

ect

and

3
CO

CD

o
o

3

S

CD
CJ

CD

,3

•P

e
^

fi

O

vi-8

INPUT AND
MATCHING FUNCTION

IRE-3

Abstract, Cosine Logical

S

S

X

CO

CO

Abstract, Overlap Logical

X

X

X

CO

CO

Text, Cosine Logical

X

X

CO

X

X

Text, Overlap Logical

X

CO

CO

X

X

Abstract, Cosine Logical

co

CO

CO

co

S

Title, Cosine Numeric

co

CO

X

co

STEM(S) VERSUS SUFFIX 'S'(X)PRECISION AT RECALL 1 NORMALIZED.2.5.8
REC. PRE.

COLLECTION

S

3k

Requests

ADI

35
Requests

The merit of one dictionary over the other in these results isalways by less than 0.05 precision, normalized recall and normalized
precision.

Table Summarizing Six Precision Versus Recall
Plots not Presented, Comparing Stem and Suffix 's1
Dictionaries for Six Options on Two Collections.

Figure U.

vi-9
These average results may be supplemented by the individual request
data given in Figures 5, 6, 7 and 8.

Using the normalized recall and pre-cision measures as indicators of merit, it can be seen that Jl% to jk^

of therequests favor stem on IRE-3 (Figure 5 ) , and 53$ to 75$ of the requests
favor stem on ADI abstracts (Figure 7) and text (Figure 8 ) . The Cran-1
result favoring suffix f s' is confirmed by figures relating to the individual
request also, with 72$ to 77$ preferring suffix f s f , ignoring those requests
which have equal merit for both dictionaries.

Each figure includes plots ofboth normalized recall and precision versus the individual requests.

In

the case of Cran-1 these plots show that suffix f s f is superior on the average
because many of the requests favor suffix *sf by very small amounts.

In

the IRE-3 and ADI collections the stem dictionary displays same large changes
in individual requests in its superiority over suffix f s f .

U.

Performance Analyses
Two phenomena require explanation:

firstly, the IRE and ADI runs

involving logical vectors and overlap correlation which sometimes show suffix 's1 superior to stem; and secondly, the superiority of suffix 's1 on the
Cran-1 collection.
The first phenomenon is less important than the second, because
logical and overlap runs are inferior to cosine numeric runs in any case.
Cases where suffix 's1 is better than stem must be caused by circumstancesof the type considered in part 2, where full suffix removal conflates sanewords that match with non-relevant documents and thus adversely affect performance.

It was noted in section III that the use of numeric vectors(weighted) gives a clear advantage over logical vectors when a dictionary
is in use that includes a reasonably large amount of mapping (i.e., itVI-10

•

•

on
h-

vo
CM

&

CM
CM

CD

-p

co
O
H

a;
-p
-p

§
a
CD

3

co
C
o

4->
CO

H
CD

ctf)

a

4->

-P
a;
&

co

co

-p

-P
CO

W

Hd

•H
H

o

<D

o
CJ
CD

CO

•p

<D

CO

<D

T3

£ a

a;
ON
LT\

cu

CO
- p

CO
CD

CD

H
cd

cti
a

3

CD

'd

cc;

>

^d

• H

CD

a

-H
H

M

CO
CD

o
d

CD
W
CD
CM
CM

ON
C\J

<D

-P
«H

O
CO
CD
CM
CM

CO

2
CO
CD
•H
M
cd

c:
o

•H
*p

o

•H

Q

1
-p

CO
CM
CM

O
M
0)
-p
-p

CD

xs
-p
CM

CD

O

i
CO

CO
- p
CO

CD
CD

05

CO

VI-11

H 00
t^- OJ

Fl
<D
4
3 P P
co CO o

<M
CM

G
O

d

-a

§

9^

O
•H
CO

cu H
aJ
P p
3
-P p
<D CD Cf
& & a;CO CO coP P P
co CO toa; a; 03 2 2o< o< o<a> CD CD
CX «

o

O

o
u
CD

H

£

LT\

CO

£

p
CO

T3

g

§

3

Q tr-

CVI H

1
CO
CD
O

VO

c;
(L>
H
0)

•H

m

CM
CM

so C\J

2

aJ XI

CO

p
CO

c
o

G*
o

M

0)
p

p

H
CD

p

p

bot

<M

a
o

3

2,
cr

£>

CD
rQ

CO

CO

CO

p

p

P

CD

CO

CO

CD

CO

<v 0) a;3 2 3o< o< o<CD cu *!
OS en tr;

H

«

CO GO

CO

1
g
p
CO

VI-12

^

CO

CO

CO

OJ

C -

CD
-P
CO

CO

:>§
C

0)

HO:
— 00

\

\

o

§ §

•H

M

M

cd

-p
^p
CD

CD
-P
4^>
CD

co
-p
co
CD

w CD
-p
CO ^ P
co

CO
-p
CO
CD

C^
CD

<D

cr1

cr-

<D

00 MD

H

d
o

M

<v

!

cd

%

•H
CO
•H
O
CD

CD

CD

cd
o

cd

w
CO
CD

&

•H

>

CD
T3
(D
(SI

a H

CD

cr;

on
w
-P

o
M

-P
CO

•H

O

CO
D

tifl

•
h

CD

CO

ctf)

M

CD
CM
CM

•H

CO
CD

CD

cd

Q

M

JO
•p

OJ

CM

O

X
•H

-P

3

CO

CO

§ o
g
u MCD

<D
-P -P
-P -P
CD CD

&

CO
-P
co
CD

&
CD

CO

•P
co
CD

&
CD
CO
P
co
CD

1
OJ

on

d

o

•H
-p

o

CO
CD
T*
3

Q

-p

-

CO

xIS
CD

i

CO

43

ng

«H

a

CO

CD

Co

vi-13

•H
<D <H

-P

3 -P

x:

CO

CO

o

I
M

o o
CD

CD

-P «P

&
CO

-p

CO
CD

o

1

CD

CO

1

u*
CD

3

-p

CO
4->
CO
CD

CO

i3

CO

CD

H

-p

CD

CO
CD

a

CD

3 3 3
o<
a* o<
OJ CD

«

o

CD
rO

*P •P
CD

O

~

«

CD
OS 05

CD

-3- oo on

N

T3
CO

-p
CO
CD

I
in
on

CO
CD

o
c:
CD
w
CD
<H

CO
CO

3
CO
CD

CD

H

M
CD

:>§

-P

•O IT
C 0)

Cw

O

Htr

-P

o

CD

C - OJ,

1
-P
CM
<M
CD

CO

-P
-P

2

CO

CO

s s
O

O

* u

5
O

CD

CD

CD

I

CO

CO

CO

CD

-p -p -p
CO
CD

CO
CD

CO
CD

U* U* CT*CD CD CD« « «^ t CO on

2 2 « "L
E go
<o
z

*

8 $

O

i
CO

§

CD

-P -P
-P -P

<H

u
o

«P
CO

VI-lU
conflates many words), and a similar, but unexplained, relationship is notedwhen the use of cosine is compared to overlap.

From a strictly experimental

viewpoint dictionaries such as suffix f s f and stem should be compared without
the addition of weighting procedures and cosine, in order that the dictionary
mapping characteristics may be tested alone.

In this case, the overlap logi-

cal results show that stem and suffix f s f dictionaries perform very similarly,
and therefore within the context of the requests and relevance decisionsin use, no advantage should be gained from full suffix recognition as performed automatically.

This finding is in accordance with the general con-clusions of the second Aslib-Cranfield Project [8], although in those results
the nearest equivalent to the stem dictionary does perform a little better
than suffix f s f .
However, a more practical conclusion in the case of SMART is thatstem is the superior dictionary on the IRE-3 and ADI collections, since thecosine correlation and numeric vectors have clearly been proved to be advantageous, and would be advocated for use in any operational version of
SMART.
The superiority of suffix f s f on Cran-1 is one of several instanceswhere the Cran-1 result differs from the other collections.

In the case of

Cran-1 the difference in word mapping between suffix f s f and stem is lessmarked than in the other collections, since Figure 9 shows that the Cran-1
stem dictionary includes 83$ of the concept classes contained in suffix's 1 , whereas the IRE-3 and ADI stem dictionaries are based on more mapping
characteristics, including only 76$ and jh% of suffix f s f , respectively.
As expected,

this affects the match with requests and documents, since

Figure 10 shows that at a cosine correlation cut-off of O.35, the stem
dictionary in Cran-1 does not retrieve so many additional documents oversuffix ' s1 than is true for the other collections.

vi-15COLLECTION
AND SIZE

CONCEPT CLASSES IN DICTIONARIESSTEM AS t
STEM
SUFFIX 'S'
OF SUFFIX 'S'

IRE-3, 790

5,150

3,933

ADI Text, 82

7,615

5,606

Cran-1, 200

3,181

2,628

76. H

82.6*

Comparisons of Stem and Suffix 's1 DictionarySizes on the Three Collections.
Figure 9-vi-16

COLLECTION

NUMBER OF DOCUMENTSABOVE CORRELATION 0.35
SUFFIX 'S'

IRE-3(3^ Requests)

157

222

ADI(Abstracts)(35 Requests)

UO

55

ADI (Text)(35 Requests)

k2

61

CRAN-1(1*3 Requests)

k2

5*

INCREASE IN NUMBER OF
DOCUMENTS ABOVE CORR. 0-35
STEM AS PERCENTAGE OF SUFFIX 'S*

37-5#

25.6^

Comparisons of Numbers of Documents with a Cosine Numeric
Correlation above 0.35 on Stem and Suffix 's' Dictionaries,
using Four Results from Three Collections.
Figure 10.

vi-17This data shows that full suffixing (stem dictionary) does not affectso many aerodynamics words as it does in computer science and documentation,
thus giving the Cran-1 collection less of an opportunity for a change inretrieval performance.

Further explanations can be given by observing in-dividual request performance for the seven requests that have performancechanges greater than 0.05 normalized recall (see Figure 6 ) ; four of theserequests are better on stem, and three are better on suffix f s f . An analysisof the three requests that favor suffix f s f reveals certain test problems,connected mainly with hyphenation and keypunch errors, and in fact the request/
relevant document match is in all cases very weak.

The many requests thatfavor suffix 's' by a trivial amount (Figure 6) are typified by request
Q269, details of which appear in Figure 11.

As Figure 11 shows, the stemf,compressM incorrectly matches the request word "compressor" with a frequently

used word "compressible,f in two non-relevant documents, so that the stemmdictionary has an inferior performance because relevant document 1590 receives
a rank position below the two non-relevant documents.

In case the matchingwords of non-relevant document 198^ appear to put into question the relevancedecision (the abstract includes the topic of "choked flow in an impeller
inlet"), the title makes it clear that it is a "centrifugal impeller", andthe matching word "axial" is a gjurious match from the phrase "axial SymmetryM.
The r&nks of the seven documents relevant to request Q190 are given
in Figure 12, because the changes in rank position observed are typical ofwhat happens to the averages. The highest ranked relevant document remains
unchanged at position 1; thus the high precision end of the curve for thisrequest will remain unchanged.

The next three relevant items are ranked

better by suffix 's f , but the final three relevant are ranked better bystem.

This result is seen to give a greater superiority to suffix f s f in

vi-18

Request Qg69 (Cran-1 Collection)
Has a criterion been established for determining the axial compressor choking line?

Suffix t s l Dictionary
CosineNumeric
Correlation

Matching Words and Wei

1591 Relevant.2817

Compressor(3) Line(2)1590 Relevant•1375

Axial(l) Compressor(2)1967 Non-Relevant.0922

Axial(3)1984 Non-Relevant.0301

Axial(1)Rank

Document

2
5
7
20

Stem Dictionary
CosineNumeric
Correlation

Matching Words and Weights

Rank

Document

1

1591 Relevant.U23U

Compressor(U) Choke(2)Line(2)5

1967 Non-Relevant.17^

Axial(3) Compressible(2)Determined(l)61984 Non-Relevant.1532

Axial(l) Compressible(2)Choked(1) Determined(l)71590 Relevant• 1365

Axial(l) Compressor(2)Individual Aerodynamics Request

Q269 Showing Superior

Performance Obtained with Suffix ? s' Dictionary for
Relevant Document 1590.

Figure11.

VI-19Request Q190 (Cran-1 Collection)

Ranks of the 7 Relevant Documents
Suffix's'

Stem

Relevant
Document

Rank

Relevant
Document

30
60
81

987
988
989
9&
985
990
986

1
7
17
21
1*
53
61

987
988
989
98U
985
990
986

Nor. Rec.

- O.8719

Nor.

Nor. Pre.

= 0.67^5

Nor. Pre. = 0.6072

Rank

1

3
6
20

•

Rec. - 0.8697

Individual Aerodynamics Request Q190 Comparing
Suffix f s f and Stem Dictionary Performance.

Figure 12.

VI-20normalized precision than normalized recall, as the averages show (Figure 1 ) ,But the precision/recall curve is little affected by dictionary change when
averaged over all requests, as Figure 2(b) shows.
A definite conclusion must await an investigation into the effectof changes in subject language and the effects of differing methods of request
and relevance decision preparation, since both factors are involved in acomparison of Cran-1 with the other two collections.

Meanwhile, the evidence

presented does point to a difference in language characteristics, and tests
on the larger Cran-2 collection of 1^00 documents will shed more light on
this.

5.

Conclusions
The comparison of the two suffixing dictionaries shows stem to besuperior on the IRE-3 and ADI collections, and suffix ! s f to be superior on
the Cran-1 collection.

All differences between dictionaries are small, andthe use of overlap correlation and logical vectors on the IRE-3 and ADI collections lessen the superiority of stem; however, the cosine numeric result
is to be preferred to these procedures.

The aerodynamics terminology ap-

pears to offer less opportunity for word conflation than the computer scienceand documentation terminologies; this remains the primary explanation so far
discovered for the Cran-1 result.
Every indication shows that the suffixing dictionaries provide aconvenient and valid base-line from which further dictionaries of the thesaurus type can be evaluated.

However, the use of some type of suffixing

dictionary does provide a good retrieval tool in its own right.

Such dic-tionaries should be considered both as tools that can be constructed with

VT-21a minimum of effort in the absence of a thesaurus dictionary, and alsoasprobable candidates for inclusion in systems in which a series of several
dictionaries are provided from which a pre-search choice can be made-Ifthe latter reason for inclusion of a stem dictionary is valid, then it would
seem that a hand edited version, which would require little human effort,
would probably overcome many of the detailed deficiencies that have been
described.

VI-22

References[l]C. Harris, Dictionary and Hierarchy Formation, Information Storageand Retrieval, Report ISR-7, to the National Science Foundation,
Section III, Harvard Computation Laboratory, June I96U.

[2]

M. E. Lesk and T. Evslin, Housekeeping Routines, Information Storageand Retrieval, Report ISR-7, to the National Science Foundation,
Section XI, Harvard Computation Laboratory, June 196k.

[3]

C. Harris, Dictionary Construction and Updating, Information Storageand Retrieval, Report ISR-8, to the National Science Foundation,
Section VII, Harvard Computation Laboratory, December 196^-[k]

M. Cane, The Dictionary Lookup System, Information Storage andRetrieval, Report ISR-9, to the National Science Foundation, Section
V, Harvard Computation Laboratory, August 1965*[5]

M* Cane, The Dictionary Setup Procedures, Information Storage andRetrieval, Report ISR-9, to the National Science Foundation, Section
VI, Harvard Computation Laboratory, August 1965*[6]

M, E. Lesk, Operating Instructions for the SMART Text Processing andDocument Retrieval System, Information Storage and Retrieval, Report
ISR-11, to the National Science Foundation, Section II, Departmentof Computer Science, Cornell University, June 1966.

[7]

G. Salton and M. E. Lesk, Information Analysis and Dictionary Constuction, Information Storage and Retrieval, Report ISR-11, to the
National Science Foundation, Section IV, Department of Computer
Science, Cornell University, June 1966.

[8]

C. Cleverdon and M. Keen, Factors Determining the Performance ofIndexing Systems, Volume 2, Test Results, Aslib Cranfield Research
Project, Cranfield, 1966.