VII-1

The Combination of Thesaurus and Word Form Vectors
B. Faith and J. Jensen

Abstract
In this study, the Thesaurus and the Word Form Dictionaries aremerged, and the performance of this new dictionary is compared to that ofits individual elements.

The new dictionary yields better normalized pre-cisions and recalls, but the improvement is only slight and the results aresometimes inconsistent.

1.

Introduction
One of the major areas of research in information retrieval is theinvestigation of dictionaries.

There have been numerous studies trying todeterming the best type of dictionary to use.

In ISR-13, Section VI [1],E. M. Keen compared the performances of stem dictionaries versus "suffix s"
dictionaries and found, with one exception, that the stem is superior to thesuffix s.

In Section VII, Keen continued these studies by comparing thethesaurus against the stem dictionary, and determined that while the results
were very close, the thesaurus is nearly always superior.

Additional inves-tigations added phrases and hierarchy dictionaries to the thesaurus.

Thephrases only slightly improved performance, but the addition of the hierarchyactually hindered it.
It appears that one other study is required — a comparison of the results of the thesaurus and the word form dictionaries separately and thencombined.

Since the thesaurus and word form vectors yield recall and pre-cision graphs that are very close to each other, it would seem that the two
dictionaries should complement each other.

The major question is whether theVII-2

improvement is significant enough to justify the extra time and cost involvedin the computer execution.

2.

Procedure
The procedure for accomplishing this study requires the use of the CRAN200 Thesaurus and the CRAN 200 Word Form dictionaries. Each is tested separately
with the 42 available queries and 200 documents, and the two are then concatenatedby use of an object module. The object module contains two subroutines, MASTERand CRDCON.

CRDCON merges the CRN2S and CRN2TH dictionaries, adding a constantto the concept numbers of the Word Form dictionary in order to maintain theseparate identities of the two dictionaries for both queries and documents.
The constant is added by means of the addition of a "DO-LOOP" to CRDCON.
The constant is introduced to the system through the subroutine MASTER.
Each of the three runs (Thesaurus only, Word Form only, and merged
Thesaurus and Word Form) are searched in the usual manner.
of all the documents is made Cthe 0 iteration).

First, a search

Second, two searches are

made with feedback Citerations 1 and 2). holding the ranks of the relevant
documents constant.

Finally a run is made with feedback but with the ranksof the relevant documents no longer being "frozen" (iteration 3).

The re-sults of these are averaged and precision versus recall graphs are drawn
for both the Document-level averages and Recall-level averages.
For ease of comparison, Tables 1 and 2 provide the normalized recalland precision for the three dictionaries and their four iterations. Graphs
1 and 2 are the precision versus recall plots for the three dictionaries
based on the Document-level averages, while Graphs 3 and 4 use the Recalllevel averages.

Graphs 1 and 3 are for the third iteration and Graphs 2 and4 are for the zero iteration.

Since iterations 1 and 2 are intermediate steps,

VII-3

their recall versus precision graphs are not included.

3.

ResultsUpon examination of Tables 1 and 2, it is seen that the merged The-saurus and Word Form Dictionary yields slightly better normalized recalls andprecisions than the Thesaurus alone, and significantly better figures thanthe Word Form alone.

The graphs also indicate this with the exception ofGraph 3 where a portion of the word form curve is higher than the combined,and of Graph 2 where the Thesaurus curve is higher for most of the values.
A survey of the 42 queries indicates that the Thesaurus alone outperforms
the merged dictionary for only two of the queries, while the Word From outperforms the merged vector in six of the instances.

For the most part, themerged dictionary yields essentially the same results as the other two, except that it outperforms the Thesaurus five times and the Word Form seventimes.
While in general, the combined dictionary seems to represent a compromise between the Thesaurus and Word Form dictionaries, a number of individual queries yield confusing results.

The relevant document ranks forquery 1 are as follows:

Relevant Document Number

Combined Rank

Thesaurus Rank

Word Form Rank

59

12715824128660

9150813293740

1

3

For documents 59, 58, and 60, the rank on the combined dictionaries are thesame or close to that of the Word Form Dictionary, while the combined rank

VI1-4

Dictionary

Iteration0

Iteration
1

Iteration
2

Iteration
3

Thesaurus
andWord Form

.8788

.9184

.9144

.9321

Thesaurus

.8733

.9070

.9119

.9321

Word Form

.8430

.8917

.8915

.8594

Normalized Recall
Table 1

Dictionary

Iteration0

Iteration
1

Iteration
2

Iteration
3

Thesaurus
andWord Form.7035

.7448

.7431

.8747

Thesaurus.6932

.7255

.7291

.8704

Word Form

.6659

.7039

.7079

.8594

l
Normalized Precision

Table 2

VII-5.0.8c .6o00

OCDA

Thesaurus andOThesaurus•

WordWordFormForm

1IA.6.8

RecallPrecision vs. RecaJ

Graphs using

Document Level Averages For Third Itera1" iGraph 1

1.0

VII-6.0'
A

Thesaurus and Word Form

OThesaurus•

Word Form.8.6.4.2.8

Recal
Precision vs. Recall Graphs using
Document Level Averages for "Zero" Iteration(Full Search)

Graph 2.0

VII-7•Thesaurus and Word FormO•

Thesaurus
Word Form

I0.4

.6

RecallPrecision vs. Recall Graphs using
Recall-Level Averages For Third Iteration

Graph 38.0

VII-8AThesaurus and Word Form

OThesaurus•Word Form.8

Reca
Precision vs. Recall Graphs using
Recall-Level Averages for "Zero" Iteration(Full Search)

Graph 4

VII-9of document 8 is lower than for the other two methods; for document 13, thecombined rank is significantly higher than that for the other two dictionaries.
The ranks for query 11 are somewhat different:

Relevant Document NumberCombined Rank

Thesaurus Rank

Word Form Rank

16116345

21

2329241

39

144

49

46

50

For this query, the combined rank ignores the word form rank and follows thethesaurus ranking very closely.

This seems to lead to problems, since theranking produced by the combined dictionary does not follow any particularpattern if the thesaurus and word form dictionaries yield radically differentresults.

However, since overall the differences between the rankings by theCRN2S and CRN2TH vectors are very small, these problems do not often occur.
A more typical query is number 36:

Relevant Document NumberCombined Rank

34

Thesaurus Rank
11

Word Form Rank
235

25

136

32

337

575

In this instance, the combined ranking produces a tighter and more accurate
ranking than either of the other two.
It is not clear whether the slightly better results of the combined
dictionary are worth the extra computer time required.

The thesaurus seems

to produce almost as good a result, and if cost is important, the thesaurus
vector alone might be used.

VII-104.

Further Studiesa) The merged dictionary might be improved by the concatenation of

all the thesaurus and only low frequency terms of the word form dictionary.
The use of low frequency terms would tend to boost the rank of documents whichcontain relevant, specific terms, thus retrieving those documents relevant to
the more important concepts of a query.

With the reduction in the amount ofword form used, the resulting dictionary would be smaller, and the execution
time and cost would be less.

Very likely, the results would not be signifi-cantly less than that of the full combined dictionary.
b) One might also experiment with weighting either the thesaurus
vector or the word form vector.

The results, while perhaps not as good as

the previous combined dictionary, might at least be more predictable and consistent .

c) Hopefully, if other collections can be located that have both athesaurus and a word form dictionary, merges can be performed on them to see
if different results appear due to a different size or type of collection.
d)Significance tests should be performed on all of these studies.

VII-11References[1] E. M. Keen, Suffix Dictionaries, and Thesaurus, Phrase andHierarchy Dictionaries, Report ISR-13 to the National Science
Foundation, Sections VI and VII, Cornell University, Department of Computer Science, January 1968.