A STUDY OF THE OVERLAP #2~NG DOCUMENT
REPRESENTATIONS i

Syracuse University, School of Information Studies,
113 Euclid Avenue, Syracuse, N.Y. 13210, USA

INTRODUCTION

ABSTP~CT

This paper presents some of the second phasedata and results of the Document Representation
Overlap Study 2 performed at Syracuse University
during the period March 1981 to February 1982.
The two phases of the study correspond to two
data bases used.
The two phases also differ in
the number of representations and the number ofintermediaries used.
The second phase investigation was designed to compare four document
representations in a portion of the PsycAbs database using performance and overlap measures. 3
This paper discusses the second phase observations and results pertaining to the overlaps
among sets of documents retrieved on different
representations.

Most previous investigations comparing theperformance of different representations haveused recall and precision as performance measures.
However, there is evidence to show that thesemeasures are insensitive to an important difference between representations.
To explain, two
representations may perform similarly on thesemeasures, while retrieving very different setsof documents.
Equivalence of representations
should be decided on the basis of similarity inperformance and similarity in the documents
retrieved.
This study compared the performanceof four representations in the PsycAbs database.
In addition, overlap between retrieved sets was
also computed where overlap is the proportion of
retrieved documents that are the same for pairsof document representations.
Results indicatethat for any two representations considered,performance values differed slightly while
overlap scores were also low, thus supporting
the evidence that recall and precision as performance measures mask differences between thesets of retrieved documents.
Results areinterpreted to propose an optimal ordering of
the representations and to examine the contribution of each representation given thiscombination.

Past studies show that representations (suchas free--text term, or descriptor phrase), when
examined using recall and precision performancemeasures, have not shown consistent results.
Further, earlier studies provide evidence that
performance measures mask systematic differencesamong representations.
Specifically, different
representations result in the retrieval ofdifferent items.
For example, the study by
McGill, et al, 1979 (ref. 3), compared documents
retrieved using free-text and controlled terms ina portion of the ERIC data base.
Users provided
queries which were searched and relevance judgments obtained.
Thirty-three of the queries wereselected for a study of overlap.
When each of
the intermediaries searched both document representations, the average overlap was only 14%.
Other queries were searched by intermediaries
using different representations.
In this situation, the average overlap dropped to 5%. Bothof these figures are surprisingly low indicatingthat users retrieve quite different sets of documents when the free and controlled representations
are used.

This is a preliminary report of work
conducted by Jeffrey Katzer, Judith
Tessier, William Frakes and Padmini
Das-Gupta at Syracuse University,
School of Information Studies (19811982).
The presentation at SIG/IR was
made by Padmini Das-Gupta.

Permission to copy without fee all or part of this material is granted
provided that the copies are not made or distributed for directcommercial advantage, the ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is bypermission of the Association for Computing Machinery. To copy
otherwise, or to republish, requires a fee and/or specific permission.

© 1983ACM0-89791-107-5/83/006/0106

Research for this study was supported in partby the National Science Foundation, Divisionof Information Science & Technology, underGrant 70-21468.
The first phase study examined seven representations using seven intermediaries in the
INSPEC data base.
Ref. 1 and 2.

$00.75106

ACM SIGIR Forum

26

Vol. 51 No. 2, July 2017

Uij =The phase of the study reported in this paper
attempted to duplicate the results in the literature using four representations and the data base
PsycAbs.

n E R i u Rj ]n

[ R i u Rj u ....... 0 R r ]Thus the union-overlap is more of a recall ratio
for a. combination of representations.
It can be
extended to combinations of more than two representations by expanding the numerator.

METHODOLOGY
Variables

Different versions of these dependent variables
were computed; they differed in terms of thestringency of the relevance criterion, where relevance was determined by the requestor.
A fourpoint relevance continuum was used from i (definitely relevant to 4 (definitely not relevant).

The key experimental or independent variable
was the representation used in searching thedata base.
The following four representations
were used.
DD - Descriptor terms chosen by an indexer~a controlled vocabulary.

The overall design can be described as afactorial design containing sixteen cells (foursearchers by four representations).
Each of the
52 queries was searched under all sixteen combinations.
This design, required that each intermediary use all representations when searching aquery.

AA - Free-text words from the abstract;
trivial words excluded.
TT - Free-text words from the title;
trivial words excluded.
II - Free-text
indexer.

Procedurephrases chosen by the

Permission was granted by the Psychlnfo Use
Service to use a subset of their 1980 data basewhose printed counterpart is Psychological
Abstracts.
Approximately 12,000 documents were
included in the subset.
Each document consistedof a series of bibliographic citation fields,
the abstract and the indexing information.

The major dependent or criterion variables wereoverlap measures.
These measures were operationalized as follows:
Assymetric-Overlap:
For two representations iand j, this measure is computed by dividing thenumber of documents retrieved by both representations by the number retrieved by one of therepresentations.
If R i and Rj are the sets of
documents retrieved by representation i and J,
then the assymetrical-overlap measure can simply
be given as
Aij =Altogether, 45 individuals served as users andsubmitted a total of 52 search requests written
in natural language to the study.
Users werefrom Syracuse University and other institutions,with information needs related to the contents ofthe data base.

n [ R i n Rj ]Four experienced search intermediaries were
employed for this study.
A three hour training
session was held.
Each intermediary was required
to submit two practice searches to the system.
Searches were conducted using DIATOM, the Boolean
online retrieval system designed to simulate mostof the features of DIalOG.

n [ Ri ]where 'n' is the counting operator.
Seen thisway, assymetric-overlap is the conditional probability of retrieval using representation jgiven that the data base is restricted to thoseretrieved by representation i.

The search procedure was started with some preliminary screening of the search requests obtained
for appropriateness to the data base and on-line
searching.
Each intermediary was given a photocopy of the search request with instructions to
conduct four searches for each query; one foreach representation.
Computer programs within
the DIATOIi system imposed a random ordering of
the representations used.
intermediaries were
instructed to carry out high recall searches.

Symmetric-Overlap:
For two representations iand j, this measure is computed by dividing thenumber of documents retrieved in common by both
representations by the total number of different
documents retrieved by either.
Or more formally,
it is the number of documents in the intersectionof the two representations divided by the number
retrieved by the union on those representations.
Sij =n [ R i N Rj ]After a query was completely searched: sixteen
times, once for each searcher-representation combination, the retrieved document set was merged
into a single listing and placed in reverse
chronological order.
This listing consisted of
citations and abstracts of the retrieved documents(if more than 200 documents were retrieved, arandom sample of 200 was used).
No clue waspresent to indicate which intermediary or representation retrieved the document.

n [ R i u Rj
Union-Overlap:
For two representations i and j,
this measure is computed by dividing the numberof documents retrieved by either of the representations by the number of documents retrievedby all of the representations.
In our case
r=4.

107

ACM SIGIR Forum

27

Vol. 51 No. 2, July 2017

Two copies of this listing were produced.
Bothcopies were sent to the user for relevance judgements.
A four point scale was used from 'I' indicating definitely relevant to '4' indicating
definitely not relevant.
All documents in thelisting were judged by the user for relevance.

reported here.
Furthermore, the design requiredthat each intermediary search each query under all
representations: the overlap results were, atbest, moderate.
It can be seen from the symmetric measures, inTable i that the maximum difference in overlaps
does not exceed 0.I0.
Also, the free-index
phrases (II) show a tendency to share more relevant documents with titles and abstract fields
than with the descriptor field -- although thesize of this overlap is still quite small.

RESULTS AND DISCUSSION
Our initial concern was to determine if theresults from this study repeated the pattern noted
earlier: relatively little difference in perform~
ance among the representations coupled withrelatively little overlap.
It is apparent thatthese results do repeat the pattern observed inother studies.
The major conclusion made from theperformance analysis is that though some performance measures are significantly different, none ofthe differences exceed .12 -- which is clearly
within the range of values reported in the literature.
Consult the NSF report (ref. 2) for a
detailed examination of the performance data.
The
overlaps range from a low of about 23% to a highof about 27%: these correspond to the earlier
results.
The remainder of this section will describe
detail the analysis of the overlap data.
Analysis

The assymetric measures indicate the proportionof documents that would have been retrieved 'anyway' -- that is, by the other representations.
For example, Table 2 reports an assymetric overlapof .378 between DD and II for the most relevant
documents.
This can be interpreted as follows:of all the documents retrieved by the descriptor
representation, approximately 38% of them canalso be retrieved by the free-index phrases.
Table 2 provides both row and column average figures (the other tables are symmetrical and asingle set of averages suffices).
A useful interpretation of the difference between row and column
averages for a single representation can be given
in terms of the sequence the representations areused in searching.
The averages of the columnsof numbers (presented along the bottom of thetable) can be interpreted in terms of being used'first' in the search process.
Given a single
representation (indicated by the column heading),
the aversge at the bottom indicates the proportionof documents retrieved by this representation that
could also be retrieved by other representations.
The averages presented in the right column are
understandable in terms of being used 'last' in
the search process.
Given retrieved documentsfrom other representations, the row average for agiven representation indicates its effect if
searching were resumed using it alone -- the lower
the average, the more the new representation willcontribute.

inof Overlaps

The simplest analysis of overlaps is pairwise,comparing each representation with every otherrepresentation.
Tables i-3 report the overlaps
for the data.
Each table contains three overlap
analysis: (i) most relevant documents, (2) all
relevant documents, and (3) all documents retrieved.
In these tables, a high value indicates greater
overlap and therefore less of the 'second'
representation.
It can be seen that the pairwise overlaps
decrease as the number of documents under consideration increase.
That is, the average overlap
is highest when only most relevant documents are
considered, it is lowest when all retrieved documents are included.
A second general finding isthat the overlap figures are lowest when overlap
is defined symmetrically, they are the highest
for union-overlap.
This, of course, is afunction of the definition of the three measuresof overlap.

Given this distinction between implementing or
using a representation 'first' or 'last', theassymetric overlaps (in Table 2) identify thedescriptors as the best choice for 'first' andfor 'last' representations.
Union-overlaps give an estimate of the combined
effect of two or more representations.
Becausethe numerator of these pairwise union-overlaps
includes all distinct documents (in the appropriate version) retrieved by the two representations, the union-overlaps will have higher values
than comparable figures for the sy~mnetrical andass~mnetrical overlaps.

The major finding in these data is that theoverlaps are quite small as indicated by theaverages.
For example, the highest symmetric
overSap among the relevant documents is only
about one-third -- .363 between AA and II.
A possible explanation for the size of overlaps
is searcher difference.
Analysis of variancecomputations (see the final report ref. 2) showedthat searcher effects occasionally account forsignificant portions of the variance.
However,the ranking study, conducted by McGill et al (ref.
3), casts doubt on the contention that searchers
are the sole or major cause of the low amount ofoverlap.
In the ranking study, overlaps between
different representations searched by the samesearcher only equalled 14% for retrieved documents.
That figure certainly falls in the range of values

The union-overlap figures in Table 3 show thatmost pairs of representations achieve higher than
50% recall levels.
The combination of descriptions
and abstracts given over 80% of the most relevant
documents and over 75% of all documents retrieved.
Union-overlaps are one way to explore the 'marginal utility' or the 'value added' of additional
representations.
Table 2 provides only pairwise
overlaps.
The extension to more than two representations is necessary in order to get overall

108

ACM SIGIR Forum

28

Vol. 51 No. 2, July 2017

conclusions.
The next section of this paper
takes this approach.
Descriptive

document representations is the sum of the uniquecontributions (Table 5).
This total is about 58%.
Thus the amount of overlapping documents is about
42%.

Models of Overlap

Going beyond pairwise overlaps, the question ofconcern here focuses on the relationship among allof the representations: what is the optimum ordering of representations?
That is, if a retrieval
environment were limited to a single representation, which one would it be? If a second one
could be added, which of the remaining representations contributes the most over and above theeffect of the first representation?
A thirdrepresentation could be added over and above
the first two, and so on.

To conclude, the four representations examineddiffered little in performance scores.
Further,the overlap measures between pairs of representations were low, therefore supporting the earlier
literature where performance measures were observed to mask differences among the sets of documents
retrieved by different representations.
Low overlaps were also indicated by the unique contribution figures for each representation.
Within the constraints of this study, a closer
examination of the overlap data revealed an
optimal ordering of the representations in termsof relevant documents retrieved.
This study provides evidence for the importance of overlap
measures in the comparison of representations.
In
addition, it identifies the descriptor field as
an effective representation (within the PsychAbs
data base) with respect to the representations
examined.

The most sensible measure to use in answering
this question is based on the union-overlap.
The
result of this analysis is a model depicting anordering of the representations.
Table 4 presents
this model.
Specifically, given the four representations, the model identifies descriptors as thebest 'first' representation and abstracts as thebest 'second' representation when descriptors isthe first representation.
Titles rank the best
'third' representation by contributing most over
and above descriptors and abstracts, while
identifiers is selected as the last representation
given the first three.
Such a model, if consistent, would allow asearcher to know which combination of fields
would be most likely to retrieve relevant documents.
Such models would also point to obvious
economies in the design and operation of retrieval
systems.

REFERENCES

An interesting result that shows up from Table 4is that the order does not change as a function of
relevance stringency.
What appears to be highly
consistent is the cumulative increase in the percentage of relevant documents accounted for as
each additional representation is included.
This
similarity may simply be due to the fact that themodels are based on highly interrelated data.
The overlap among document representations canalso be viewed from the perspective of a representation's 'unique' contribution.
For a given
representation, what documents does it contribute to the relevant retrieved that were notretrieved under any other representation?
Thequestion is equivalent to the observed improvements in the model when the representation is thelast one entered.
Table 5 reports the effect ofeach representation, assuming the representationentered the model first or last.
These are themaximum and minimum incremental improvements
for each representation.
The 'unique' effectof each representation is reported as the minimun contribution.

11.

Katzer, Jeffrey, et al. A Study of theOverlap Among Document Representations.
Information Technology, 1982.
i, pp. 201-273.

2.

Katzer, Jeffrey.
A Study of the Impact ofRepresentations in Information Retrieval
Systems.
Final report for Grant NSF-IST79-21468 to the National Science Foundation,
July 1982.

3.

McGill, Michael J. et al. An Evaluation of
Factors Affecting Document Ranking by
Information Retrieval Systems.
Final Report
For Grant NSF-IST-78-10454 to the National
Science Foundation, October 1979.

The lack of overlap among representations isagain evident in the unique percentages.
Given
the data base and the four representations, thefourth representation can contribute a sizeable
number of additional documents -- approximately
25% for the DD representstion.
One final indicator of the lack of overlap among

109

ACM SIGIR Forum

29

Vol. 51 No. 2, July 2017

TABLE 1
Symmetric Pairwise Overlaps

II

DD

AA

TT

AVG *

0.363
0.273
1.000
0.277

0.351
0.264
0.277
1.000

0.334
0.275
0.304
0.297

0.319
0.233
1.000
0.256

0.328
0.234
0.256
1.000

0.305
0.245
0.269
0.273

Version - Most R e l e v a n t
II
DD
AA
TT

1.000
0.289
0.363
0.351

VersionIIDD
AA
TT

All R e l e v a n t

1.000
0.269
0.319
0.328

VersionII

DD
AA
TT

0.289
1.000
0.273
0.264

0.269
1.000
0.233
0.234

All D o c u m e n t s

1.000
0.199

0.199
1.000

0 .1820.150

0 .2150.159

0 .1990 . 169

0.182
0.215

0. 1500 . 159

1.000
0.127

0.127
1.000

0.153
0.167

*Averages w e r e c o m p u t e d w i t h the diagonal element omitted.

110

ACM SIGIR Forum30

Vol. 51 No. 2, July 2017

TAB LE 2
As_~=ymetric Pairwise Overlaps

II

DD

AA

TT

AVG *

0.469
0.452
1.000
0.364
0.428

0.551
0.551
0.536
1.000
0.546

0.437
0.413
1.000
0.351
0.401

0.523
0.500
0.485
1.000
0.503

0.439
0.479
0.458
0.375

0.264
0.256
1.000
0.178
0.233

0.394
0.364
0.307
1.000
0.355

0.316
0.337
0.315
0.240

version - Most Relevant
IX
DD
AA
TT
AVG*

1.000
0.552
0.616
0.491
0.553

0.378
1.000
0.407
0.336
0.374

0.466
0.518
0.520
0.397

Version - All R e l e v a n t
II
DD
AA
TT
AVG*

1.000
0.524
0.54
0.468
0.511

VersionII
DD
AA
TT
AVG*

*

0.357
1.000
0.348
0.305
0.337

All Documents

1.000
0.39
0.371
0.321
0.361

0.289
1.000
0.267
0.220
0.259

Averages w e r e c o m p u t e d w i t h

the diagonal element omitted.

** The representations in the columns
the overlap measure.

form the d e n o m i n a t o r ofIII

ACM SIGIR Forum

31

Vol. 51 No. 2, July 2017

TABLE

3

Union pairwise

II

DD

Overlap~

AA

TT

AVG *

0.640
0.821
0.495
0.651

0.528
0.701
0.651
0.336

0.629
0.747
0.704
0.627

0.624
0.806
0.454
0.624

0.525
0.704
0.624
0.329

0.621
0.742
0.685
0.618

0.640
0.753
0.442
0.619

0.469
0.587
0.619
0.256

0.575
0.652
0.671
0.558

version - M o s t R e l e v a n t
II
DD
AA
TT

0.377
0.719
0.64
0.528

VersionII
DD
AA
TT

All

0.719
0.550
0.821
0.701

Relevant

0.368
0.715
0.624
0.525

0.715
0.539
0.806
0.704

Version - All Documents
II
DD
AA
TT

0.314
0.616
0.640
0.469

* Averages were

0.616
0.424
0.753
0.587

computed withthe d i a g o n a lelement omitted.

112

ACM SIGIR Forum

32

Vol. 51 No. 2, July 2017

TABLE

4

Representations Orderedb_y I n c r e m e n t a l I m p r o v e m e n t

ist

2nd

3rd

4th

Representation

DD

AA

TT

II

Cum.

No.

339

506

573

616

Cum.

Percentage

.550

.821

.930

1.000

Representation

DD

AA

TT

II

Cum.

Docs.

871

1302

1489

1615

Cum. P e r c e n t a g e

.539

.806

.922

1.000

Order

Most Relevant

Docs.

All R e l e v a n t

No.

113

ACM SIGIR Forum

33

Vol. 51 No. 2, July 2017

T/d~LE 5
M a x i m u m and M i n i m u m C o n t r i b u t i o n s_of F o u r R e p r e s e n t a t i o n s

Repr.

Maximum Contribution*
No.Docs,
Percent**

Minimum Contribution*
Percent**
No.Docs,

Most Relevant
AA
DD
II
TT

310
339
229
210

.475
.520
.351
.322

112
158
42
50

.172
.242
.064
.077
.555

728
870
579
518

.440
.526
.350
.313

286
429
120
131

.173
.259
.072
.079
.583

All R e l e v a n t
AA
DD
II
TT

* M a x i m u m c o n t r i b u t i o n is the e f f e c t of t h a t r e p r e s e n t a t i o n alone-e i t h e r it is the sole r e p r e s e n t a t i o n in the data b a s e or it w a s
used (entered) first, b e f o r e the o t h e r s are used.
Maximum contrib u t i o n is t h e r e f o r e e q u i v a l e n t to m i c r o - r e c a l l .
M i n i m u m c o n t r i b u t i o n is the "unique" e f f e c t of that r e p r e s e n t a t i o n
a f t e r all d o c u m e n t s r e t r i e v e d b y the o t h e r three r e p r e s e n t a t i o n s
h a v e b e e n r e m o v e d ; thus, it can b e c o n s i d e r e d to h a v e e n t e r e d thes e a r c h p r o c e s s last.
* * P e r c e n t a g e s are b a s e d on all d o c u m e n t s r e t r i e v e d b y allr e p r e s e n t a t i o n s in each category.
Here the numbers are 652
m o s t r e l e v a n t and 1653 for all relevant.

for

114

ACM SIGIR Forum

34

Vol. 51 No. 2, July 2017