- 106 -CHAPTER 7
Additional Teats
The first year of the project coincided with a time when a number of groups,
who had been investigating various methods of statistical association, were becominginterested in the possibility of putting their methods to the test, and we received
some enquiries regarding the possibility of the project test collection being used asa 'common sample 9 . All such groups were, of course, working with computers, sowith the agreement of the National Science Foundation, it was arranged that a tapeshould be prepared of the indexing for the 1400 documents in the test collection.
This was done by I. B. M. (U. K.) Ltd., and an example of the printout for document
1420 is given in Appendix 8 . 1 .
In the end, for various reasons, none of the groups in America was able to
make use of these tapes. However, in England, Drs. Soger and Karen Needham
decided to use the Cranfield collection for a test of the 'clumping' technique developed
at the Cambridge Language Research Unit. (ref. 38). Since the computer to be used
was the Atlas, it was necessary to prepare a set of paper tapes from the punchedcards. The problems involved in this are not for us to relate, but the indexing has
now been completed, and a copy of the printout for document 1420 is also given in
Appendix 6 . 1 .
At a later stage in the project, when the results were coming through, a meetingwith Professor Salton made it clear that the research which he had been undertaking at
Harvard was basically along similar lines to the work at Cranfield, in that both groups
were concerned with comparing the performance of various index language devices,
The difference lay in the methods adopted for the clerical processes of the testing, andthe SMART programme (ref. 30) gave the flexibility of rapid testing of any set of
documents for which the necessary relevance assessments had been made in relation
to a set of questions, so long as these were in a subject field for which suitablevocabularies had been prepared. The original testing of the SMART programme had
been carried out on a collection of abstracts dealing with computers, and for bothgroups the prospect of using the programme to test the subset of documents taken fromthe Cranfield project was very attractive. For Professor Salton, it gave the opportunityof testing his programme in a different subject area; for us it opened up a completely
new field. There would be the. opportunity for directly comparing the results of thedevices being investigated at Cranfield with the similar, but more complexly calculated,devices used at Harvard. Secondly, there was the possibility that it would assist in
solving some of the interesting problems involved in the presentation of results. The
recall-precision curves, based on a series of cutoffs, were producing at Cranfield
quite different figures from the normalised recall and normalised precision based on
the ranked output at Harvard. This was only to be expected, since the method of
calculation was so different, but it was important to be able to find how to equate thedifferent sets of figures. The final point of interest was that though the Harvard
searching was normally done on document abstracts, the flexibility of the SMART
programme made it practical for the searches to be carried out on both the abstracts
and the indexing which had been done at Cranfield, thus providing for the first time
a comparison between searches based on abstracts and on indexing.
A member of the Cranfield group spent a week at Harvard, and as a result of
the visit, it was arranged that a subset of the collection, consisting of 200 documents
and 42 questions, should be processed at Harvard, and that searches should be made

- 107 -on both the indexing of these documents and also their abstracts. Subsequently
the decision was taken to extend this work so as to cover the whole of the Cranfieldcollection.
Citation Indexing
It was in 1961 that the first major grant was given for a citation index (ref. 32),
and the following year we were asked for our views on how a citation index could be
evaluated. Citation indexing is basically a method of forming classes of documents
which are all related through a common reference to a base document. There will,of course, be many occasions when the class consists of only a single entry; however,in the more numerous cases where the class consists of two or more documents,
then citation indexing can be considered equivalent to bibliographic coupling at astrength of one. Bibliographic Coupling has been developed by Dr. M. Kessler at
the Massachusetts Institute of Technology (ref. 20), and in relation to citation indexing,can be considered as a precision device t since it progressively narrows the class of
documents as the demand for common references increases in number. Citationindexing and bibliographic coupling could therefore be tested in the same way as anyother device; it was, however, necessary to prepare an index for this purpose.
The first stage was to prepare xerox copies of the citations in the 1400 documentsin the test collection; against each citation was put the code number for the citing
documents, after which each citation was cut up so as to appear on a separate slip.of paper. This resulted in some 20,000 slips of many various sizes, which had tobe sorted into author alphabetical order. This being done, the slips were pasted
onto sheets erf paper; where two or more slips related to the same cited document,
only one example was pasted in; the references to the additional citing documents
were entered alongside. This can be seen in Fig. 7 . 1 , which covers a series ofreferences to papers by H.J. Allen, in particular a paper written with A.J. Eggers
entitled 'A study of the motion and aerodynamic heating of missiles entering the earth's
atmosphere at high supersonic speeds. 1 (NACA TN 4047). This is shown to havebeen cited by thirteen papers in the test collection.
This procedure resulted in a normal citation index; to obtain the index for
bibliographic coupling required three further stages. First, each cited reference
having two or more citations was given a code number, the paper by Allen and Eggers
being A2S, and a separate card was prepared for each cited reference. On thisreference card was written the code for the cited document ( i . e . A25) and then, in
numerical order the codes for the citing documents. Fig. 7.2 illustrates thereference card prepared in connection with the paper by Allen and Eggers shown in
Fig. 7 . 1 .
The reference cards were sorted into numerical order depending on the lowest
number on each card. Since these numbers represented the codes for the citing
documents, they ranged from 1001*2400. Each card was then taken in turn, and theinformation from all reference cards having the same starting number was transferred
to a master card. As an example, the master card shown in Fig. 7.3 illustrates the
position with regard to document 1067, this number being poated in the top left handcorner. In the column headings are entered the code numbers for the documents whichhave been cited by document 1067, this information being obtained from the reference
cards such as Fig. 7.2, and these being, in this particular case, A25, W32, A23, E23,
F90, and 024. In the first column of the master card are entered the document numbersof all other citing papers, this information being again obtained from the reference
cards. A tick is put against each number in the column under the appropriate heading- 108 -•Aliens D. J.: The Application of Multhopp'e Subsonic Lifting Surface Theory to the Calculation of the Aerodynamic Farce* /fitina
on a Wing of Finite Aspect Ratio Oscillating in Arbitrary IlA4ic
Modes With Control Surface Freedom, Design Dept. Rep. Ho. 1191,• Havker Aircraft, Ltd. [Kingstem-on^Tnames, KnglandJ, June 19??.'j

D. N. de G, Allen and R. V. ' Relaxation methods applied to determine the motion, in two dimensions, ofSouthwella viscousfluidAfft&Rd cylinder. Quart. /. idtch. App. Math. III.
Par_t_2. June, lv55.
__\°z »*>.

'• Altai, f. J., An BJntU-Piastic Thtvy^JkeRiipont* tff
CanHUmt h Air Blast Loading BRL MR fc^WbflpriUWS.

Alien, BY J . t General !m«ory of Aiyftoil Sections having arbitrary Shapeor Pressure^^ g i j i t r ^ t i o n , NACA fl*eJu Kept, .'Jfr.t 8?3# 19^7. 1 * * *

\
J

fab
'

(&**
\!Ss

Allen, H. JUliant Motion of * Ballietio Missile Awrularlv M*«m™^

\f^>7

W&

4. Alton, H. Julian: A Simplified MoUioil tor iho Calettletleii of Airfoil PrwwMro DlatribuUon. K A & i S r W m 708,1030.

3. Allen, H. Julian: Estimation of the Forces and Moments Acting on
Inclined Bodiee of Revolution of High Fineness Ratio> KACARM A9I26, 19^9. T V * *8. Allen, H. Julluu: Calculation or 4ho ChonU'Jrfo l»a<l DJMirilnttioiVover Airfoil taction* w i t h / W o , Sj>ft,J«tt Jbrl&Wy l l l i w l• j h y.Trftinifjt-lSciiio F1«|M. X'ACA Ht»j». N<>. «!»-#. Ifl3«.

"

Alien, H* Julian, and Kggers, A* JY, Jr.x A Study of the Motion and
Aered^naaio Seating of mssiles Bfcfcering the Earth** Atmosphere at
Higft Supersonie Speeds. KaCA.W ^ V J $ ? 7 »V*?..
«Allen. HHrri^n* Jr. and PteM*«r. «• ^ST^Zj^
7-WMIH/. NASA MKMO1-HWJOK. 2 2 0

H. J. Allen,
M. A. Heaalet andG. B. Nitsberg:

V

5he interaction o f boundary layer andoocjoresala* shook and i t * effeol^ ^pon
airfoil pressure distr±bujeto«L| \. *+A-*0,A. HMA7A02 ( l 9 f c 7 ) J * "

(OJO

tn'^d

7433 / 3 7 3f^a?)

Allen, Jl, Julian, and Kitsberg, OeraldB.i ' W w t a - W « m r e s 6 i >Uity on the Orovtb of « * I****** Boundary S ^ e T l S S a T
Win^andBodi#t, SAOAA0»,^aT^^

i ASIM, irjulia^a^^
Of VtaooiUy on lUc Mow OverHlomtor Incline UocHc* o/ HaveluUon. N A ^ g * JO**, Iflfrl. (Httpenwileii KAOA f K

' , /
/#£»

T , y,/4*P¥, ^ - J C I
V' * * *

Allen, H, Julian, and Perkins, IdwerdW.j Charecterfaticfll&:]tim
Over Inclined Bodies of Revolution. KACA RM A3010T, *|Jft. f f f 3
in » lWDniuaiekmal Flow Wind Tunnel, wltlj Cen> ;aldoration of the Kflbct of Oonnm«-ibaity,M HJiX)A>t •

' ^J
/3#3

l4 Allan, R.AV Keek, J,C. end Cama, J.C., M0he Reoowbinetionof Kitrogen at otOO K, ^yoe^&verett Research Lab., Research
Note 2*3, June 1961. ( 0 9 1

FIGURE 7.1

CITATION IN&fcX SHEETV^^". \llQ7 : fy2&/ f 23 22/3

M %f)V„„,. ,^/

(frig)

-109 r

•

\A25

1

I0h~f

101-}

vte

nt>k

\jV5.

niq
jq&z

Wis

iSlt>

too%

*Hb 1

\<jl&

\

2*J<f

FIOURE T.| CITATION INDEX REFERENCE CARD

ijon-ji ft ftp; vs/ai [ ! ^?t ;1

1 ' ' • # i vj
nbl\
>i>B I V •

'I'l
the

•

•

•

g

.jr

.:'v

•

•

.-••;;

,/
y/

•

)St%

•

'.iff,•,ijpjr! '*HH

•
•
i/

/

'7**7

\*n
\
tith \•,^..;..
/ *^^yxjijfelllilJI.IIMIilllliil.Wit I Hi J

»V/bi""!"1'': : '« ''* l»"""*H Vfll'.'.'i 'M*M<« * i i|

™M
2l4F J

r—'—T""1

V

y/

It™

^t^p;]•••"""• J " r

I

[liW

™i
nPr

•

•

-

fib

liZr

V

#

\iny ',

HMMMnanBtai

i 'V
•

^^*?(

f

^PJ^B^W^^W

i^

-X001

l

| , f # •-!.

V

••

tf ib
j ifik
•tWs,

£ 23

•

V
•

y

>/
•

|

•

•
•
"

•'•mi " iI M J

> l . m i , > . [ iiii< l m.i.<i«i.i«ij- l ? V U ';'«UlM|ll«l|aMIII|ll« . 1

FIGURE 7.3

CITATION INDEX MASTS* CARDi'.

- 110 -to Indicate that it cited this particular reference. However, when a document number
appears in connection with another cited reference, the number is not repeated, but
a tick is put in the appropriate column. For instance, in Fig. 7.3, it can be seenthat this document has two references in common with document 1163, 1164, 1639
and 1716, three references in common with document 2379, and four references incommon with document 1715. To return to Fig. 7 . 2 , when document 1067 had beenentered, then this number was crossed off and the reference card was re-sorted in
the pack under the next number, namely 1077; again this number was crossed off
when the master card had been entered for document 1077, and the reference card
re-filed on the next number and so on until all the document numbers had been entered.
The final stage was to go through the master cards and prepare the bibliographic
coupling card (Fig. 7.4). This showed the master document and all the otherdocuments with which it had two or more references in common.
It is clearly a matter for argument as to how a citation index should be tested
operationally, but within the context of these experimental investigations, it was
relatively simple to decide on the method to be used. Our concern was how acitation index operated in regard to recall and precision and the procedure adopted
was as follows. For a certain question, the relevant documents were known as wellas their relevance level. The numbers of the relevant documents were written acrossthe score sheet as shown in Fig. 7. 5, referring to question 34. In order to avoid
complexity, a fairly simple example has been taken, where there were six documents
all of relevance 3.
The numbers in the left hand column indicate the coupling strength going froma maximum of 6+ down to 1, which latter represents citation indexing. The
appropriate bibliographic coupling cards were then taken from the pack, the first ofthese relating to document 1067. As can be seen in Fig. 7.4, document 1715 had amatch of 4 with document 1067; there were no other documents at this level of match,and since document 1715 is also a relevant document to question 34 (see Fig. 7.5),
this is counted as a success and the score is entered appropriately. t The document
which matches at a level of 3 is not relevant, so this now makes the score one relevant
and one non-relevant. At a match of 3, three of the documents are relevant (1164,
1639 and 1716), so the score here becomes four relevant and two non-relevant. By
referring to the cards shown in Fig. 7.3, we can calculate the number of documents
involved with a stogie match. There are no other relevant documents in this set,but many non-relevant, so the score for this is shown as four relevant and thirty-two
non-relevant.
This process is repeated for all the other relevant documents, as shown in
Fig. 7.5. When this has been done/ the scores can be totalled to give a set of figureswhere obviously the maximum recall and the lowest precision will be obtained at amatch of 1, and maximum precision with lowest recall obtained at a match of 6+.
However, there are Various approaches that can be taken in compiling the score, andthese will be considered in the volume of tept results. Such analysis was, done fordocuments of all degrees of relevance*
In bibliographic coupling as discussed by Kessler, account is only taken of theactual match rather than what might be called the proportional match. For instance,
twp review-type articles may each have fifty references, as against two other papers
which have only three references. If the former pair of papers have five referencesin common, this would be considered a stronger coupling strength than the latter pair-Ill -robi
3z4
\ libZ -2379 (7/5

~|

nib

PIGUBE 7.4 BIBLIOGRAPHIC COUPLING CABDRE LEVANT DC CUMEIN TS
1164

1067

COUPLING
STRENGTH

R

1715

1630

N-R

R

N-R

R

N-R

1717

1716

R

N-R

R

N-R

R

N-R

6+
5+
4+

1

0

0

1

1

0

3+

1

1

0

2

1

0

1

0

1

0

2+

4

2

1

6

a

3

1

5

2

2

0

3

1+

4

38

1

49

2

63

1

47

2

33

0

18

SCORE
RELEVANT

6^
4+

a

5+

NON-RELEVANT

RECALLRATIO

PRECISION
RATIO

-

-

-

1

33%

66%

3+

4

3

66%

57*

84

5

19

83%

21%

1+

5

301

83%

2%

FIGURE 7.5

nwiw
3.
7.

SCORE SHEET FOR BIBLIOGRAPHIC COUPLING
FOR QUESTION 34.

1

nis fa)n*(T)u.

3t37«?£n)it\b.

1)43 (<?)•ii«?fi')

Jtff.

iMl'T)FIGURE 7.6 RECALCULATED BIBLIOGRAPHIC COUPLING CARD
Figures in bracket* rtprotnt number of referencesin document*. Figure* in fire* column give thecalculated weighting.

- 112 -which have, for example, two references in common. However, proportionately,
it could be argued that the latter represents a stronger match than the former.
To test this, the number of references in each document of a matching pair were
multiplied, the resultant figure was then divided by the square of the number ofmatches and the final figure was considered as the level of coupling strength.
For
example, document 1067 (see Fig. 7.4) had six references, and document 1163 hadeight references, giving a multipland of 48. These had a match of 2, so dividing by 2 8gives a final weighted figure of 12. Document 1715, however, had nine references
which, combined with document 1067, gives a multipland of 54. In this case, since
there is a match of four, this figure has to be divided by 4*, giving a final weighted
figure of 3. When the matches for document 1067 had all been worked out, the weighting
becomes as in Fig. 7.6. In many cases, the result of this exercise showed significant
changes in coupling strength, and therefore the collection was re-tested in the manner
described earlier, only this time the scoring was based on these new coupling levels.