The SPIRIT collection: an overview of a large web collection
Hideo Joho and Mark Sanderson
Department of Information Studies
University of Sheffield
H.Joho|M.Sanderson@sheffield.ac.uk

Abstract
A large scale collection of web pages has been essential for research ininformation retrieval and related areas. This paper provides an overview
of a large web collection used in the SPIRIT project for the design andtesting of spatially-aware retrieval systems. Several statistics arederived and presented to show the characteristics of the collection.

Introduction

SPIRITi - Spatially-Aware Information Retrieval on the Internet - is a research project fundedthrough the EC Fifth Framework Programme. The main aim of the project is to design andimplement a search engine to find documents and datasets on the Web relating to places orregions referred to in a query. (Jones, et al., 2002). One of the resources the project is exploiting
is a large scale static web collection. While the Internet is dynamic in nature, a static collection
enables researchers to analyse retrieval systems with consistent data.
The web collection used in the SPIRIT project was originally crawled by Dr. Clarke and hiscolleagues at the University of Waterloo in Mid-2001 (Clarke, et al., 2002), and distributed to Prof.
Fox at Virginia Tech. We obtained a version of this crawl from Prof. Fox which constitutes thedata in our collection. It is our understanding that the crawl was bootstrapped by a set of
educational web sites.
At Sheffield, we processed the data set so that every web page is tagged in a standard format
based on the Web Track of TREC (Craswell, et al., 2003). Our version of the collection has beenused not only by the SPIRIT project but also by the researchers at Dublin City University, theUniversity of Glasgow, and the University of Brighton (e.g., Gurrin and Smeaton, 2004, Cacheda,et al., 2004). Our version is sometimes referred to the SPIRIT web collection.
This paper will show a sample document along with the tagging information, followed by a set ofstatistics derived from the collection.

Sample document
A sample document of our collection can be found in Figure 1. Every document starts with a<DOC> tag and ends with a </DOC> tag. <DOCNO> is a unique document ID, <DOCURL>ii is theoriginal URL of the document, and <DOCHDR> contains the URL and the length of the original
data in bytes followed by the HTTP response of the crawled web server. The rest of the data is
the document contents.

<DOC><DOCNO>SPRT-022-021-086-0042537</DOCNO><DOCURL>http://www.shef.ac.uk/~is/</DOCURL><DOCHDR>
http://www.shef.ac.uk/~is/3875
HTTP/1.1 200 OK
Date: Wed, 23 May 2001 22:03:00 GMT
Server: [removed]
Cache-Control: max-age=7776000, public, must-revalidate
Connection: close
Content-Type: text/html</DOCHDR><!DOCTYPE HTML PUBLIC "-//SoftQuad//DTD HoTMetaL PRO 5.0::19980907::extensions to HTML 4.0//EN""hmpro5.dtd"><HTML><HEAD><TITLE>Sheffield - Department of Information Studies</TITLE></HEAD><BODY BACKGROUND="gfx/g_back01.jpg" VLINK="#CC0000" LINK="#000099">[omitted]</BODY></HTML></DOC>

Figure 1. Sample document (DOCNO: SPRT-022-021-086-0042537)

Collection statisticsiii
The total number of documents in the SPIRIT collection is 94,552,870 on an approximate size of
1 TB. A comparison with existing document collections is given in Table 1. The period in the first
three collections is the date of the documents published, while the period for the last fourcollections is the date of the crawling.
Collection
Cranfield
CACM
TREC Disk 1-5
TREC .GOV
NTCIR-3 Web
TREC .GOV2
SPIRIT

PeriodNo. of docs
1922-1963
1400
1958-1979
3204
1987-1994
1,634,234
Early 2002
1,247,753
2001
11,038,720
Early 2004
25,205,179
Mid 2001
94,552,870
Table 1. Comparison of the collection size

Size (MB)
1.6
2.2
5,321
18,000100,000
426,000
1,000,000

The distribution of document size is given in Table 2. The size in bytes is of the original web pages,
while the number of words is based on the actual text displayed by a browser (lynx was usedhere). Words were counted as character strings separated by whitespace. In our collection, thesize of a displayed text tends to be approximately a third of the original HTML page.
Size(byte)
10
20
50
100
200
500
1,000
2,000
5,000%

Size%
No of%
No of(byte)
words
words
2.730910,00019.3494
10<0.0000
5,000
0.003820,00018.910910
6.616010,0000.018950,00014.561720
8.431020,0000.1532100,000
2.291350
9.181750,0001.4795200,000
0.49891009.3134
100,000
9.6051500,000
0.1751200
16.0338
200,000
4.12471,000,000
0.0389500
25.1390
500,000
7.32582,000,000
0.01661,000
13.7170
1,000,000
18.7087
5,000,000
0.00652,000
6.6723 >1,000,000
Table 2. Size of documents in the SPIRIT collection%
3.47890.93790.31780.12210.02540.01040.00300.00030.0000

The distribution of the 20 most frequent top-level and second-level domains is shown in Table 3,along with the data provided by Internet Software Consortium (ISC) which are based on 169
million pages as of July 2001 (a similar period to our collection). Although the rate of portions
varies, the top-level consists of a reasonably similar set of domains in both cases, while thesecond-levels are more scattered.
SPIRIT Mid-01
ISC Jul 01
SPIRIT Mid-01
ISC Jul 01
TopTopSecondSecondDomain
Domain
Domain
Domain
%%%
com
42.29
com
36.12
co.uk
1.44
lucent.com
edu
17.05
net
24.76
ac.uk
0.95
aol.com
org
10.17
edu
4.38
co.jp
0.70
uu.net
net
4.82
jp
3.54
edu.au
0.47
ne.jp
us
3.05
ca
1.87
rootsweb.com
0.45
home.com
uk
2.75
uk
1.68
ne.jp
0.44
rr.com
gov
2.42
de
1.46
com.au0.43
pacbell.net
de
2.35
us
1.32
amazon.com
0.42
t-dialin.net
jp
2.15
it
1.21
yahoo.com
0.40
avaya.com
ca
2.00
mil
1.21
lycos.com
0.38
qwest.net
au
1.30
au
1.13
ac.jp
0.37
co.uk
fr
0.78
nl
1.07
or.jp
0.36
genuity2.net
it
0.60
org
0.86
sun.com
0.32
dialsprint.net
nl
0.47
fr
0.84
geocities.com
0.32
hinet.net
ru
0.46
tw
0.77
tripod.com
0.32
ac.uk
se
0.43
br
0.63
nasa.gov
0.27
att.netch0.42se0.62
amazon.de
0.27
splitrock.net
es0.33
es
0.56
ca.us
0.25
net.tw
mil
0.32
gov
0.52
aol.com
0.24
ad.jp
cn
0.32
fi
0.52
uiuc.edu
0.22com.br
Table 3. Top and second level domains, compared with the data from ISC in Jul 2001
(Source: Internet Software Consortium (http://www.isc.org/))

%
3.20
2.65
2.61
1.97
1.86
1.36
0.84
0.81
0.80
0.73
0.68
0.64
0.60
0.60
0.58
0.53
0.51
0.48
0.45
0.45

European domains are of most interest for the SPIRIT project. A total of 22 European domains
were found in the 50 most frequent domains. The European domains consist of over 9.6 million
web pages which is approximately 10.24% of the entire collection. The distribution of European
domains is shown in Table 4. As can be seen, nearly half of European pages are of uk or de.
However, there are between 34,000 and 730,000 pages for the rest of 20 European domains.
CountryUnited
Kingdom
Germany
FranceDomain%Country

ukdefr

26.87822.9807.635

Italy
RussianFederation
SwedenSwitzerland
Spain
NorwayFinlandit

5.819

rusechesnofi

4.487
4.238
4.060
3.176
3.0782.917

Austriaat

Denmark
PolandBelgium
Czech
Republic

Ireland
Portugal
Hungary
GreeceTurkeyEstonia
Slovak
2.388 Republic
Table 4. European domains (N=9,682,678)

Domain%

dk
plbe

2.187
1.838
1.711

cz

1.398

ie
pt
hu
gr
tr
ee

1.292
0.829
0.828
0.778
0.604
0.525

sk

0.353

Table 5 shows the 20 most frequent character sets defined in a meta tag of web page. As can beseen, most pages do not define the character set. Of those defined, iso-8859-1 andwindows-1252 are the dominant ones.

Charset%
Charset(Empty)71.876 windows-1250
iso-8859-1
18.558 iso8859-1
windows-1252
6.221 euc-kr
gb23120.609 iso-2022-jp
shift_jis0.506 euc-jp
x-sjis
0.459 windows-1256
windows-1251
0.194 us-ascii
big5
0.182 windows-1254
utf-8
0.172 windows-874
iso-8859-2
0.158 x-euc-jp
Table 5. Charsets defined in a meta tag

%
0.137
0.125
0.108
0.077
0.059
0.045
0.037
0.033
0.032
0.031

Table 6 shows the distribution of URL length and depth. URL length is the number of characterswithout URIs (e.g. http://). For example, the length of x.com is 5 and www.google.com is 14.
URL depth is the number of hierarchies where the web page is stored. For example,
www.shef.ac.uk, www.shef.ac.uk/, and www.shef.ac.uk/index.html are counted as
1, while www.shef.ac.uk/~is/index.html is counted as 2.

The table shows that approximately half of the URLs fall into the length between 31 and 50
characters. As for the depth, the majority of URLs consist of up to four levels of the hierarchies
with the second as the most frequent level.
Length5<510152025
3035404550%
Length%Depth0.000055
8.959910.000260
6.487020.073265
4.419730.947770
2.848942.185575
1.917854.228680
1.36666
8.310685
1.0130712.6004
90
0.6561815.1254
95
0.4518914.5864100
0.32091012.0536>100
1.4463>10
Table 6. URL length and depth%
18.380928.329924.2538
14.8797
7.50393.52301.60650.89540.23510.12310.2086

Table 7 shows the number of <IMG> tags defined in web pages. Nearly two in three pages
contains at least one image tag, and one in four pages contains more than 10 image tags. Of
those pages with the image tags, approximately 70% define at least one ALT attribute for thealternative text information.
NumberIMG (%)
ALT (%)
033.2830.57
529.74
40.29
10
10.81
11.49
20
11.09
9.46
50
10.42
6.40
100
3.70
1.45
>100
0.96
0.33
Table 7. Number of images and ALT attribute

Linkage analysis
Linkage information is an important factor when examining web document collections. The
density of outgoing and incoming links from on-sites and off-sites is used to analyse the structureof web collections. Gurrin and Smeaton (2004) compared the linkage density of several web

collections including the SPIRIT collection. Their analysis showed that the density anddistribution of outgoing links of SPIRIT documents are very close to an estimate of the level on
the Web (i.e. 4.9 off-site links, 14.2 on-site). The density of incoming links from off-sites is 1.24
which is lower than .GOV collection (1.98). The ideal level is estimated to be 4.9. However, theirexperiment suggests that a subset collection of 1.5-2 million pages with the ideal linkage density
can be generated from the SPIRIT collection.

SummaryIn this paper we have presented the SPIRIT web collection and a number of statistics derived
from our initial analysis. The collection appears to be a useful resource for those who require
geographically more heterogeneous data than existing web collections. The sampling of the real
web is not a trivial task. A realistic approach to understand the structure of the Web would be togather information from a number of different samples. The authors hope that the data provided
in this report will contribute to such an approach.

Acknowledgments
The authors wish to thank Dr. Charlie Clarke for the conducting original crawl, and Prof. EdwardFox for giving us an access to his copy. The authors also wish to thank members of the SPIRIT
project for the ideas of “what to count”. Financial support for the work was provided by the EC 5thFramework RTD project SPIRIT: contract number IST-2001-35047.

References
Cacheda, F., Plachouras, V. & Ounis:, I. (2004). "Performance Analysis of Distributed
Architectures to Index One Terabyte of Text". In: McDonald, S. & Tait, J. (eds.), Advances inInformation Retrieval, Proceedings of the 26th European Conference on IR Research, Lecture
Notes in Computer Science, Vol. 2997, Sunderland, UK. pp. 394-408. Springer.
Clarke, C. L. A., Cormack, G. V., Laszlo, M., Lynam, T. R., and Terra, E. L. (2002) “The impact ofCorpus Size on Question Answering Performance”. In: Beaulieu, M., Baeza-Yates, R. & Myaeng,S.H. (eds.), Proceedings of the 25th Annual ACM Conference on Research and Development inInformation Retrieval, 369-370, Tampere, Finland: ACM Press.
Craswell, N., Hawking, D., Wilkinson, R. & Wu, M. (2003). "Overview of the TREC 2003 WebTrack". In: Voorheer, E. (ed.), NIST Special Publication 500-255:The Twelfth Text REtrieval
Conference (TREC 2003), Gaithersburg, MD. pp. 78-92. NIST.
Gurrin, C. & Smeaton, A. (2004). "Replicating Web Structure in Small-Scale Test Collections".
Journal of Information Retrieval, 7 (3-4), 239-263.
Jones, C.B., Purves, R., Ruas, A., Sanderson, M., Sester, M., Kreveld, M.v. & Weibel, R. (2002).
"Spatial information retrieval and geographical ontologies an overview of the SPIRIT project". In:
Beaulieu, M., Baeza-Yates, R. & Myaeng, S.H. (eds.), Proceedings of the 25th annualinternational ACM SIGIR conference on Research and development in information retrieval,Tampere, Finland. pp. 387-388. ACM Press.
i

The project web site: http://www.geo-spirit.org/ [Last accessed: 27/09/2004].
This tag is not used in the Web Track.
iii
Due to the diversity of document structures, the percentages are not always based on the total number ofdocuments in the collection. The margin of error in the figures is estimated to be approximately 0.01%.
ii