INEX REPORTThe Wikipedia XML Corpus
Ludovic Denoyer, Patrick Gallinari
Laboratoire d’Informatique de Paris 6
8 rue du capitaine Scott75015 Paris
http://www-connex.lip6.fr/denoyer/wikipediaXML{ludovic.denoyer, patrick.gallinari}@lip6.fr

1

Introduction

Wikipedia1 is a well known free content, multilingual encyclopedia written collaboratively by contributors
around the world. Anybody can edit an article using a wiki markup language that offers a simplified
alternative to HTML. This encyclopedia is composed of millions of articles in different languages.
Content-oriented XML retrieval is an area of Information Retrieval (IR) research that is receiving an
increasing interest. There already exists a very active community in the IR/ XML domain which started
to work on XML search engines and XML textual data. This community is mainly organized since 2002
around the INEX initiative (INitiative for the Evaluation of XML Retrieval) which is funded by the DELOS
network of excellence on Digital Libraries.
In this article, we describe a set of XML collections based on Wikipedia. These collections can beused in a large variety of XML IR/Machine Learning tasks like ad-hoc retrieval, categorization, clusteringor structure mapping. These corpora are currently used for both, INEX 20062 and the XML Document
Mining Challenge3 . The article provides a description of the corpus.
The collections are downloadable on the website:• http://www-connex.lip6.fr/∼denoyer/wikipediaXML

2

Description of the corpus

The corpus is composed of 8 main collections corresponding to 8 different languages4 : English, French,German, Dutch, Spanish, Chinese, Arabian and Japanese. Each collection is a set of XML documents
built using Wikipedia and encoded in UTF-8. In addition to these 8 collections, we also provide different
additional collections for other IR/Machine Learning tasks like categorization and clustering, NLP, machinetranslation, multimedia IR, entity search, etc.

2.1

Main Collections

The main collections are a set of XML files in 8 different languages. The table 1 gives a detailed descriptionof each collection.
Each collection contains a set of documents where each filename is a number corresponding to the id of
the file (for example : 15243.xml ). Each id is unique and each file corresponds to an article of Wikipedia.
1

http://www.wikipedia.org
http://inex.is.informatik.uni-duisburg.de/20063
http://xmlmining.lip6.fr
4Some additional languages will be added during the next months.
2

ACM SIGIR Forum64

Vol.40 No. 1 June 2006

Collection name

Language

Number of documents

main-english
20060130 french
20060123 german
20060227 dutch
20060130 spanish
20060303 chinese
20060326 arabian
20060303 japaneseEnglish
FrenchGermanDutch
SpanishChineseArabianJapanese659,388
110,838
305,099
125,004
79,236
56,661
11,637
187,492

Sizeofthe(MegaBytes)≈ 4,600≈ 730≈ 2,079
≈ 607
≈ 504
≈ 360
≈ 53
≈ 1,425collection

Table 1: General statistics about the Main Collections
We only kept articles and removed all the wikipedia pages corresponding to ”‘Talks”’, ”‘Template”’, etc..
Each file is an UTF-8 document which is created from the wikitext of the original article. Figure 1 gives anexample of an English article extracted from the corpus.
Tag labels We introduced different tags in order to represent the different parts of a document. We
distinguish two types of tags:• The general tags (article,section, paragraph,....) that do not depend on the language of the collection.
These tags correspond to the structural information contained in the wikitext format (for example :== Main part == is transformed into <title>Main part< /title>)• The template tags (template infobox,...) represent the information contained into the wikipedia templates. Wikipedia templates are used to represent a repetitive type of information. For example, eachcountry described into wikipedia starts with a table containing its population, language, size,... Inorder to uniformize this type of information, wikipedia uses templates. These templates are translated
into XML using tags starting by template ... (for example : template country). The template tags
depend on the language of the collection because the templates are not the same depending on thelanguage of the wikipedia collection used.
The DTD is downloadable on the Web site.
Statistics about the collections These statistics are given in table 2Language
EnglishFrenchGermanDutch
SpanishChineseArabianJapanese

Mean size of document (bytes)
7,261
6,902
7,106
5,092
6,669
6,664
4,826
7,973

Mean Document Depth
6.72
7.07
6.76
6.41
6.65
6.91
5.85
7.1

Number of Nodes/Document
161.35
175.54151.99122.8
165.74179.23
182.1
94.96

Table 2: Statistics about the structure of the documents from the Main Collections

ACM SIGIR Forum65

Vol.40 No. 1 June 2006

the wiki text

The XML obtained
Figure 1: Example of wiki → XML transformation for the Anarchy article (12.xml )ACM SIGIR Forum66

Vol.40 No. 1 June 2006

2.2

Categories

The documents of the wikipedia XML collections are organized in a hierarchy of categories defined by theauthors of the articles. For each main collection, we propose a set of files describing:• the hierarchy of categories (file : categories hcategories.csv)• the categories of each articles (file : categories categories.csv)• the categories names (file : categories name.csv)
Table 3 gives statistics about the categories.
Language
EnglishFrenchGermanDutch
SpanishChineseJapanese

Number of categories in the hierarchy
113,483
28,60027,981
13,847
12,462
27,147
26,730Mean number of categories for each document
2.28491.95702.58401.66281.61802.07972.0039

Table 3: Statistics about the categories of the Main Collections

3Additional collections

We also propose additional collections. These collections can be used in a large variety of Information
Retrieval and Machine Learning tasks. Some other collections will be added in the future and are notdescribed here.

3.1

Categorization/Clustering Collections

English Multi-Label Categorization Collection For the English collection, we also provide a MultiLabel Categorization Collection with :• A list of articles from the Main English Collection• A set of categories (without hierarchy, based on the Portals of wikipedia): a document belongs to one
or more categories
This corpus can be used to compare categorization algorithms and it is described in table 4.
Number of Documents
Number of categories
Mean number of categories for each document
Mean number of documents for each category
Number of documents in the larger category
Number of Documents in the smaller category

415,31072
2.2
12,5137,9108

Table 4: Statistics about the English Multi-Label Categorization Collection

ACM SIGIR Forum67

Vol.40 No. 1 June 2006

English Single-Label Categorization Collection - XML Document Mining Collection We
provide a specific collection where each document belongs to exactly one category. It is composed ofthe documents of the preceding collection belonging to a single category. This collection can be used for
categorization and clustering of documents (see table 5). This collection is aimed at categorization/clustering
benchmark.
Number of categories
Number of documents
Number of train documents
Number of test documents
Mean number of categories for each document
Structure of the corpus

60
150,094
75,04775,047
1
The directory documents contains all thecorresponding articles.
The directory
relfiles contains one file per category giving the id of the documents that belongs
to this category5 .

Table 5: Statistics about the XML Document Mining Challenge Collection (Single-Label CategorizationCollection)

3.2

Multimedia English Collection

This collection corresponds to the Main English Collection with the pictures of the different articles. This
collection can be used for Multimedia Information Retrieval. Table 6 gives statistics about this collection.
Number of documents
Number of pictures
Approximate size of the corpus659,388
more than 300,000≈ 60GbTable 6: Statistics about the Multimedia English Corpus

3.3

Entity corpus

We provide an Entity Corpus where each article of the Main English Corpus has been tagged using a set ofpossible entity types extracted using the different categories of wikipedia. For example : Silverster Stallone
has been tagged as <actors>Silverster Stallone< /actors>. Table 7 gives statistics about this collection.
Number of Documents
Number of documents
Size of the corpus

XML Entity collection
659,388≈ 6 GbTable 7: Entity Collections

4Conclusion

This technical report describes XML collections based on Wikipedia and developed for Structured Information Retrieval, Structured Machine Learning and Natural Language processing. Other collections will beadded in the future.

ACM SIGIR Forum68

Vol.40 No. 1 June 2006

5Acknowlegment

The wikipediaXML corpus is distributed under the GPL Documentation license. It is completely free andcan be used for non-profit educational and research purposes. All publications based on the wikipediaXML
corpus should cite this technical report.

ACM SIGIR Forum69

Vol.40 No. 1 June 2006