dbpedia/extraction-framework — pull request #91 by ninniuz (2013-11-04 14:09:44+00:00)
Title: Split mediawiki dump into chunks starting from multistream version

I wrote a simple Mediawiki XML dump splitter which uses the multistream version of the pages-articles dump [1]
The splitter also needs an index file which contains offsets, as released together with the multistream dump, see [2]-[3]

The splitter can be configured to produce combarable-sized XML dump chunks (based on the file size and not the number of pages in it).
On my machine it takes about 5 minutes to split the huge dump into ~64MB chunks.

Pros:
- can be useful when the number of cores available is bigger than the number of chunks produced by Wikimedia (e.g. a 32 cores machine)
- could reduce the extraction time since the chunks are similar in size

Cons:
- the multistream file is bigger than the sequential file (download times increase)
- it is required to also download the index file [3]
- splitting the multistream dump takes time (some minutes on my machine with a SSD)
- the multistream file is generally produced and distributed at the end of the dump process

[1] http://permalink.gmane.org/gmane.science.linguistics.wikipedia.research/1803
[2] http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2
[3] http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream-index.txt.bz2