openplanets/nanite — pull request #8 by willp-bl (2013-12-09 11:44:56+00:00)
Title: Map-reduce program to check whether gz files can be opened successfully

Some of the warc files we have for testing are zero length, or may not be valid gz files. This map-reduce program quickly checks all gz files to see if there are any issues, status is shown in the output from the reducer.

It would be possible to chain this map-reduce to run immediately before the format-profiler.

Approx runtime for this program is ~5mins/~15k warc files totaling 1TB on our cluster.