dbpedia/distributed-extraction-framework — pull request #15 by nilesh-c (2014-05-26 01:04:25+00:00)
Title: Scalable job outputs - iterate over RDD elements locally and write to local file

An rdd.collect is replaced by iteration - now results aren't bounded by the driver (master) node's memory.

Note that this solves writing the output to local files on the **driver** node/machine. With huge data (running for days) people may want to write to HDFS too because it's faster. Once we have the InputFormat up I'll add provision for that too.

Note: The pull request is to the **nildev** branch. Let me know what you think and I'll merge this into nildev.