dbpedia/distributed-extraction-framework — pull request #56 by nilesh-c (2014-08-17 00:34:58+00:00)
Title: Completed distributed downloads

This PR completes distributed downloads more or less. A few more things will be added:
* More documentation, perhaps a diagram to explain how the actors work
* Optimize for clusters with unique public IPs for each slave node
* Clean up logging, log download progress in terms of percentages, how much done, how much left. The infrastructure is there, just need to polish up the crude list of messages and compute percentages. :)

I tested this with one master and 2 workers using three machines at my home. Not tested on GCE yet, will need to refactor the spark gce cluster setup script for that. @sangv I'll let you know after I add more documentation and a README so that it's easy for you to test it out.