apache/incubator-storm — pull request #71 by ptgoetz (2014-04-10 18:01:08+00:00)
Title: STORM-150: Replace jar distribution strategy with bittorrent

This is a resurrection of a pre-apache pull request: https://github.com/nathanmarz/storm/pull/629
(original discussion and code review can be found there)

I've updated it to merge cleanly with the latest master branch and fixed a bug that would prevent multi-lang topologies from working.

Using bittorrent to distribute topology code will make is easier to implement HA Nimbus easier since multiple nimbus nodes could seed to one another when a topology is submitted (i.e. each would have a full copy of the code). We could also incorporate it into leader election such that only a nimbus node with a complete copy of topology code could be elected leader.

It also helps alleviate the "thundering herd" issue where supervisors all hit nimbus at once to download jars.

I've tested this on multi-node clusters with regular, multi-lang, and trident topologies, as well as topology jars larger than 100MB.