mesos/spark — pull request #147 by mosharaf (2012-07-15 18:17:06+00:00)
Title: Broadcast refactoring/cleaning up

- Removed DfsBroadcast completely.
- Removed ChainedBroadcast (TreeBroadcast is a superset of this)
- Refactored code for TreeBroadcast and BitTorrentBroadcast (still there seems to be a lot of code overlapped, but they have differences in the details). The common object is called MultiTracker that basically acts as the torrent indexing service for ongoing broadcasts.
- Added stop() for the broadcast subsystem. Haven't added it to SparkContext though
- Merged several configuration options to one for simplicity. The ones that remain are setup to work without any changes. Just changing the broadcastFactory should do.
- Changed "isLocal" in SparkContext to not consider localhost as local. To use local you have to use "local" and not "localhost". The reasoning is that using the localhost IP still acts like that, but localhost doesn't; plus, it helps in checking broadcast in the same machine without having to use the actual IP.

TODO
- Add a HttpBroadcast backup for TreeBroadcast and BitTorrentBroadcast methods just in case they fail. TreeBroadcast shouldn't. BitTorrentBroadcast has a weird corner case that will take some time in figuring out and hasn't been addressed in this patch.