Add support for specifying master's memory : critical when non trivial processing is done on it. Also add -server to the params
In yarn mode, run as the job user - not yarn user
Add support for changing write-timeout, and set default to 30s from the default 10s in yarn mode
When a block is dropped from memory to disk, it is not necessary that the disk size is same as memory size.
This change passes along the block's size in memory in the heartbeat - so that the masters view of the available free memory is same as the slaves.
Add property spark.akka.askTimeout to allow for configuring remote'ing timeout
Modify exception so that it can be used to indicate issues not just with fetching data from peer slave, but also fetch locations from master
For tight loops, allow for checking if trace is enabled before creating msg/invoking trace
Fix MT bugs, sync issues
Fix bugs with empty env var, comment out gc expt
Fix bugs with empty env var, comment out gc expt
Add support for fetching cached rack to host list (not exhaustive list, but ONLY cached based on previous queries)
Changes to ConnectionManager and Connection are related.

ConnectionManager

1) Try to make selector loop more responsive - offload processing (read, write, connect) to its own thread pool.
This severely improves performance and responsiveness under heavy load (including preventing shuffle from failing when data per key is high).

2) Make it MT safe as part of (1).

3) Fix multiple bugs (not waking selector as part of registering interest, multiple outbound connections in receiveMessage, etc).

Connection
1) Multiple containers can be hosted on a single node - hence, relying on ConnectionManagerId.fromSocketAddress(remoteAddress) will not work.
Infer remoteManagerId for ReceivingConnection from the incoming message header.
Consistent getRemoteConnectionManagerId is required for closing pair of send/receive connections.

2) Modify read and write to return boolean to indicate whether to re-register for read/write op.
1) Add a thread to revive requests periodically if there are pending tasks available.
By default turned off - controlled via spark.tasks.revive.interval specified in milliseconds.

2) Add support for more aggressive scheduling when host local tasks are not present.
Controlled via spark.tasks.schedule.aggression - takes values
a) HOST_LOCAL (default - preserves existing behavior)
b) RACK_LOCAL
c) ANY
What this allows is - after all host local tasks are scheduled, if there are pending tasks and nodes available, depending on
the aggression specified, we will try to schedule RACK_LOCAL or ANY tasks.
This means we do not need to wait for locality wait before scheduling non host local tasks.
Note, this is NOT the same as setting spark.locality.wait = 0, and results in much better task scheduling and locality performance.
Mostly relevant only when size of cluster is large.
1) Remove BitLocker from BlockManager - it is horribly buggy idea and leads to all sorts of non trivial deadlocks.
Thankfully, this is pretty much what is in trunk too right now - though there are a few more bugs fixed related to not calling waitForReady.
Brought over some excellent changes by Reynold Xin (rxin) from trunk - fixes a lot of our severe issues !

2) Fix bug related to incorrect reporting of rdd size to master via reportBlockStatus : master ends up having wildly divergent memory size for slave as iterations proceed.
Related to removing block from memory (to disk) : the resulting difference was not handled properly.

3) Fix core dump due to bugs in use of memory mapped buffers. Rewind assumes that setting position to 0 is correct (which was not the case at times), buffer directly modified when a duplicate should be modified, etc lead to these issues.

4) Do atomic updates via putIfAbsent on ConcurrentHashMap - else we end up with issues. (This is a general issue with the codebase and hopefully has been fixed everywhere).

5) Moved MemoryStore to (mostly) what is in trunk - too many issues observed, most of which seem to have been thankfully fixed in trunk.
Add assertion to ensure rewind on the nuffer will not cause issues
Register paths to be deleted by shutdown hooks and manage them - so that different hooks do not end up trying to delete each others path resulting in IOException and improper cleanup
1) Improve task placement for rack locality.

2) Fix bugs introduced in initial commit of rack locality and task scheduling in yarn branch.
Schedule no-pref tasks after rack local tasks
Remove internal comment
Always use 30 - remove is-yarn condition