BlockManagerMasterActor - memory visibilty bug fix - blockManagerInfo,
blockManagerIdByExecutor mutable maps replaced with volotile immutable
maps.

We have encountered the following issue:
Driver craches every 1-3 days.


Here are 2 lines from my Driver log file:

2014-09-10 18:50:44,510 ERROR [spark-akka.actor.default-dispatcher-46]
TaskSchedulerImpl - Lost executor 201408311047-3690990090-5050-30951-12
on spark106.us.taboolasyndication.com: remote Akka client disassociated
2014-09-10 18:51:46,062 ERROR [spark-akka.actor.default-dispatcher-15]
BlockManagerMasterActor - Got two different block manager registrations
on 201408311047-3690990090-5050-30951-12

BlockManagerMasterActor.scala:
private def register(id: BlockManagerId, maxMemSize: Long, slaveActor:
ActorRef) {
	...
	logError("Got two different block manager registrations on " +
id.executorId)
	System.exit(1)
	...
}

Looks like Driver is disassociated from Spark worker.
One minute duplicated block manager registration happens.

Production environment details:
We have a 24/7 Spark running job on Mesos (in coarse mode).
Each machine has 24 cores.
We confiure Akka to use 70 threads



Both blockManagerInfo, blockManagerIdByExecutor are modified together
(add at 'register', remove at 'removeBlockManager')
We suspect that it is a memory visibilty bug. (Akka Actor runs on more
then one thread (event that process messages it synchronously).

We replaced went for the 'copy on write' solution, witch is volatile
immutable map (instead of syncronized map).
Since blockManagerInfo is read frequently and modified rarely.