apache/incubator-spark — pull request #546 by CodingCat (2014-02-05 21:48:32+00:00)
Title: LRU-based load balancing in master (SPARK-979)

this PR is for SPARK-979(https://spark-project.atlassian.net/browse/SPARK-979), where @rxin pointed out that the Spark Scheduler is very deterministic.

I built a LRUMap with twitter-scala-util, and used actorToWorker structure in master.scala for load balancing (to my surprise, I found that this data structure is actually not used in the program, except insert/delete when register/remove the worker)

This data structure is the best candidate for LRU load balancing. Others like idToWorker are not so good because the program will call .get in other places (e.g. Heartbeat), causing the order of the workers always change. In fact we shall only change the order with LRU when assigning in-cluster driver and executor to the workers.

Hi, @xiajunluan, I know you are the current assignee, can you help to check it?