apache/spark — pull request #636 by CodingCat (2014-05-04 22:08:08+00:00)
Title: SPARK-1706: Allow multiple executors per worker in Standalone mode

https://issues.apache.org/jira/browse/SPARK-1706

In current implementation, the user has to start multiple workers in a server for starting multiple executors in a server, which introduces additional overhead for more JVM processes...

In this patch, I changed the scheduling logic in master to enable the user to start multiple executor processes within the same JVM process.

Other small change includes

1. change memoryPerSlave in ApplicationDescription to memoryPerExecutor, as "Slave" is overrided to represent both worker and executor in the documents... (we have some discussion on this before?)

@pwendell, I think we don't need to change anything in scheduler part, as we indexed the executor by executorId instead of host IP address?