I am not very sure of this piece of code ... But it is too late in the
night right now, so I don't want to make obviously stupid comments due to
exhaustion :-)
I am not sure if I can get to this PR coming week; please do get it checked
out by someone else too !
On 12-May-2014 5:26 am, "Nan Zhu" <notifications@github.com> wrote:

> In core/src/main/scala/org/apache/spark/deploy/master/Master.scala:
>
> > @@ -532,6 +516,99 @@ private[spark] class Master(
> >      }
> >    }
> >
> > +  private def startMultiExecutorsPerWorker() {
> > +    // allow user to run multiple executors in the same worker
> > +    // (within the same worker JVM process)
> > +    if (spreadOutApps) {
> > +      for (app <- waitingApps if app.coresLeft > 0) {
> > +        val memoryPerExecutor = app.desc.memoryPerExecutor
> > +        var usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE).
> > +          filter(worker => worker.coresFree > 0 && worker.memoryFree >= memoryPerExecutor).
> > +          sortBy(_.memoryFree / memoryPerExecutor).reverse
> > +        val maxCoreNumPerExecutor = app.desc.maxCorePerExecutor.get
> > +        // get the maximum total number of executors we can assign
> > +        var maxLeftExecutorsToAssign = usableWorkers.map(_.memoryFree / memoryPerExecutor).sum
> > +        var maxCoresLeft = maxLeftExecutorsToAssign * maxCoreNumPerExecutor
>
> the idea here is, user has an expectation on the maximum cores to assign
> to the application, but this expectation is usually not achievable due to
> the limited cores in each worker;
>
> so the allocation here is to decide executorNum per Worker according to
> the memory space on each worker (because this is a hard limitation), and
> meet the user's expectation on the cores with the best efforts
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/731/files#r12511614>
> .
>