I'm concerned there's a starvation issue, because the TaskSchedulerImpl
doesn't take CPUS_PER_TASK into account.  If you consider a case where the
cluster is busy running jobs where CPU_PER_TASK is set to 1, and a new job,
call it job A, arrives with CPUS_PER_TASK set to 2.  Every time a single
CPU becomes free (due to a task ending), job A will reject the offer,
because it needs 2 CPUs.  However, the scheduler doesn't understand this
reason, so it will offer the slot to other jobs.  This means there can be a
situation where job A never gets any resources, because it is never offered
2 CPUs.  Maybe I'm missing something here?

This issue is unrelated to your pull request so maybe it makes sense to go
ahead and fix the bugs you've mentioned here, and open a separate JIRA for
the starvation issue.


On Sun, Jan 26, 2014 at 11:16 PM, Mridul Muralidharan <
notifications@github.com> wrote:

> In core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
>
> > @@ -241,7 +241,8 @@ private[spark] class TaskSchedulerImpl(
> >              taskIdToExecutorId(tid) = execId
> >              activeExecutorIds += execId
> >              executorsByHost(host) += execId
> > -            availableCpus(i) -= 1
> > +            availableCpus(i) -= taskSet.CPUS_PER_TASK
>
> CPUS_PER_TASK per-se is not buggy : it allows users to specify how many
> cores they need per task (for example, if the task requires 4 cores per
> task in case it is doing something non trivial).
> The default is 1 - folks who customize this value would probably know what
> they are doing (and ensure cores per worker is a multiple of CPUS_PER_TASK,
> for example).
>
> I assumed you meant about availableCpus variable.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/apache/incubator-spark/pull/507/files#r9181129>
> .
>