Yes, but it minimizes the cost of network IO - which can be non trivial
when the cluster is relatively loaded.
And if the task scheduling ends up running the job on the same container
which is hosting it - then the network IO cost goes done to zero.

- Mridul



On Tue, Apr 30, 2013 at 5:49 AM, Reynold Xin <notifications@github.com>wrote:

> In core/src/main/scala/spark/scheduler/ShuffleMapTask.scala:
>
> >    extends Task[MapStatus](stageId)
> >    with Externalizable
> >    with Logging {
> >
> >    protected def this() = this(0, null, null, 0, null)
> >
> > +  // data locality is on a per host basis, not hyper specific to container (host:port). Unique on set of hosts.
>
> Even if the partitions are on the same host, the worker would still need
> to serialize / deserialize those data when they are in different jvm
> processes. Often that is very costly ...
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/mesos/spark/pull/575/files#r4010789>
> .
>