Hmm, looking at the submitted diffs - I think we left hyper local scheduling in, but removed it from locality preference from RDD. Dont recall why this was done - but I suspect it was done to minimize disruption since we had other significant issues to resolve (too many host -> hostport changes, and there was no way to validate this easily since it can be both host or hostPort).

So the final code which was committed into spark yarn branch, and now subsequently to spark master does not have this change. Apologies for my oversight in the response above !


The change to enable this is actually pretty simple - once the patch stabilizes, I can submit a pull request which adds it back in. We have does this before anyway :-)


Thanks for looking into this and bringing it up !