To add to above (late night comments are not always very optimal :) )
There were two intentions behind the earlier piece of code :

a) Given a seq of WorkerOffer, ensure that the order in which they are passed on to TaskSetManager is such that relatively optimal scheduling of workers occurs.
Namely, prioritize instance local, node local, rack local and any in that order.

b) Once all optimal schedules have happened, expand the scope so that non optimal schedules are also done such that cluster utilization is mazimized (within bounds controlled via aggression and time spent waiting).
These are also done in order of our prioritity - so even if aggression was set to ANY, we will try to do rack local schedules first, before ANY.

This tries to ensure that we balance optimal scheduling with maximizing utilization of cluster resources.
Invocation of resourceOffers with single workoffer happens when a task finishes/fails; which is unfortunately not very optimal - I never got around to resolving this and relying only on reviveOffers() (bulk submission of all available workers)

Regards,
Mridul