fix locality inversion bug in TaskManager by moving nopref branch
avoid unnecessary delay when we only have nopref tasks
simplify the patch