Back off of empty queues better to avoid wasting time

* This also helps the loop switch into other priority faster

Change-Id: I20aba066cae5b73cd69a8f15004576e554797812