Don't even bother blocking if we know our queues are empty