Only get extra work in fill_queue if we don't have any unqueued work in the list.