Queue depth and num workers can race, so make sure that we start at least one worker when there are none active for that queue.