Make sure we have correct number of workers at restart or graceful. This may happen for instance if workers die too soon and we restart again.