Cope with missing workers

Also, improve the Redis new job queue monitoring mechanism, so the system shuts down cleanly.