core: Fix and tidy up removal of disconnected workers

This simplifies removal a bit, making it an explicit part of the
lifetime of a worker and allows us to reduce the complexity of the parts
of the code that deals with shutdowns/disconnects.

Signed-off-by: Anton Lofgren <alofgren@op5.com>