Do not assume an error when removing a worker

Fixes 500 errors when reloading code via SIGHUP