Make sure to always clean up test background processes

You can't have multiple traps for EXIT, and this is less error prone.
Drop workers who have connection errors

This makes sure we don't send clients to workers that don't exist. We
actually have a complete test for this using a channel based connection.
It still needs a little work to remove the need for a certain number of
GOMAXPROCS but it's better than an integration test.
WORK IN PROGRESS