let remove_specialized actually remove all workers

It looks like this was always intended, but to_remove==NULL wasn't doing anything. This stops the FD leaking and worker lack-of-disconnect in #10 

Fixes #10