actually remove the worker from the workers set, too