Removes deleted workers from task.workers