Check worker exists before removing