Handle proper deactivation of workers