remove worker listeners when they exit