Fix bug where workers sometimes never destroy themselves

Fixes a bug where, if a pending message returned an error response, a pending
destory() operation may never complete.