Add back destroy fix unassign

Ah sorry didn't realize that the method is called in the `#destroy` method, nm this comment.