This still feels racy because at the same moment 2 threads could end up doing local cleanup for the destroy.

There is no need for this; I would introduce volatile state field, if it isn't available yet; and use a cas to determine which thread is going to do the actual destroy. If we do it, we should do it good.