Fix bug where delayed destroy() would destroy too much