Only remove listeners on destroy for nodes that exist