Don't add the reference element to the parent node if there is none.

If added to a parent, ensure it is still actually removed when an error
occurs.