Don't add the reference element to an absent parent

If added to a parent, ensure it is still actually removed when an error
occurs.