Only remove observer if we're observing

We’re only observing this object if we have a delegate for it

Closes #1763