Only remove observer if we're observing

We’re only observing this object if we have a delegate for it

Closes #1763
Merge pull request #1769 from kylef/remove-observer

Only remove observer if we're observing