Removes the observer when it's not necessary any more. Fixes #21