Ensure all observers are removed when destroyObservable is called.