Properly erase/remove a DebuggerObserver