Remove observer only when it was added