Fix an issue where KVO observer was not properly deregisterd