Fix KVO issues.

The previous implementation would sometimes over remove causing a crash. Switched to a much simplier version of tracking if we've observed it or not.