Make sure managed context observers are removed