Properly remove old keys after set changes child ids