Use pointers rather than UIDs to track adds/removes